Over the years having implemented quite a few file interfaces, with similar goals but different design principles (at the time best practice), of which the core is file based ETL with flow routing. With the recent project and juniors to train, I gave Nifi a try, partly because it sports a great UI and is fairly simple to hand over to system administrators, with build in authorisations and policy management.
To give some context to my use of the word serious, the recent integration required files to be interfaced from the logistics system to the financial system and then document management system. The weekly export process was running batch exports and produced approximately 2000 files within minutes with about 100k line items in total each line item translates to 3 financial transactions, which the interface had to generate and track until completed.
More about Nifi on the official site https://nifi.apache.org
Due to the vast library of ready to use processors, it is a great tool for fast POC work or simple flows but.... perhaps my use case of XML file integration between systems supporting SOAP,REST and SAP RFC is an abuse of the intended use, but the user interface quickly became very crowded, with lots of groovy scrips on the operating system level. It soon reminded me of the reason why big business workflows are typically implemented in code, or eventually migrated to code. Point in case, Uber and AirBNB. To say the least, managing failures became a tedious task.
The good:
To extend the tool with custom processors java, was quite trivial. In my case I wrapped the JcoSon library to directly interface to SAP ERP. This solution works great.
The not so great:
Flows need error paths, a good thing, but handling the errors in a sane way with complex flows is a challenge in the UI.
In the end I decided to take a micro service approach in gloang, with technologies like docker, minio, cadence, and kubernetes all running on premise, with a simple UI in Vue.js showing file interface status, and some additional data to enable a user to quickly spot failures. Golang is just a great systems language.
No comments:
Post a Comment