So over the last few months I have been working on a large system that I would like to be easily customizable and fast to deploy. The core idea is building workflows for myself and evaluating the performance over time using my own custom metrics. The workflows are quite complex (think of getting api data, enriching using different requests and transforming the data both using code and ai models).

Now, I have been running this on n8n as an orchestrator that sends requests to my own api to perform certain tasks using my services. The issue is now that I’ve been noticing some performance issues with n8n. I am running the community edition on a 8c16gb vps using docker and allocated 8gb to n8n and 4gb to the runners.
My biggest issue is that with some workflows, once it gets into large volumes of data (think 80-100 loaded html pages, moved to md, then sent to ai), with all data transformations n8n just freezes for minutes.

Using n8n is in some ways handy and I have spent wayyyy too much time on the current workflows, but it also has its quirks that make it a pain in the ass sometimes. Moving most of the code and transformations to python would make it way more efficient (especially since I can just delete unnecessary data from memory) and I will be able to make it more intricate. My main fear though will be that it will be harder to update, since I will have to delve into the code instead of using a simple GUI. Does anyone have any experience with this?

P.S. I might be able to implement some variable management into my dashboard, but that might also take some time.