Service recommendation for running/hosting agent loop
Posted by Unlikely_Secret_5018@reddit | ExperiencedDevs | View on Reddit | 4 comments
I have a Flask web app for small business owners to automate certain processes. I've integrated an LLM chat via calls to OpenAI using their agents SDK. The agent loop just runs in my web app currently, doing web search, reading context from the customer's uploaded data, and mainly just streams text back.
I'd like to add the ability for the agent to generate multi-page invoices and other documents for my users to give to their clients, which tend to take many minutes.
It seems like I'd need to move the agent loop out of my web app into a more robust worker service. But which one should I choose? It needs to:
- Support several agent loop instances generating documents for many users
- Be able to stream the initial response back to my Flask app (eg. "Ok, I'll generate that for you now...")
Advice appreciated, thanks.
Leading_Yoghurt_5323@reddit
separate the loop from your web app for sure… use a queue (celery, dramatiq) + workers, and stream updates via websockets or polling
Unlikely_Secret_5018@reddit (OP)
How do you stream results with websockets? Do Celery workers support that?
Hieulam06@reddit
Moving your agent loop to a better service is a smart move... Check out AWS Lambda or google Cloud Functions for scaling up. They can run a bunch of instances and send responses back to your Flask app without too much trouble. For hosting your Flask app, I usually look at ipickedyourhosting; they have some good picks for different setups you might like.
Unlikely_Secret_5018@reddit (OP)
Interesting, I never thought of the "function" runners services.
My main app already runs on Google cloud run, containerized.
Will I be able to stream the initial response back from those function runners? Do you foresee any any issues from those jobs being able to stream results back or access my app DB for more context in the middle of the agent loop?