How are you maintaining your AI apps post-launch? Model bugs vs engineering bugs, and what's your debugging stack?
Posted by fgp121@reddit | LocalLLaMA | View on Reddit | 5 comments
I've been going down a rabbit hole tinkering about what actually happens after you ship an LLM-powered app, and I'd love to hear how others here handle it…
A few things I keep getting stuck on:
Continuous optimization. Once your app is in users' hands, how often are you tweaking prompts, swapping models, retraining adapters, or rebuilding RAG pipelines? Is it a constant grind or do you reach a good-enough plateau?
Model bugs vs engineering bugs. When something breaks, how do you even tell whether it's the model hallucinating or regressing vs a plain old code or infra issue? Do you have evals catching it, or is it mostly user reports?
Do you also regularly update your evals or is it once built and forget about it workflow?
Your dev loop. Are you debugging and iterating with local models using harnesses like Pi, Hermes, Aider, or Cline? Or are you just leaning on Claude Code or Cursor and calling it a day? Anyone running a hybrid setup?
Curious whether the local-first crowd here has fundamentally different workflows from the API-only folks, especially around catching model regressions when you swap weights or quantizations.
What's working, what's painful, what would you change?
rashaniquah@reddit
I test out everything with gemini-flash-lite then swap out for a slower/better model on prod.
I usually aim for at least 90% F1.
Embedding and reranker models are very underrated. It's 1000x faster, 1000x cheaper and sometimes doesn't even require a query to process the data.
Future_Manager3217@reddit
One thing that helps is to stop labeling failures as just “model bugs” vs “engineering bugs”. I usually split the incident into five surfaces:
Then every real incident should add either a fixture, a replay slice, or an invariant. Evals are not build-once for me; they become the regression suite from production failures.
For model swaps or quant changes, I’d freeze a small set of input+state snapshots and compare action traces, not only final answers. A local harness is great for cheap loops, but the pass/fail should still mirror the production tool/schema/context boundary.
Party-Log-1084@reddit
I just log every raw prompt and output to a database. When something breaks, I check exactly what was sent to the model. If the final prompt is missing the RAG context, it's a code bug. If the prompt is perfect but the answer is trash, it's the model. Optimization usually hits a plateau pretty fast. I only mess with prompts or swap models when a significantly better open weight model drops. Evals are annoying to maintain, so I just keep a small json of known edge cases and run those locally whenever I test a new quant.
gvij@reddit
Honestly my workflow has split into 2 layers right now..
For specific code-level issues and fixes I bounce between Pi and Roo Code, with Qwen 3.6 27B q8 running locally through llama.cpp. Fast iteration, no API costs, and I can actually see what the model is doing.
For the Agent harness engineering side and model evals/regression testing and optimizations I lean on Neo AI. Saves me from rolling my own eval setup every time I want to test a prompt change or compare model outputs.
Works well as a combo so far.
Disclosure: I am founder of Neo AI. So a natural plug but genuinely sharing my workflow here.
ai_guy_nerd@reddit
The distinction between a model bug and an engineering bug usually comes down to the delta. If the output is wrong but the logic flow is correct, it is a model regression. If the agent loops or hits a 404, it is engineering. The best way to catch this is a golden set of 20-50 prompts that must never change. Run them every time you swap a model or a prompt.
Regarding the dev loop, a hybrid approach is usually the only way to stay sane. Use a local harness for rapid iteration on logic, then a high-end API model to verify the ceiling of what the agent can actually achieve. The biggest mistake is trusting a demo of a new model without a proper eval suite.
For those building actual business automation, the most reliable eval is just doing the work manually for the first ten clients. Once the human process is documented, you can actually write tests that mean something. OpenClaw uses a similar manual-first approach to nail the workflow before automating it.