I've been trying to make a real production service that uses LLM and it turned into a pure agony. Here are some of my "experiences".
Posted by DaniyarQQQ@reddit | LocalLLaMA | View on Reddit | 102 comments
Hello everyone. I hope this won't be an off topic, but I want to share my experience in creating real production service. Like a real deal, that will earn money.
For this service I've been using ChatGPT-5 and Claude Haiku 4.5 but I think this could be suitable for other LLMs too.
The idea was as simple as rock. Make an assistant bot that will communicate with people and make a scheduled appointments to the doctor.
Well in a short time I've implemented everything. The vector database that will inject doctor specific knowledge to the conversation at the right time. Multiple tools that will work with doctors data, and couple other integrations. I've extensively made very detailed system prompt, and each tool call returns instructive results. Each tools' parameters' descriptions were written in very detailed way. After testing for a week we finally deployed on production and started to receive conversations from real people.
And then real life had showed a lot of annoying and downright frustrating caveats of these LLMs.
The first frustrating thing is that LLMs makes an assumptions without calling required tool, which deceives people. It happened like this:
User: Please give me an address where this doctor will be on tomorrow.
LLM: Tomorrow is sunday, which is weekend, doctor is unavalable.
There is a tool that explicitly returns that address, and doctor actually works on Sundays. It did not call that tool and made up a bullshit excuse. Then I have emulated this question again by myself:
Me: Give me address where this doctor will be on tomorrow.
LLM: <DID NOT CALL THE TOOL>. Tomorrow is sunday, which is weekend, doctor is unavalable.
Me: Are you sure about that?
LLM: <Finally starts calling the tool which returns address for tomorrow and outputs this address.>
This happens always. No matter what kind of prompts you write, telling it not make any assumptions without any tool calls it still made ups bullshit, which deceives people. Even if you explicitly inject the constraints, it fights them and keeps its own bullshit.
Another problem is close to the first one. LLMs always agrees with requests without calling tools which confuses people. Which looks something like this:
User: I want an appointment for tomorrow. Is it possible.
LLM: Of course you can make an appointment. I've prepared everything. What time?
User: At 18:00
LLM: <Makes a tool call which returns negative result that next day is unavailable>. I'm sorry tomorrow is unavailable.
User: WTF?
Instead of asking proper question before agreeing, it agrees and then shits itself, confusing the user. Especially ChatGPT-5 has this problem, Claude is more rarer but still could shit itself.
And another problem is that LLMs output text which is complete opposite of it's tool results. I've seen this only single time, but I'm now getting paranoid that this could be happening for a long time. It looks something like this:
User: I want an appointment for tomorrow at 18:00.
LLM: <Makes a call to the tool that returns that it is impossible for this user to make an appointment, because user has another pending appointment>
LLM: Good. I've scheduled your appointment for tomorrow at 18:00. Don't forget about that
That was an epic failure, LLM completely lied it's own results. I don't even know what to say about that.
And finally the funny one. Looks like ChatGPT does not like that tools can return negative results, and it keeps calling until it fully overloads all context and finally shits itself. It looks something like this:
User: I want an appointment for next friday at 18:00
LLM: <Calls a tool for available window for next friday. No available window>
LLM: <Calls again this tool but for the next next friday. No available window>
LLM: <Cals AGAIN this tool but for the next next next friday. No available window>
------- And so on and so on | By the way, this doctor does not work on fridays, it was explicitly given in a system prompt, but ChatGPT wants to persevere.
These problems are fixable. You can make even more detailed prompts. Make tools return better and understandable results. You can tune some of LLM parameters. However it is game of whack-a-mole, frustrating one. You fix one thing, another thing comes out. I think some of these models, at least ChatGPT and Claude, were so overly trained on positivity, that they generate deceiving or downright wrong results.
Currently It seems to be that these LLMs can at mostly do their jobs correctly, but these fails, even if they happen rarely, are completely negating all of their reliability. It is not a wonderful magic thing that can solve everything. It is very finnicky (and sometimes very frustrating) tool, that maybe can do what you want. You think you have prepared it for everything, but users can make it shit itself just with a single sentence.
At least I've learned a lot, from these models.
daaain@reddit
These use cases sound much more like workflows to me that could be put on "rails" with good old programming. So what I mean is, not to make it an autonomous agent with tools, but instead a scripted decision tree where the LLM is interpreting and generating responses with preset, structured JSON.
edapstah_@reddit
Why does appointment booking need to be via chat/bot anyway? Navigating and selecting appointment slots from an online interface has been a solved problem for >20 years.
Alokir@reddit
While I prefer to click a few buttons as well instead of typing text, I know many people who get annoyed by, or even get lost in UIs that have a bit of complexity to them. Especially older people who don't use technology other than maybe Facebook or texting. When they see the calendar screen, they'll immediately make a phone call instead.
My guess is that the project's goal is to take the load off of the receptionist or call center.
DaniyarQQQ@reddit (OP)
The funny thing is that I've been thinking about that too while fixing these issues. It seems to be simpler if I just made simple bot with pre programmed buttons.
However, there is another issue. People generally don't like that kind of preprogrammed bots, and prefer human like interaction. These LLM's cover that, mostly, but these rare failures preventing it being a reliable tool.
datashri@reddit
Is it a valid idea to consider using the LLM only for crafting the responses in human language, and using NER to extract relevant information?
koflerdavid@reddit
Then use the LLM to generate questions and answers and to interpret what the user is asking. These are much more specific tasks that you can properly QA. Like this:
moncallikta@reddit
This is the way. Split up each step into classification tasks and build the workflow from those components.
appakaradi@reddit
Can you have another LLM double check the work of the first LLM. I bet this will solve 99% of the issues.
DifficultyFit1895@reddit
Have a third for the tie breaker
AutomataManifold@reddit
If you have control of the front end, the key is to have the LLM figure out the actions to take but don't allow it to tell the user what it did: only verified actions should be reported.
In a lot of cases it is a mistake to show the generated text to the user. Particularly when you have better ways of reporting what actually happened.
If you don't have control of the front end, the less separated version is just manually writing the response.
DaniyarQQQ@reddit (OP)
Don't worry, I don't show that kind of details to users. These examples are way more simplified.
daaain@reddit
Yeah, I guess you could wrap those workflows as tools, but you're kind of back to the same issue then. Definitely try improving the system prompt, but you should not write it yourself, but instead ask Claude or GPT to write it for themselves, based on the existing prompt and all these failure cases. They know quite well what they would pay attention to!
broknbottle@reddit
But then the solution won’t be agentic and the mandate from CEO is that the company is all in on agentic AI.
Photoperiod@reddit
I hate how accurate this is.
daaain@reddit
The CEO won't be able to tell the difference anyway 😹
roosterfareye@reddit
The CEO probably doesn't even know what agentic means. Just another business wank word.
TedHoliday@reddit
Years ago, the owner of my company described for me “an AI” he wanted me to build which could do a certain set of things with accounting systems that he described in magical woo woo language. I’m like uh okay.
So I build this “AI” feature into our system with some SQL and some procedural code, and no learning algorithms or any kind of statistical methods at all, and it’s 100% accurate every time. I did exactly what he wanted it but had nothing to do with AI.
He proceeded to market this feature to clients using the same magical woo woo language he used with me, but they ate that shit up and it made him a lot of money. He was happy, they were happy, I was happy. Everyone got what they wanted but nobody got an AI.
He’s worth 100’s of millions and doesn’t know what he’s selling, but he’s damn good at it. That’s all that really matters. Oh and to his credit, the idea for the feature was all his, and it was valuable, it just wasn’t “an AI.”
Maximum-Wishbone5616@reddit
Agentic is expensive and unreliable. Any change with model and you f****.
Original_Finding2212@reddit
It actually is agentic if you integrate the model well enough.
The rest is semantics
SkyFeistyLlama8@reddit
Agreed. Having an LLM handle most of your application logic is a recipe for disaster. Keep it simple, use shorter chained LLM calls, and implement everything as workflows in a decision tree.
This "agentic" bullshit is beginning to get on my nerves.
ConstantinGB@reddit
Would you know a resource where I could learn about that?
daaain@reddit
This is a great high level overview (which talks about workflows too despite the title having "agents" in it): https://www.anthropic.com/engineering/building-effective-agents
goflapjack@reddit
This might help: https://github.com/Marcus990/Cymbal-Bank-Orchestra/tree/main
Google had a hackathon, and it's on YouTube as well.
I'm impressed by how big their tool prompts are for this project.
Personally, I've been working with vLLM and Granite. Meh... at least we don't have to "pay" for every token.
Btw... be very careful with how people can abuse your bot. You're basically opening your wallet in public and saying, "Hey, people, take only one cent, ok?"
Conscious_Cut_6144@reddit
GLM4.6 wouldn't make these mistakes based on my empierces with tool calls.
That said I have seen results like this in OpenWebUI if you forget to switch the model from Default tool calling to Native tool calling.
FullOf_Bad_Ideas@reddit
as a potential customer I think I'd prefer a HTML form here with some preview of doctor's availability in the corner over non-deterministic AI system. HTML form is pretty reliable and doesn't require GPUs running in the background on some server.
Interesting experience about tool calls nonetheless, I didn't make those sorts of apps yet but I'd have assumed it would be mostly fine by now with current models, though there's a tail of errors that is non-zero and it won't be 100% there. What was your error rate?
do you at least give users an ability to use the old system that is deterministic or did this deployment to prod remove legacy booking with it?
balder1993@reddit
Add to that your medical data being now forever in some cloud LLM provider.
Avelanche23@reddit
I think the tools need to be described well in the code. The functions(tool calls) such as get appointment should have a detailed doc string which helps if it is missing.
huzbum@reddit
I had good results with GPT5-Mini. I recently added an AI assistant to our phone number number search tool. I literally built it in a week including the UI, then added tweaks here and there as issues were reported.
We've got it feature flagged so only internal users can use it for now, but the customer service manager has been thrilled using it internally to help with customer issues.
Maybe it's because my use case is different, or all in english, or maybe I'm in for a rude awakening when it goes into production for real, but I've had very few hallucinations. The only hallucination I remember was process related. It assumed it could do things like update items in the cart, which it did not have any tools for, so I just threw a bullet point in the prompt explaining it could add items but not modify them, and it just stopped.
LLMs seem to like Markdown, so my system prompt is in Markdown. I've found it's best to use the language and formatting you want the agent to use. My actual prompt contains puns related to the character because that's how I want it to talk. I feel like users will be more sympathetic to mistakes if the bot has a semi-silly character.
simplified outline of system prompt:
huzbum@reddit
I think to avoid hallucinations, you need to instruct it what topics it can discuss, then either provide all of that data, or instructions to obtain it.
divinetribe1@reddit
The trick is to stop letting the LLM decide anything. I force all tool calls to happen BEFORE the LLM even sees the query. Like if someone asks about availability, my code calls the availability tool first, gets the real answer, then just has the LLM format it nicely.
Also I built a cache of like 400+ pre-written answers for common questions. Those queries never even touch the LLM so they can’t hallucinate.
The human feedback loop is KEY though - I log everything to Telegram and mark what worked vs what didn’t. It takes like 2-3 months to really dial it in but that’s how you catch all the weird edge cases that make it shit itself.
My accuracy went from 30% to 95% basically by treating the LLM like a text formatter instead of a decision maker. Let YOUR code make the decisions, LLM just makes it sound human.
Honestly the whack-a-mole game never fully stops but with guard rails + caching + human feedback it gets way better.
Crafty-Run-6559@reddit
I've had similar experiences. I've found that LLMs are really only good for use cases that can tolerate hallucinations/errors.
DaniyarQQQ@reddit (OP)
I don't know use case where hallucinations are tolerable. Even in creative writing, that I have used, it gets frustrating at some times.
wolttam@reddit
Where there is an expert human reviewing the output.
ballinb0ss@reddit
Well... the recent Wharton study showed increasing productivity for existing roles. But it emphasized even businesses are beginning to agree that these things can't do much of anything autonomously. Barring another "attention is all you need" architectural shift it appears we won't either for the foreseeable future.
Crafty-Run-6559@reddit
Some examples I can think of are things like classification or specific types of entity extraction, where you don't have enough data or time to finetune/train a more traditional model.
There are a lot of use cases for those where it's tolerable if it's wrong 5-10% of the time.
Other scenarios can be rag style Q/A where the LLM is told to cite the source text, and the end user is warned and told to verify the actual sources that the LLM is citing.
But yeah - I don't really know of any world changing use cases where hallucinations are acceptable either.
dheetoo@reddit
Don't let the LLM decide which tools to call on its own. Remember, LLMs are non-deterministic - they're gonna be unpredictable. What works better is using a router pattern to send user queries to specific workflows you've already defined. But real talk, a lot of tasks are just easier to handle with regular old programming lol. Bottom line: keep your agent tasks as small and focused as possible. The smaller the scope, the less chance of things going sideways.
MrMeier@reddit
You already have the tool API, so you're just a little way from a good traditional appointment system. But I suppose that LLMs are all the rage and sell well.
One option would be to create a decision tree and follow it until the user does something unexpected, at which point you could switch to LLMs. If you proactively ask specific questions, most users tend to answer correctly. Instead of starting with "Hello", start with "Which doctor would you like to see from this list: [Insert list here]" or "On what date would you like to make the appointment? You can extract the data in the traditional way (with a long list of possible answer formats), and if that fails, you can use an LLM that you have finetuned or at least fewshot-prompted.
Even if the user doesn't follow the tree exactly, this method can still work without switching completely to an LLM. Do the same thing you have done with data extraction. Make a list of all the ways you (or an LLM) can think of to write a question or request. Augment this with synonym detection and simply search for a match. This way, you can stay in the traditional pipeline.
If your long list fails or a user does something unexpected, you can fit the whole interaction into the context of the LLM. The LLM should really be fine-tuned and run with a low or zero temperature, otherwise you will never get these problems under control. Even with fine-tuning, you will still have a certain percentage of failures. Be prepared for that.
To make things more robust, you could add a classifier. This detects whether the LLM gives the user a confirmation, and you can then check whether the appropriate tool was called with a positive result. If not, you can retry or abort.
Most importantly, as LienniTa said: Don't test in production!
EternalNY1@reddit
Use structured outputs and JSON:
Structured model outputs
shrug_hellifino@reddit
LLMs sit in multiple tiers of the stack. They can index knowledge, orchestrate workflows, and generate code. That is real. This post is about a different concern. When we discuss what the user experiences at the point of interaction (what most users expect when they ask for AI), I treat the LLM as part of the UI surface.
The takeaway is simple. Treat an LLM UI like any other high-variance input widget with additional contract and control layers. The job is to shape inputs, constrain outputs, and prove the behavior.
Operating model:
Scope. Define the interaction contract. Inputs, allowed intents, output schema, error states. No open fields without rules.
Input controls. Validate, normalize, and enrich before inference. Strong typing, pattern checks, deny-lists, allow-lists, and semantic guards. Prefer deterministic preprocessing to prompt gymnastics.
Output contracts. Enforce schemas with tools like JSON mode plus validators. Reject or auto-repair on contract violations. Never trust free text for system actions.
Safety and policy. Apply PII redaction, role constraints, and policy checks both pre and post inference. Log decisions and reasons.
Grounding. Retrieve only what the task needs. Pin sources. Surface citations to the user if the UX requires trust.
Determinism layer. Keep critical logic out of the model. Use functions, stored procedures, and rules engines for anything that must be exact.
Observability. Capture prompts, inputs, outputs, latencies, and guardrail events. Version everything. Enable replay.
Evaluation. Run offline test suites and online canaries. Measure accuracy, coverage, and contract adherence, not vibes. Fail fast on drift.
Cost and latency. Budget tokens like you budget CPU. Cache aggressively. Bound maximum context.
UX truth. Make the system state explicit. Show what was understood, what was executed, and any constraints applied.
Yes, LLMs do more than UI. They can power agents, analysis pipelines, and developer tooling. Still, when the discussion is about what the user touches, the mental model is “LLM as UI widget with strong contracts.” Same discipline as any other control, with the extra work of validating both sides of the interaction. Build it like a form. Verify it like a service. Operate it like production software.
netnem@reddit
You need to ground it with traditional code and structured output. Using something like pydantic AI (or structured-outputs), create a basemodel that the LLM can ONLY respond with traditional code that includes an additional prompt.
For example:
User: Hello I'd like to make an appointment for tomorrow
First pass LLM call: make sure it can ONLY respond with X possible values in the structured response: "get_appointments_schedule, unknown_question, etc".
you can force it to always include a tool call at first, and then you pass the results of that tool call into a second LLM call to interpret the results, and then only pass the 2nd LLM call back to the user.
Take a look at https://platform.openai.com/docs/guides/structured-outputs
DaniyarQQQ@reddit (OP)
I've looked at PydanticAI and it is too overly engineered. I do use structured inputs for tool usage, and I do provide constrained information. The problem is happening around it. Sometimes it just does not want to call proper tool and wants make an assumption. Sometimes it calls the tools, sends proper input and receives properly composed response, but it interprets it back to user in very bad and deceiving way.
LienniTa@reddit
PydanticAI is one of the simpliest frameworks i used, and i used 9(langgraph, crewai, autogen(new)+ag2(old autogen), agno, agents mcp, llamaindex and pyspur). It lets you make production ready multi agent systems in no time, literally. Its even simplier to use than no-code solutions.
SkyFeistyLlama8@reddit
How good is it with RAG? I've found most of these frameworks are over-engineered and I have to jump through hoops to do simple stuff like query rewriting. For example, I do query rewriting for better vector search results, but I feed the original query and conversation history into the LLM along with the search context.
LienniTa@reddit
there are no advanced RAG stuff built in pydantic_ai, no connection stacks, no reranker, no query transforming, but baseline retriever is not bad at all. I honestly dont think you should solve RAG at framework level, at max, make a separate agent for query transforming using framework, but outsource cross encoder reranker to something like minilm and at least one of the retrieving steps to something ol reliable like turbopuffer or other search. Then again yes there are good RAG integrated frameworks like llamaindex but that one is for sure overenginnered, thats true.
netnem@reddit
Structured outputs, not structured inputs.
jwpbe@reddit
???
Cool-Chemical-5629@reddit
Rather than using LLMs to orchestrate algorithms, use algorithms to orchestrate the LLM.
Let the LLM analyse user's intent, but make it so that the LLM's output is a JSON format with all the data you can work with and use algorithms to parse it. Once you got the input data, make the tool calls on your end in the code and once you get results from that, instruct the LLM to pass that information to the user.
Crafty-Run-6559@reddit
Like others pointed out, why do you even need the LLM then?
Just give the user a few buttons to select, it's basically the same thing isn't it?
You're just classifying their text input to a specific hard coded button/action.
ventomareiro@reddit
The LLM is the user interface.
Randommaggy@reddit
The benefit is that it allows for a softer interface that better adapts to the end user's level of knowledge.
Cool-Chemical-5629@reddit
I see your point, but if the customer wants an LLM, I guess they are going to get it, right?
Teetota@reddit
It was a WTF moment when I was watching Langchain demo video where one of the Langchain creators runs into this sort of problem but downplays it. In that video, the LLM could not update the ticket date and decided to delete the ticket and then try booking a new one. In the videos they published a few months later they gave up on the idea of having LLM decide on tool calling and recommended intent recognition by LLM and hardcoded workflows for everything else.
DaniyarQQQ@reddit (OP)
Oh my god Langchain. Working with that is different kind of agony. Before that you were fighting against LLM, now you are fighting against LLM and Langchain. I've shown very simplified examples, the workflows are more convoluted, and people really hate bots that answers them in pre defined and robotic ways.
Teetota@reddit
Honestly, if I want to see a doctor I'd love to click on a discipline then on doctor's name (or use a search), look at the profile and reviews and get straight to their available slots. Honestly I believe most people could get an entertaining conversation somewhere else. The idea of an AI agent is not about chat. AI agent assumes that LLM can combine and chain tool calls to get a creative and customized solution for your request, but we are not there yet :(
DaniyarQQQ@reddit (OP)
In my country it is very different. Each person has personal relationship at least with some kind of a doctor, and everything happens via messengers.
SkyFeistyLlama8@reddit
Having freeform conversations via Telegram or WhatsApp is precisely the kind of nightmare scenario that will show up LLM weaknesses. It's too unstructured, rambling, with users typing in a bunch of messages at once and hoping for instant replies.
hyperdynesystems@reddit
This is more of a Langchain problem than an LLM problem. Langchain isn't regarded very highly among people who are at least moderately deep into the LLM space.
Personally my thoughts about LLMs is they are good for the main thing they operate on: generating and understanding text.
For everything else, use regular programming.
AICatgirls@reddit
That all sounds really frustrating. I tried to make a chatbot think once, only for it to decide it would rather talk with its own thoughts than with the user.
Have you tried adding more examples of making the tool API calls in the context? Or using a LoRA to reinforce what you're going for?
MikeHuntTifArt@reddit
Did you just throw LoRA cause you ran out of other abbreviations?
AICatgirls@reddit
They're a way to add additional training to a model. Oobabooga's text generation webui has a tab you can use to train a LoRA if you're interested in learning more.
Barafu@reddit
Sounds like you somehow messed up the mandatory formatting for a thinking model.
DaniyarQQQ@reddit (OP)
I've been using ChatGPT and Claude for these tasks, added everything. It is unending game of trying to tame these llms.
segmond@reddit
skill issue, this is basic agent 101.
appakaradi@reddit
Have a judging LLM judge the work of the regular LLM. It will catch 99% of the issues and fix them.
appakaradi@reddit
Great write up and thanks for sharing.
FreeUsePolyDaddy@reddit
As crazy as this will sound, I would bet money that LLM had part of its training happen on a Saturday and it caused it to lock in a specific day-of-week awareness. You see this in DeepSeek where it thinks the current date is a training artifact, it has no idea what the current date is.
LienniTa@reddit
cant believe i will be the first person to mention evals - scrolled till the end to upvote but nop
brotherman dude! without evals you are literally blind!
put your examples into evaluation pipeline, detect tool chain that agent tries to produce( i use placeholders to not actually call them) and if agent produces no tool calls.... its a red square on your evals heatmap. This will be your starting point.
Change prompt, tool descriptions, tool call confirmations/denials until it stops being red. Ask eval team to make harder evals, get new reds, work on them while keeping old one as sanity check. Thats how you get working system, by verifying every change with autotests.
Cull queries that are always green. Ask the most denegerade assholes to help your evals team break your agent with the most horrendous jailbreaks, adversarial prompting, ambiguos queries and such so that you have room to improve. If on asshole evals you get 60% f1 score, in production it will be 90+ xD
thanks for sharing your experience, sad you didnt have a researcher in team who is used to this type of autotesting - now you will become one :3
Naernoo@reddit
Your use case is way to handle with classic programming and state machines. AI is an overkill for that. Sorry but you went the wrong path
Majinsei@reddit
This sounds more like a graph in langgraph~
You implement well in the conversation graph, and several additional steps in each message with several cheaper models~
This is a problem of waiting to execute several steps and actions in a single call~
radarsat1@reddit
I would love to read a full article on how to test for and overcome these types of issues, especially if it's grounded in real experience like this. Way too hard to find real world experience reports for this kind of thing.
BumbleSlob@reddit
You need to add a critic into your workflow which analyzes the conversation to catch the bot doing stupid things you don’t like like ignoring tool calling.
no_witty_username@reddit
Big agree here. For my agent I have a lot of subagents that watch over the main agent and relay their condensed thoughts back to the main agent before any decision. This broadens the scope of the main agents attention and helps him make more informed decisions.
x0xxin@reddit
What temperature are you using? Curious if you could get better results at 0 temp.
no_witty_username@reddit
This is quite serendipitous as I am building my own agent from scratch now and have been learning a lot about how LLM's process information in the process as I've found your exact problems. Its really funny. But I am beginning to build a decent understanding and in my mental model I now see LLM's as "horses" with their "blinkers" restricting their field of view. Its always good to assume that the model is stupid, has lazer focus on a very narrow field of view and is always assuming things. With that mental model ist become easier for me to troubleshoot their issues as now I am building various dynamic subsystems, scripts and other solutions that help steer the agent towards the right path. Things that help are rigorously tested system prompts, helper sub-agents, ground truth data, detailed meta cognitive metadata on all the capabilities of the system and its shortcomings, verification systems, etc.... as you layer on more and more of these the system becomes more stable but its hard work. it requires a lot of theory of mind on your part to understand why the damned thing is doing what its doing. At first its baffling but in time you become a LLM whisperer of sorts, you grasp their shortcomings easier and find the solutions for it faster. Also some other suggestions for you. Give very explicit concrete examples in its system prompt on exactly how every call should be performed and also the style of its thinking traces. That can further reinforce proper behavior as llms tend to stick in same manner as theve responded before. Theres also other techniques like attention prepend which literally forces the model to start the conversation in a specific manner, etc... anyways i could talk for ages about the many interesting things and techniques. Though I am not doing this for a job so I am having a lot of fun learning all this and researching the quirks.
Leopold_Boom@reddit
For stuff like this do not give LLMs full context. Your goal is to make each step more like a deterministic function call, not like some organic experience.
If you know you are in a booking appt flow, don't show the full chat context to the LLM, just give it the prompt and the info from the tool call and give it a clear prompt to find time slots meeting a certain pattern. Have the LLM generate the pattern in a previous step.
Basically the more you can encapsulate and treat LLM calls as deterministic function calls, with the LLM orchistrating which deterministic functions to use, the less dumb variability you'll get.
It's stupid to allow the full variety of possible outcomes / responses if you know that all booking requests are 1). we found a time that aligns with your needs and the schedule 2). We did not and here are some compromise options 3). You have a serious complex issue, and should be escalated.
Fun-Wallaby9367@reddit
Very very important post, thanks for sharing. Remembered me of Gemini using Google old gem ai sdk google had to beg him in the prompt not to use it.
I have two suggestions:
false79@reddit
Super interesting post. It does look like whack-a-mole but every disappointed users along with the correct resolution looks to be fine tune data gold that no other LLM has.
Must of the LLMs are based on text found througout the interenet and these types of interactions exists outside of the training data.
DaniyarQQQ@reddit (OP)
It is a whack-a-mole. I think the main issue is that these models needs to improve their tool intelligence. They are overly positive and try to make an assumptions before they even call proper tools.
Maximum-Wishbone5616@reddit
No you simply cannot replace alghorithms with statisticall models. It is great for integration for certain things but if you rely on the prompt then do not expect to see anywhere close to even awful 80% reliability. You can drop the temp but still hard to expect to see 100 same results on same calls if you only use AI.
Maximum-Wishbone5616@reddit
Nope just tree decision, backend interprets what should be checked and then returns available slots. Then AI generates nicely looking response.
fuutott@reddit
What I did was I preprocess user prompt for known entities. Names emails dates phone numers. Then do a database lookup with that and return results that I would like to have known if I was an llm. So in your case recognise doctor name. Return usual schedule and next available appointment. The pass this on with user prompt. Mistral with a good prompt and few examples is really good at extracting entities if you don't want to train your own model
I-cant_even@reddit
So.... How much finetuning have you done for your use case?
hiper2d@reddit
Nice reading and experience, thanks for sharing that. I have a few projects with AI, but my use cases don't require tools; lucky me I guess.
It looks like the models you use are not so good with tooling. I had a bad experience with the previous generation of Haiku. I have a game where AIs have to play, and I put rules and lots of in-game information into system prompts. Haiku was bad at following these prompts and hallucinated badly, so I had to exclude this model. Sonnet was good. The Haiku 4.5 is weird; I cannot test coding in Roo Code or Cline because it simply doesn't work there. But this is probably because of some mismatch in the function definition format. Still, larger models can deal with this. Have you tried the same use cases on Sonnet 4.5? It's a coding model; it follows prompts with functions quite well. Try gpt5-codex for the same reason - maybe it is trained to use tools excessively.
It
redballooon@reddit
Up until recently I was test manager for a very similar project. You lined out some of the reasons we didn’t move onward from gpt-4.1.
Your examples look like you mix normal text with tool calls. Have you tried to forbid it using normal text and only allow tool calls? One of them could be ‘output()’ or so where it can put in text that’s forwarded to the user. Then you can control the order in which tools are called much more precisely.
Of course then you won’t be able to use ChatGPT but will have to implement a front end yourself.
ataylorm@reddit
Ah welcome to the newbie club. You will learn you need to structure this much more clearly.
Call 1: is this user requesting any of these, if so which one?
Call 2: user message with tool data already attached.
Or you can try using higher levels of thinking, as they will often help.
Also try things like “Never respond about scheduling questions unless you have called the scheduling tool to confirm.” But again this will work best with higher thinking levels.
DaniyarQQQ@reddit (OP)
Did all that. Examples are simplified. Still shits themselves.
zhambe@reddit
Great to read of a real experience trying to productize this tech. Just spitballing here, but I don't know whether letting the LLM manage the workflow is the right approach. I get the appeal, but it's too easy for it to immediately go sideways. Maybe with a specifically finetuned model? I don't know enough about that to say.
I would constrain the workflow: define your use cases, and implement that in code. This part you want to be deterministic.
The chat interface is still useful - for transforming whatever way people phrase their requests, into choice of and input into your use case funnels. Similarly with results - you can leverage the LLM to write "human" responses from ugly and terse programmatic results.
ambassadortim@reddit
Maybe look at n8n
wichwigga@reddit
quakquakquak@reddit
Yeah, I think this is the reality of actually using them seriously. It will take some time for the hype to die down though, since now it's trillions riding on it. I've seen very similar things and unlike software types, users wouldn't know to ask it or even see the missing tool call to ask, "did you really do that?"
DeepInEvil@reddit
Welcome to reality
Kimononono@reddit
Stop giving the llm agency to decide the decision tree. Have it act purely as a translator and classifier on a decision tree. Don’t have the Chat UI as the only method to schedule bookings. You should expect the llm to fail and treat it only as a nice-to-have. The more agency you give an llm, the more agency you give it to mess up.
Rich_Artist_8327@reddit
First of all the whole idea is ridicilous and super dangerous. This kind of health related personal data should not in any case be given to any AI API. This is absolutely mind blowing that simone even tries to do this kind of absolutely crazy shit. If this is done, first of all it should be done with local models so that the user given data is not sent anywhere. Oh jisus I am done bye.
Mystical_Whoosing@reddit
Nice example of testing in production.
Azuriteh@reddit
You could likely train a SLM that's good at tool calling like Qwen 4b with a QLoRa using unsloth and achieve really good results, but you'd have to build the dataset yourself (or use a bigger AI to generate it lol). The problem is that these model even though they're "smart" are mostly trained towards coding tool calling and try to generalize rather than specialize in your niche tool call scenario.
truth_is_power@reddit
Fyi I have done some work with tool calls;
You need the right model that's tuned specifically for tool calls in my experience.
Granite3.2 vision really did it for me.
Llama3.1 works too.
This project uses tool calls so the user can set their stats, class and upload a picture and it generates a character that uses it all combined.
https://x.com/CarltonMackall/status/1970264236505845971?s=20
ive been living in hotels but ill move into a place on monday...finally be able to start coding again D:
DaniyarQQQ@reddit (OP)
How good they are at non English texts? People in my country write in their native language mixed with a Russian.
truth_is_power@reddit
Granite is specifically english.
You might want to translate to english/chinese first before sending it to tool use - since most models (in my ignorance) are probably focused towards those two languages.
that could explain some of your issues as well.
2BucChuck@reddit
Thanks for posting real issues - so tired of the Ai slop promotion crap. Great examples everyone struggles with
ParthProLegend@reddit
Best post. I loved it.
AfterAte@reddit
Claude and ChatGPT5 are retail products that the companies use to entertain people (eg: r/MyBoyfriendIsAI, ChatGPT will have NSFW content soon, Claude loves saying "You are absolutely right!" and too encouraging), and be sycophantic to keep them coming back more and more and hopefully keep up with their subscription.
I feel like Qwen3 Coder 30B A3B is trained to be honest. It'll say it doesn't know. It's quite unsure of itself. Try a local model not made to be someone's loving boyfriend.
m1tm0@reddit
What's your system prompt? System prompts for reliable ai tools like claude code are gigantic and detailed.