Reliable ways to get structured output from llms
Posted by amit13k@reddit | LocalLLaMA | View on Reddit | 7 comments
What are the current best ways to get structured output from local llm/openai reliably ? I have found the following options and tried some of them,
Microsoft guidance - https://github.com/guidance-ai/guidance
LMQL - https://lmql.ai/
llama.cpp grammar - https://github.com/ggerganov/llama.cpp/discussions/2494
langchain https://python.langchain.com/docs/modules/model_io/output_parsers/
jsonformer - https://github.com/1rgs/jsonformer
salute - https://github.com/LevanKvirkvelia/salute
outlines - https://github.com/outlines-dev/outlines
Was looking for something that could work with both local llm’s (gguf/gptq models) and openai but i guess that’s difficult right now ? (also, i am more inclined towards typescript based solutions(zod) if possible)
I ran into a few problems for eg, guidance-ai doesn’t seem to work with text-generation ui because the openai api adapter doesn’t support logit_bias.
It will be great to know the experience of others with these approaches.
Eastern_Ad_6767@reddit
Instructor JS is by far my favorite for Typescript.
https://github.com/instructor-ai/instructor-js
Works with Zod, includes retry mechanics, and more. Great system.
CaptTechno@reddit
which among these would you recommend?
MagentaBadger@reddit
If you care about reliability of the values in your structured outputs, you might be interested in this post which explores the performance of a number of frameworks on a task that involves reasoning and output structuring.
Link: https://www.instill.tech/blog/llm-structured-outputs
Conscious-Eagle-870@reddit
I am new to the field and I have been trying to find ways to extract structured JSON output from LLMs using libraries like Outlines, Guidance, Langchain etc
I couldn't find out a single library that supports direct JSON/Pydantic extraction for my LLM so far!
I am able to do basic completion tasks but not able to call the concerned structure extraction APIs.
Guidance doesnt seem to support JSON extraction on chat models.
And Outlines VLLM API defaults to models from Huggingface and not the locally hosted ones. The OpenAI API of outlines doesnt have json output generation functionality implemented.
https://github.com/guidance-ai/guidance/issues/203
https://outlines-dev.github.io/outlines/reference/models/vllm/#:\~:text=Models%20are%20loaded%20from%20the%20HuggingFace%20hub
I understand there might be a fundamental reason for these libraries not being able to support this, but I am not able to understand it.
Appreciate any help regarding this!! Thank you!
Meeterpoint@reddit
LMQL is really good and very actively maintained. I use both LMQL and grammar but recently I am spending way more time with LMQL. Mainly due to new features being integrated and due to the ability to easily switch out a local model with chatGPT. Grammar is good but if at any point in time you want to apply the grammar to OpenAI models you can’t do that.
So it depends a bit on the use cases. I guess both are great.
Guidance unfortunately seems to be stale for a while… so that’s not an option I would seriously consider anymore.
yahma@reddit
We use LMQL in production. Very happy with the results.
OkChard9101@reddit
May you please tell me how to use & practice LMQL using Gemini model as I don't have a free OpenAI API. Or else if you have any doc with regards to loading transformer model in google collab to practice LMQL