Has anyone tested GenAI vs Advanced NER?
Posted by dalhaze@reddit | LocalLLaMA | View on Reddit | 9 comments
What i’m wondering is — Can i achieve more reliable/consistent entity extraction using NER ML models?
I realize my question shows my ignorance.
I know which one is cheaper, but i’m wondering what 99% accurate entity extraction looks like on moderately complex documents. ie- Associating an Entities as a subset of a group from a chunk of text that contains multiple entities/groups
What would the most robust enormity extraction looks like? Fine tuned GenAI? or other?
VulcanizadorTTL@reddit
i have compared results for NER using BERT variants, LLMs and spaCy. from my results BERT was the best combination of price/quality.
LLM (4o-mini/gemini 1.5 flash) was easier to deal with, structured output + huge input context.
in the end we decided on LLMs because not only the entity but additional information was necessary and NER alone couldn't do it.
brewhouse@reddit
LLM is the way to go these days, simply because how cheap it has become. Especially with batching and prompt caching.
And then it may be worth trying to fine-tune some model once your domain dataset is big enough, but if LLM is already cheap and fast enough then probably better to invest your bandwidth elsewhere.
VulcanizadorTTL@reddit
i agree, but it doesnt apply to all tasks in spanish, for example structured generation and tool calling.
brewhouse@reddit
Yeah, for multilingual and solid structured generation Gemini flash is probably the best choice. The flash 8B one is serviceable with good enough instructions & examples, might be worth trying out.
VulcanizadorTTL@reddit
i love the gemini models. but right now they are not usable in spanish, i wrote an article about it: https://cmidt.org/2024/11/19/el-sesgo-de-gemini/
Describe Black Shoe Say Black in Spanish
brewhouse@reddit
I hear you, it does suffer from the woke mind virus. For me it's still worth it to still use it as primary, since the 8B model is still one of the most cost-effective capable models for multilingual basic tasks (4x cheaper than 4o-mini).
You can set up backup secondary, for example if the api response has finish reason blocked then run the inference using gpt-4o-mini.
VulcanizadorTTL@reddit
that's a really great idea, gonna try it!
LengthinessJumpy8409@reddit
for BERT variants is Spacy still the best? I have heard about Flair as well but they really don't have a good documentation. Am doing NER and looking for the current best library if there are any other then these two
VulcanizadorTTL@reddit
for bert im using a spanish finetune: mrm8488/bert-spanish-cased-finetuned-ner
The huggingface transformers library is the one i use: https://huggingface.co/learn/nlp-course/chapter1/3?fw=pt#named-entity-recognition