Carbon: Decoding the Language of Life

Posted by loubnabnl@reddit | LocalLLaMA | View on Reddit | 19 comments

Hey, it's loubna from Hugging Face. Very happy to share our latest release: Carbon 🧬, a family of open DNA foundation models. Carbon-3B matches the current SOTA (Evo2-7B) while being 275x faster.

We borrowed a lot from how modern LLMs are trained and from our SmolLM work, but DNA isn't language. Genomes are noisy, redundant, and shaped by evolution rather than communication. So we adjusted the recipe:

Tokenizer. Most genomic models tokenize at the nucleotide level, which blows up sequence length. BPE is the obvious LLM-style fix, but it doesn't behave well on DNA. We use deterministic 6-mer tokens (one token = 6 nucleotides): 6× shorter sequences and cheaper attention.

Training loss. With 6-mer tokens, cross-entropy scores a prediction that gets 5 of 6 nucleotides right the same as one that's completely wrong. This gets brittle late in training and produces loss spikes. We switch mid-training to a more flexible factorized loss (FNS).

Data. Genomes are mostly sparse, repetitive background. We curate down to a staged functional DNA + mRNA mixture, with every ratio chosen by ablation. Like mixing a web corpus, but for biology.

- Technical report: https://github.com/huggingface/carbon/blob/main/tech-report.pdf
- Demo (with a biology primer for our ML friends): https://huggingface.co/spaces/HuggingFaceBio/carbon-demo

Happy to answer questions in the comments 🤗

[-]

slavetothesound@reddit

if I have a sequence of my DNA, can I ask this questions about it? what diseases do i have, etc? Can I buy a crisper thing and use this to design some fixes without getting a biology degree?

Ylsid@reddit

If I told it to generate a pig with wings could it do it

charmander_cha@reddit

Soube deste modelo estes dias, eu não entendo desta tecnologia, só queria saber se está tecnologia se envolve de alguma forma com a sua tecnologia, não sobre o que elas fazem mas sobre como cada abordagem entende uma LLM e o que ela de fato faz com representações textuais de informação

AppealThink1733@reddit

Is it good for use in CrispCas9?

mouseofcatofschrodi@reddit

When can we do genetic tests at home locally, without sending the most private data that exists into a company?

r_kdethrowaway1337@reddit

https://genomes.io/

They claim to use blockchain/encryption/etc. to do it in a fully privacy preserving manner. The lab doesn't know whos sample they're sequencing; they store the data on their server for you to download, but it's encrypted with a key that only you have, so they can't access your data. Or so they claim. 400 GBP = $540 USD

This is the closest you can get without spending $4k+ to do it at home like the other commenter linked.

Also, not to be too much of a "well akshually" Redditor, but your DNA is some of the least private data you have. Your body is shedding it all over the place 24/7/365. Your relatives have practically identical DNA to you, so if someone has access to their data they pretty much have access to yours too.

fantum13@reddit

https://iwantosequencemygenomeathome.com/

lewtun@reddit

Well, this model runs on a laptop so at least that part of the problem is solved :)

Inevitable_Ear132@reddit

The FNS switch is the interesting part. Cross-entropy treating 5/6 nucleotides right the same as 0/6 right always felt off for genomics, where one SNP can flip pathogenicity but most positions are silent or redundant. Did the late-training loss spikes correlate with high-conservation regions, or were they roughly uniform across the corpus?

Alarming-Ad8154@reddit

Not sure this is the place for technical questions, why not 3-mer encoding and encoding the “genetic code” table so the model could learn proteins and protein structure as well? You could then probably even train on protein data…

The choice of a 6-mer was informed by the preceding work of the GENERator models, which examined the impact of various k-mer choices on sequence recovery (i.e. generate b base pairs and compute token-level accuracy) and found k=6 was best https://arxiv.org/abs/2502.07272

PaceZealousideal6091@reddit

Well, this is interesting. His question is still valid. Looking at the data , there isn't much of a difference between 3mer and 6mer. Can you tell me what exactly is in the y-axis. If the difference isn't too drastic, wouldn't it make more sense to keep it at 3mer and be able to train it for amino acid sequences and prediction as well?

Thin_Pollution8843@reddit

That’s cool. But I have a question how to use it for a regular person? How it can help me as example? I have full sequence of my genome.

-dysangel-@reddit

I wish I knew enough about what you're saying to be able to ask questions..!

Carbon-3B matches the current SOTA (Evo2-7B) while being 275x faster.

Incredible work - congrats!

Thank you!

This is very cool. I have long been thinking of playing with DNA model. I just didn't think it's matured enough. You guys might just push me over. Am I right in assuming that it can technically be converted and quantized to ggufs and run on llama.cpp? I'll just need to make sure parse my FASTA sequences and generate the raw integer Token IDs using python. I should be able to feed them to the the model. Right?

Yes you can totally convert to GGUF format and run it with llama.cpp! The only subtlety is that Carbon has a custom tokenizer and that needs to be upstreamed to llama.cpp in order to work out of the box (I'm taking a look at that)

svpaub@reddit

This is really cool, to me it feels like this is first DNA LLM that makes proper design decisions based on the specifics of genomes. It indeed never made sense to me to use BPE, like DNABERT and others did. Your dataset does seem really focussed, is there maybe not too much bias towards known/predicted genes? The rest of the genome is not completely random/useless.