The Synthetic Data Playbook: Generating Trillions of the Finest Tokens

Posted by joelinho95@reddit | LocalLLaMA | View on Reddit | 10 comments

Introducing the Synthetic Data Playbook: We generated over a 1T tokens in 90 experiments with 100k+ GPUh to figure out what makes good synthetic data and how to generate it at scale [https://huggingface.co/spaces/HuggingFaceFW/finephrase](https://huggingface.co/spaces/HuggingFaceFW/finephrase) https://preview.redd.it/hq6abr3p3ung1.png?width=1200&format=png&auto=webp&s=1dd47fa704669648c5fab08b1a02552c0b2fe8ce

Reply to Post

10 Comments

[-]

Due-Cat6317@reddit

Am I the only one seeing this error on the page? 👀

[-]

joelinho95@reddit (OP)

Seems like there were too many requests so we redeployed the space. Should be up again now.

[-]

Middle_Bullfrog_6173@reddit

It worked for a bit, the stopped working. Actual dataset is here: https://huggingface.co/datasets/HuggingFaceFW/finephrase

[-]

Due-Cat6317@reddit

thnx for sharing the dataset link. I thought they were going to share the playbook for synthetic data though.

[-]

Middle_Bullfrog_6173@reddit

The research and "playbook" is in the currently broken page.

[-]

ClimateBoss@reddit

is this a joke? They wrote 4 prompts and called it gamechanging research? Rewrite clear, step-by-step tutorial or instructional guide. Use numbered steps or bullet points where appropriate to enhance clarity. Preserve all essential information while ensuring the style feels didactic and easy to follow. Output only the tutorial, nothing else.Rewrite the document as a clear, step-by-step tutorial or instructional guide. Use numbered steps or bullet points where appropriate to enhance clarity. Preserve all essential information while ensuring the style feels didactic and easy to follow. Output only the tutorial, nothing else.

[-]

Middle_Bullfrog_6173@reddit

That's the final dataset. The research is in the OP linked space which is still broken. I managed to read through most of it and it seemed really thorough. They tested different organic datasets as sources, prompts from other synthetic datasets, different models etc. These four are the ones that worked best.

[-]

The Synthetic Data Playbook: Generating Trillions of the Finest Tokens

Reply to Post

10 Comments

Due-Cat6317@reddit

joelinho95@reddit (OP)

Middle_Bullfrog_6173@reddit

Due-Cat6317@reddit

Middle_Bullfrog_6173@reddit

ClimateBoss@reddit

Middle_Bullfrog_6173@reddit

Xamanthas@reddit

Expensive-Paint-9490@reddit

Xamanthas@reddit