Completely novel to me: im-also-a-good-gpt2-chatbot on LMSYS Arena using codeblock to draw Diagrams to supplement its explanations

By compressing the idea of RLHF to single diagram, a lot of information is lost and it gets confusing, somewhat inaccurate. The lineage from from initial model (actually one after SFT training already, but this information was lost in the diagram to make it smaller) through "Generate Response" > "Human Evaluator" > "Reward Model" is fine. It does get you a reward model. But what's happening with the branching here? A finetune is based on "Generate Responses" and "Human Evaluator" combined?? Why is it branching off before it reaches "Reward Model"? That doesn't really allow for coherent understanding of the diagram. Is "Human Evaluator" needed for every step of the training even if we assume that branching happens from "Reward Model" and not from space between it and "Human Evaluator"? Well it is in the lineage for every step, so you might assume so based on a graph. [Here's](https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2023/08/31/ML-14874_image001.jpg) an example of a diagram that actually explains it in a great way. The best single loop diagram I found is from [wikipedia](https://upload.wikimedia.org/wikipedia/commons/b/b2/RLHF_diagram.svg), but it is way harder to read than the one from AWS.