So, as someone who works professionally in the ML infrastructure space, I do think it's possible - but it has been attempted many times before, with generally disappointing results.

The main problems are:

1. Non-homogeneous hardware. Say your GPU is from the Turing generation, and another person's GPU is from the Blackwell generation. Turing doesn't support many of the optimizations that Blackwell has - so do you turn off those instructions for the whole group, or do you throw the older models out of the pool? This is a broad theme with distributed consumer hardware, and extends to not just instruction sets, pytorch versions, etc, but also things like sharding strategies. If you put some layers on a really fast GPU, and other layers on a really slow GPU, the slow GPU becomes the bottleneck. It requires some very creative approaches to find a solution that doesn't end up just slowing everything down to the slowest potato in the node group.

2. Bandwidth, latency, and generally shit internet. You can't beat general relativity, and consumer networks don't have anywhere close to the kind of bandwidth that typical training networks have. They're also flaky, which is another factor that can cause whole node groups to stall until recovered.

3. Data, privacy, and bad actors. Who chooses what training data goes into the training set, and how do you enforce it? What happens if some internet troll decides to ignore the rules and start poisoning the training set or just giving junk gradients to try to stall training or produce unwanted behavior? If data is crowdsourced, how do you make sure people don't accidentally upload sensitive data? How do you keep the training set balanced? What kind of model quality can you expect when you have a model trained on 30% code, 10% 4chan troll data, and 60% anime waifu smut? How do you actually get a coherent ML model out of that instead of just recreating Arch linux?

To be clear, I don't think it's impossible. I've thought about it a fair bit even, and think that there's a good chance that the technical issues could likely be mitigated enough to be practically useful (though horribly inefficient) with a couple novel training techniques.

But, well, there's a reason nobody has ever gotten it to work. It is genuinely a highly complex and challenging problem.