Obliterators and other Thinking Machine Empire tech is more in line with threats the Federation takes seriously, yes. It's not too far out of their reach, though. The Genesis Device had a similar output, the Planet Killer and Red Matter (as much as I hate that one) were more powerful, and technically a capital ship of any Federation or peer is supposed to be capable of the same level of destruction (according to Kirk and Picard at different points), though we never actually see that happen even in situations where it makes sense for an alien to do it. The real problem with the Thinking Machine Empire is the sheer scale of it. We're not given exact details, but the intentionally mysterious phrasing and some of the implications are very similar to the thinking machines we see jump to the defense of the Soong type androids in Picard. And, to be honest, if I were going to make a crossover point, that would be it.

Worms don't transfer between planets easily. They tried that in Dune and it only succeeded after Leto II manipulated things, and even then it was limited. It's also fairly easy to manage the sandworm territorial expansion with water management, something the Federation has the technology for (see the DS9 episode "Let He Who is Without Sin" for an example.)

Picard individually is almost certain to fall prey to the Voice since he likes to go and talk to everyone like a diplomat, but we've seen more effective non-technological weapons in Star Trek so that's more mixed. I could see it probably carrying through the shields of the brig, so capturing a Voice user would probably lead to a compromised ship or two, which might help with your second point of their patience. Capture a ship, then go dark for a while until you've reverse engineered it and improved on it, capture another ship to see what you've missed while going dark, etc. This is sort of a longer scale version of what the Romulan Empire did, and the Romulans had far fewer resources than the Dune powers. That said, they would need a charismatic leader pushing for that, so it really just depends on who is in charge of the raiding party. If it's someone in the Atreides line with futuresight abilities, they probably won't see Starfleet's future but they would see their own future quietly building off the technology. The main downside to this that I see is Starfleet tech heavily relies on technology they might see as "thinking machine" tech and thus reject.

It also depends a lot on why they would fight. The first engagement is most likely going to be whatever human empire is in charge of the Dune side at the time deciding to expand into Starfleet's territory and they would just flat lose. But after losing, the question is if they would prepare for a long-term hostility or negotiate. By the tie of the Honored Matres invasion, things had gotten a lot more diverse and futuresight was less useful, but there is room for actions to arise who won't negotiate and will go off and plan like you suggested. And at that point they were largely unaware that the machines were still a looming threat, so they might see Starfleet as the new great threat and do what the Romulans did.