How to handle OTP-based interruptions in scraping workflows?
Posted by Bitter-Tax1483@reddit | LocalLLaMA | View on Reddit | 2 comments
In an LLM-driven web scraping pipeline (using tools like agents or VLMs), how do you handle OTP-based verification systems that repeatedly interrupt automation?
The platform only supports OTP authentication (no email/login/signup alternatives), and frequent OTP prompts are breaking the scraping flow.
What are practical ways to deal with this kind of constraint in an automated or semi-automated setup?
More-Flight-7741@reddit
otp prompts are a real pain for automated scraping, they basically break the whole flow. i had a similar issue with a data collection project last year.
we ended up using Qoest Proxy for the residential ips and pairing it with an anti detect browser. the sticky sessions and realistic ip rotation helped a ton, it just made the traffic look way more human. that setup got us past most of those verification walls.
NoFaithlessness951@reddit
https://www.kernel.sh/docs/auth/overview
https://www.kernel.sh/docs/auth/faq#external-actions-that-prevent-auto-reauth
If you need things like email based otp instead of totp, you can also handle it automatically just a little bit more work of coordinating when the email arrives and then inputting the otp from that.