should you learn web scraping?
Posted by OneNiceGuy124@reddit | learnprogramming | View on Reddit | 11 comments
I'm not really too familiar with this, so I might have no idea what I'm talking about, but I want to make a website where that showcases movies and their ratings and what websites they are available on and stuff just because I need something to do while I wait until Monday for a client. If I put on my resume for a job that I made a website like this and then I also say that I just used 3 API's, are people going to be unimpressed? Should I learn how to do it with things like web scraping and all that difficult stuff? I am very good with Python, if that helps; I just started to learn web development though.
promptcloud@reddit
Absolutely! Learning web scraping can be super useful if you work with data or want to automate repetitive tasks. It helps you extract valuable information from websites, like prices, reviews, or research data, which can save tons of time. That said, it's important to scrape ethically and respect website terms. If you need large-scale or complex data extraction, services like PromptCloud can handle it for you, but having basic knowledge of web scraping is a great skill to have!
promptcloud@reddit
hi
recursion_is_love@reddit
> should you learn web scraping?
Me ?
Yes I should, but I still browsing reddit even I know that I need to start doing it. Don't know why.
OneNiceGuy124@reddit (OP)
Lol
hoolio9393@reddit
For a hobby yes. It is useful to scrape and data hard
rustyseapants@reddit
You didn't look up what web scraping was before posting this?
Alternative_Eye3452@reddit
Learning web scraping could definitely enhance your project and showcase your Python skills even more; consider using FetchFox AI for a smoother experience!
joanthebean@reddit
Do things you care about, not just things that’ll “look good” on a resume. Personally, I enjoyed learning about web scraping because you learn about the DOM, along with how to programmatically deal with varied and occasionally incomplete data
OneNiceGuy124@reddit (OP)
I think it's something I'm interested in, but I wouldn't really want to spend more than 20 or 30 hours learning it because I want to focus on other things. Do you think I'd be able to develop a fair understanding of it with that amount of time, and maybe be able to make that movie site in the end, or should I just stick to apis?
joanthebean@reddit
Honestly, your use case sounds like it’s probably doable in 20-30 hours of learning scraping, at least for an MVP, if you’re confident that you generally know python. With additional time, you’d also learn some more web dev, especially if you try to make a basic Flask or Django app to show your data. And then the data you scrape needs to be stored somewhere. If you have no time, you can toss it in a list/dict/etc and pickle it, or you can pick up some SQLite or Mongo or some database system. I think it’s worth learning as much as you care for, so long as you make some kind of project output (or not, I sometimes learn stuff just to learn without a cohesive project in the end, whoops)
Every-Swimmer458@reddit
Having a project that involves APIs is always good to put on a resume in my opinion. Same for webscraping.
As someone who has built a few web scrapes in Python, you should first determine how many scrapes you plan on doing per week. If you're doing 100 or less, use a simple method like BeautifulSoup and Selenium. If you're scraping by the thousands or millions, use sracpy (scray-pie). Do some research on how to be stealthy, using different browser profiles, randomized timing delays, and even proxies. Don't test more than a handful at once until you are comfortable with what you're doing, as you'll get IP blocked otherwise.