I am building a customized crawler to regularly scrape restaurant data from Deliveroo.
The odd thing is that without making any changes to the crawlers, it sometimes records data from 26 restaurants and sometimes from more than 1000 restaurants (for the same webpage).
The "Go to Webpage" is instructed to scroll-down 300 times with a 1s wait time when it opens, in a way to make sure all the data is loaded.
When trying to debug the issue, I realized that when only partial data is extracted, Octoparse had trouble scrolling down the webpage after opening it, either by not scrolling down at all, or by scrolling down less than instructed.
Why does the scroll-down function not always behave in the same manner? And how can I get around this issue?
I'm a big fan of Octoparse, I have been testing it and it can be pretty powerful, currently contemplating subscribing if it turns out it can serve my purposes.
Thank you all!
Please sign in to leave a comment.