Is there a template I can use to scrape a site with a drop down menu + search button?
Hi, I'm tracking protest arrests in my city and I'm hoping there is a template that might be close enough to what I need.
I want to leave the name fields blank and search arrests by "Booked Today" and then "Booked Yesterday." Those might need to be two separate steps or functions.
Once search filter is selected (let's say we start with "Booked Yesterday"), the site returns a list of name URLs to each person's booking page. I need to scrape all that data (including bail amounts, etc) for every person on the list, then return to the main page and run the "Booked Today" search. It doesn't matter to me if all the name URLs are clicked simultaneously or if Octoparse loops through the list until all items are clicked and logged.
For example, I'd like the process to look sort of like this:
(select "Booked Today" from drop down) -->click "Search" button-->click name URLs-->scrape all data-->return to homepage
-->
(select "Booked Yesterday" from drop down)-->click "Search" button-->click name URLs-->scrape all data
Does anyone know of a template that would be applicable?
-
Hi rocktopus,
Thank you for reaching out.
To scrape this website, you'll need to configure a workflow, first, you can follow this tutorial to select a specific item from the drop-down menu:
After arriving at the result list page, you can utilize the auto-detection function with version8 to generate the workflow, to learn how, please refer to this tutorial:
Best regards,
Please sign in to leave a comment.
Comments
1 comment