Click each link in a list and scrape data from new pages
Follow
You may need this link to follow through:
https://www.ebay.com/b/Car-Audio-Amplifiers/18795/bn_887008
1. Use "Auto-detect" to set up the workflow
2. Set up the workflow manually
1. Use "Auto-detect" to set up the workflow
1) Once you've created a new task using the example URL, select "Auto-detect web page data". Octoparse will now detect any data on the page and you can click "Create workflow" to generate the workflow.
2) Select "Click on link(s) to scrape the linked page(s)" on the Tips panel and choose an option from the dropdown menu. Here you can choose "Title_URL".
Octoparse will now take you to the detail page of the first product.
3) Auto-detect the web data again or click on target data fields such as title, condition, price, etc. to scrape them
2. Set up the workflow manually
1) Click on the first product title that contains the product page URL.
The selected title will be highlighted in green while all the other similar product titles will be highlighted in red.
2) Click "Select all" on the Tips panel
|
Tips! If there is no "Select all" option on the Tips panel after you select the first URL, please continue to select the second URL. |
3) Select "Loop click each element" or "Loop click each URL" from the Tips panel. Notice a Loop-click step is being auto-generated and added to the workflow.
|
Tips! To loop click-through all links on the list, it is important that you select the anchor element. Octoparse automatically identifies tags of selected items. So when you select an item with a URL, the selected tag would be "A", which stands for an anchor that usually links one page to another. If you find Octoparse does not locate the A tag, you can click the "A" on the Tips panel.
|
4) Click on target data fields such as title, review, price, etc. to scrape them
|
Tips! Setting up a wait time in "Options" for steps like "Click Item" or "Extract Data" can effectively avoid data skipping and make the crawling process more human-like. (Usually, 2-5 seconds would work well). Then click "Apply" to confirm.
|
If you have any questions, you're welcome to submit a ticket to our Support team.
Author: Joy
Editor: Yina
