Articles in this section

See more

Click each link in a list and scrape data from new pages

  • Updated
Follow
Clicking each link in a list and scraping data from a new page is a common scenario in web scraping. This tutorial will show you how to click through a listing page to a detail page for getting the data you need. This is especially useful when extracting from E-commerce sites (Amazon, eBay, etc) and business directories (Yelp, Yellowpage, etc).

web scraping with octoparse - extract from item page

Let’s see how it is done with an example. We will use URL: https://www.ebay.com/sch/Vehicle-Electronics-GPS-/3270/i.html for the example below.

1. Use "Auto-detect" to set up the workflow

2. Set up the workflow manually

 

 

1. Use "Auto-detect" to set up the workflow

1) Once you've created a new task using the example URL, select "Auto-detect web page data". Octoparse will now detect any data on the page and we can click "Create workflow" to generate the workflow.

1.gif

 

Tips!

You can switch the detected results to locate the elements you want if you find Octoparse does not select the correct data. And if all the detected results do not work for you, please refer to 

2. Set up the workflow manually

switch.png

 

3) Select "Click on link(s) to scrape the linked page(s)", choose "Click on an extracted data fields" and select a data field(here we select the Title_URL) from the dropdown menu.

2.gif

 

Or you can choose "Click a web link on the web page " and select a link on the page manually.

2020-07-28_19-15-08.png

Note that you can only select a link from the detected sections.

 

4) Auto-detect the web data again or click on target data fields such as title, review, price, etc. to scrape them

2020-07-28_19-31-33.png

 

 

2. Set up the workflow manually

1) Click on the first product title that contains the product page URL.

The selected title will be highlighted in green while all the other similar product titles will be highlighted in red.

2020-07-28_19-44-59.png

2) Click "Select all" on Tips panel

2020-07-28_19-47-52.png

 

Tips!

If there is no "Select all" option on the Tips panel after you select the first URL, please continue to select the second URL.

 

3) Select "Loop click each URL" from the Tips panel. Notice a Loop-click step is being auto-generated and added to the workflow. 

2020-07-28_19-48-53.png

 

Tips!

To loop click-through all links on the list, it is important that you select the anchor element. Octoparse automatically identifies tags of selected items. So when you select an item with URL, the selected tag would be "A", which stands for an anchor that usually links one page to another.

If you find Octoparse does not locate the A tag, you can click the "A" on the Tips panel.

2020-07-28_19-50-50.png



4) Click on target data fields such as title, review, price, etc. to scrape them

2020-07-28_19-31-33.png

 

Tips!

Set up a wait time in "Advanced Options" for steps like "Click Item" or "Extract Data" can effectively avoid data skip and make the crawling process more human-like. (Usually, 2-5 seconds would work well). 

mceclip1.png

 

If you have any questions, you're welcome to submit a ticket to our Support team.

 

Author: Fergus

Editor: Yina

Related articles