Lesson 5: Getting data - Click through links in a list and capture data from each item page

April 03, 2020 07:14
Updated

By now I believe you’ve already mastered extracting simple text and text contained in a list . Next, I'll show you how these techniques can be used together for capturing information from detail pages when you need to click through links contained in a list. This is especially useful when extracting from E-commerce sites such (Amazon, eBay, etc) and business directories (Yelp, Yellowpage, etc).

web scraping with octoparse - extract from item page

Let’s see how it is done with an example. We will use URL: https://www.ebay.com/sch/Vehicle-Electronics-GPS-/3270/i.html for the example below. [Download the task file in this lesson]

1) Click through the links in a list

To do this, we will create a "Loop Item" to loop click each product link on the search result page.

Click on the first product title that contains the URL to access the product page. The selected title will be highlighted in green while all the other similar product titles will be highlighted in red.
Click on the second product title containing the URL
Select "Loop click each URL" from "Action Tips". Notice a Loop-click step is being auto-generated and added to the workflow.

Tips!

To loop click-through all links on the list, it is important that you select the anchor texts. Octoparse automatically identifies tags of selected items. So when you select an item with URL, the selected tag would be "A", which stands for anchor that usually links one page to another.

2) Capture the information you need from the product page

Once the "Loop Item" is created, Octoparse goes on to click the first link in the list and load the first product page in the built-in browser.

Now, click to capture the specific data fields you need - this will be used as a template for all the other product pages.

Click on target data fields such as title, review, price, etc.
Select "Extract data" from "Action Tips" to complete the extraction action. Notice an "Extract data" step gets auto-generated and added to the workflow. Data extracted will be shown in the "Data field" pane next to the workflow designer.

Tips!

Set up a wait time in "Advanced Options" for steps like "Click Item" or "Extract Data" can effectively avoid data skip and make the crawling process more human-like! (Usually 2-5 seconds would work well).