Scraping product information from Target.com

Target.com, one of the largest online retailers in the United States. In this tutorial, we are going to show you how to scrape the product information from Target.com. 

If you would like to know how to build the task from scratch, you may read the following tutorial. 

Because of the website structure, we need to use 2 tasks to achieve the goal. We will scrape the URL of each item page in Task 1, and then scrape the detailed product information, such as the product title, price, and description from the product detail page in Task 2. By splitting one task into two, the extraction speed could be improved to a certain degree, especially when we use the Octoparse Cloud Extraction.

To follow through, you may want to use this URL in the tutorial:

https://www.target.com/c/milk-substitutes-dairy-grocery/-/N-5xszh?lnk=MilkMilkSubstit

 

This tutorial will also cover:

• Deal with AJAX for pagination

 

Here are the main steps in this tutorial:  [Download task file here ]

Task 1: Extract all the URLs of detail pages on the search result pages

1. "Go to Web Page" - open the target web page
2. Create a pagination loop - scrape all the results from multiple search results pages
3. Build a "Loop Item"- loop extract each URL on the search results pages
4. Start extraction - run the task and get data

Task 2: Collect the product information from scraped URLs

1. Input a batch of the scraped URLs - loop open the detail pages
2. Extract data - select the data for extraction
3. Start extraction - run the task and get data

 

 

 

 

Task 1: Extract all the URLs of detail pages on the search result pages

• Click "+ Task" to start a new task with Advanced Mode

Advanced Mode is a highly flexible and powerful web scraping mode. For people who want to scrape from websites with complex structures, like Target.com, we strongly recommend Advanced Mode to start your data extraction project.

• Paste the URL into the "Website" box and click "Save URL" to move on

 

• Scroll down and click the ">" button
• Click "Loop click the selected link" on the "Action Tips" panel

• Set up AJAX Load for the "Click to paginate" action

Target.com applies the AJAX technique to the pagination button. Therefore, we need to set up AJAX Load for the "Click to paginate" action.

• Uncheck the box for "Retry when page remains unchanged (use discreetly for AJAX loading)"

When you have set up the AJAX load, remember to uncheck "Auto Retry".

• Check the box for "Load the page with AJAX" and set up AJAX Timeout as "3" seconds

You can set up a longer timeout to make sure the page loads well.

• Click "OK" to save

Tips! If you want to learn more about AJAX, here are the related tutorials you might need: Deal with AJAX Video: Octoparse: AJAX 101

 

• Click "Go To Web Page" to go back to the first page and then click the "Pagination" box

When extracting data throughout multiple pages, you should always begin your task building on the first page.

• Select the title of the first item in the list
• Click "Select All"
• Click "Extract the URLs of the selected elements"

 You may notice that not all the items on the list are selected correctly. In this case, we need to revise the default XPath of the Loop Item to locate all the items.

• Select "Loop Item" in the workflow
• Select "Variable list", and then enter the revised XPath below:
  • //li[contains(@class,'h-padding-a-none')]
• Click "OK" to save

You can also add a wait time to this step so that the website will have enough time to load.

• Go to "Wait before execution" and select a time from the drop-down menu based on your Internet condition. For the demonstration, we set "10" seconds.

 In addition, some items in the Loop Item cannot find a corresponding URL. In this case, we need to customize the XPath of the data field.

• Click the  icon, and then select "Customize XPath"
• Enter the revised XPath below to the text box of "Relative XPath":
  • //A[1]
• Click "OK" to save

Tips! 1. "Variable list" is a loop mode in Octoparse. For more about loop modes in Octoparse: 5 Loop Modes in Octoparse  2. If you want to learn more about XPath and how to generate it, here are the related tutorials you might need: Locate elements with XPath  Video: Octoparse: XPath 101

 

• Click "Save"
• Click "Start Extraction" on the upper left side
• Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)

 

Here is the sample output for Task 1.

 

Tips! When the content of the page has already shown out, but it is still loading, you could click the "X" button at the right end of the navigating bar to stop loading.

 

Now you have the data you want. There are some blank fields in the output because there are no rating values for some product items.

A splitting task can help improve the efficiency of data extraction as well as minimize some problems caused by small changes happened in websites.

Tips! By default, if Octoparse cannot find the element of the defined pattern on the page, the field will be left blank. However, Octoparse may fail to find the element of the defined pattern even if the element needed is shown on the website. If you encounter this problem, here is a related tutorial you might need: What to do with those blank fields I got in the extracted result?

 

Happy data hunting!

 

Related articles:

Scrape reviews from Amazon

Scrape Amazon product information with ASIN/UPC

Scrape product information from Sam's Club

 

Author: Vanny

Editor: Kara

 

Was this article helpful? Contact us at any time if you need our help!