Scrape product data from Canadian Tire

In this tutorial, we will show you how to collect product information on canadiantire.com with Octoparse.

We will scrape each detail page URL in Task 1 and extract the data such as the product title, price, and brand from product details page with Task 2. By splitting one task into two, the extraction speed could be improved to a certain degree, especially when we use the Octoparse Cloud Extraction.

 

To follow through, you may want to use this URL in the tutorial:

https://www.canadiantire.ca/en/search-results.html?q=bulb

 

This tutorial will also cover:

• Deal with AJAX for pagination

 

Here are the main steps in this tutorial [Download demo task files here]

Task 1: Extract all the URLs of detail pages on the search result pages

1. "Go to Web Page" - to open the target web page
2. Build a "Loop Item"- to loop extract each URL on the search results pages
3. Set Scroll Down - to load all items from one page
4. Start extraction - to run the task and get data

Task 2: Collect the product information from scraped URLs

1. Input a batch of the scraped URLs - to loop open the detail pages
2. Extra data - to select the data for extraction
3. Start extraction - to run the task and get data

 

Task 1: Extract the detail page URLs on the search result pages

• Click "+ Task" to start a new task with Advanced Mode

Advanced Mode is a highly flexible and powerful web scraping mode. For people who want to scrape from websites with complex structures, like canadiantire.com, we strongly recommend Advanced Mode to start your data extraction project.

• Paste the URL into the "Extraction URL" box and click "Save URL" to move on

 

• Select one of the product items on the search result page

By the above step, Octoparse is able to detect similar elements and highlight them in red.

• Click "A" tab on the bottom of the "Action Tips"

As we need to extract the URLs in a loop, we’d better make sure you select the "A" tag when you extract the URL. ( "A" tag stands for anchor.)

• Click "Select All"
• Click "Extract the URLs of the selected elements"
• Copy and paste this XPath expression "//div[contains(@class,'search-results')]/div[position()>last()-36]/div/a[contains(@class,'product')]" into the box for "Variable list"

We need to modify XPath in order to locate all the items correctly.

Tips! If you want to learn more about XPath, here is a related tutorial you might need： Locate elements with XPath

• Scroll down and click the "LOAD MORE RESULTS" button on the web page
• Click "Loop click selected element" on the "Action Tips"

Octoparse Version 7.2.2 can detect the AJAX technique and automatically set up AJAX Load. To ensure the web page is fully loaded in Octoparse built-in browser, we need to set up a time for AJAX timeout.

• Select an appropriate time for "AJAX Timeout" in the drop-down menu
• Click "Save" to have the step saved.

Tips! 1. To learn more about how to deal with infinite scrolling in Octoparse, please refer to: Dealing with Infinite Scrolling/Load More  2. If you want to learn more about AJAX, here is a related tutorial you might need： Deal with AJAX

 

• Click "Save"
• Click "Start Extraction" on the upper left side
• Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)

 

Task 2: Collect the product information from scraped URLs

With Task 1, we are able to get a batch of URLs.

• Click "+ Task" to start a task using Advanced Mode to build Task 2
• Input batch URL

There are three ways to batch import URLs to any single task/crawler (up to one million URLs). In this case, we will batch import URLs from local files. For further study, please refer to Batch Import URLs

 

 

As we can see, we are on the detail page now.

• Click the information you need on the page
• Select "Extract data" on the "Action Tips"

In this step, we are able to rename the fields by selecting from the pre-defined list or inputting on your own.

 

• Click "Save"
• Click "Start Extraction" on the upper left side
• Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)

For a premium user, Cloud Extraction is highly recommended.

 

Author: Erika

Editor: Yvonne