Scrape car information from Kijiji (V7.3)

May 10, 2022 06:48
Updated

In this tutorial, we will show you how to scrape car data from Kijiji as an example.

For Kijiji, you could visit our easy-to-use "Task Template" on the main screen of the Octoparse scraping tool. All you need is to type in several parameters and the task is ready to go. For further details, you may check it out here: Task Templates

To follow through, you may want to use this URL in the tutorial:

https://www.kijiji.ca/b-cars-trucks/guelph/c174l1700242

We will scrape data such as the title, location, price, and rating from car details page with Octoparse.

This tutorial will also cover:

Deal with AJAX

Tips!

It is recommended that you use the URL of the search result page directly whenever possible. Adding keywords/filters within Octoparse can complicate the task and leads to less efficient scraping.

Here are the main steps in this tutorial: [Download task file here ]

Go To Web Page - open the targeted web page
Create a pagination loop - scrape all the details from multiple pages
Create a "Loop Item" - loop click into each item on each list
Extract data - select the data for extraction
Save and start extraction - run the task and get data

1. Go To Web Page - open the target web page

Click "+ Task" to start a task using Advanced Mode
Paste the URL into the "Website" box
Click "Save URL" to move on

2. Create a pagination loop - scrape all the details from multiple pages

Scroll down to the bottom of the page, click the "Next" button
Click "Loop click the selected link" on the "Action Tips" panel
Set up an AJAX loading for 5s (optional according to your local network condition)
Uncheck "Retry when page fails to load"
Click "OK" to save

Tips!

AJAX timeout can often be used as a web page timeout for the Click Action. For example, when you have a page that takes forever to finish loading, long after the data you need gets loaded, you can conveniently use AJAX timeout to tell Octoparse to move on to the next action when the set time is reached.

If you want to learn more about AJAX, here are some related tutorials you might need:

3. Create a "Loop Item" - loop click into each item on each list

Click on a product title on the page
Click "Select all" on the "Action Tips" panel
Click "Loop click each URL"
Uncheck the "Retry when the page remains unchanged"
Click "Load the page with AJAX" and set AJAX Timeout as 5 seconds (optional)
Click "Save" to move on

Tips!

The first and last several products on this page are advertisements. Be careful not to click them if you only want to scrape the real products on this page.

4. Extract data - select the data for extraction

Click on the data you need on the page
Select "Extract text of the selected element" on the "Action Tips" panel
Click "Add predefined field " and choose "Add current page information" and select "Web page URL" (Optional)

Rename the fields by selecting from the predefined list or inputting on your own

5. Save and start extraction - run the task and get data

Click"Save"
Click "Start Extraction" on the upper left side
Select "Local Extraction " to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)

Here's the data we extracted.

Artículo en español: Scrape la información de bienes raíces de Kijiji
También puede leer artículos de web scraping en el Website Oficial

Writer: Yanni

Editor: Vanny