Scraping product details from Wayfair

May 20, 2022 05:27
Updated

You are browsing a tutorial guide for Octoparse's latest version. If you are running an older version of Octoparse, we strongly recommend you upgrade because it is faster, easier and more robust! Download and upgrade here if you haven't already done so!

In this tutorial, we are going to show you how to scrape the product details from Wayfair, an American home improvement retailer commerce company that sells home goods.

For this example, we will use the URL below in order to scrape data such as the product title, description, and price from each product detail page.

https://www.wayfair.com/lighting/sb1/table-lamp-night-lights-c416395-a76645~280587.html

Here are the main steps in this tutorial:

"Go To Web Page" - open the targeted web page
Create a pagination loop - scrape search results from all pages
Create a "Loop Item" - scrape all the items on each page
Extract data - select data for extraction
Run the task - to get the desired data

Note: Download the demo task file at the bottom of this page

1. "Go To Web Page" - open the targeted web page

Enter the target url into the search box at the center of the home screen
Click Start to create a new task in Advanced Mode

2. Create a pagination loop - scrape search results from all pages

Scroll down to the bottom of the page, click the "Next" button
Click "Loop click single URL" on the "Tips" panel

3. Create a "Loop Item" - scrape all the items on each page

Click on any product title on the page, the frame of the product will turn green
Click "Select all" on the "Tips" panel and the frames of all products will turn green
Click "Loop click each element"