Scrape different variants of a product

April 23, 2021 07:42
Updated

If you need to scrape e-commerce data, especially the product data, you may have such a need as follows.

For some products with different options, you want to collect each variant's price, SKU, etc. Taking this hair dye product as an example, you may need to scrape its pricing for each color.

Solution:

To show you how to do it with Octoparse, we can take this web page URL as an example: https://www.walmart.com/ip/SoftSheen-Carson-Dark-and-Lovely-Fade-Resist-Rich-Conditioning-Color/10314047

For this product, its color, pricing, images, page URL, and product ID will vary when you switch the option.

Here are some steps you can follow when you encounter this kind of situation.

1. Enter product URL(s) to start a new task

You can enter a list of URLs if you have a list of products to monitor. For the demonstration, I just enter one product URL.

2. Create a loop item to loop through each color option

Click the 1st color option on the list, and then choose "Select all" on the "Tips" panel
Then, choose "Loop click each element"
AJAX is detected for this web page, and you can modify the time based on your local Internet to load the page content (Learn more about Handling AJAX)
Double-click "Click Item" inside the "Loop Item" to uncheck "Open in a new tab".
(Optional) Double-click "Loop Item" to change the "Loop Mode" from "Fixed List" to "Variable List". Then, enter the Element XPath: //DIV[@class="variants__list"]/LABEL/DIV[2]. This is important when you have different products with different numbers of colors to scrape.