In this tutorial, we will show you how to collect product details on bukalapak.com with Octoparse.
We will scrape each product page URL of Huawei products in Task 1 and extract the product title, price, and seller information from each product page with Task 2. By splitting one task into two, the extraction speed could be improved to a certain degree, especially when we use the Octoparse Cloud Extraction.
To follow through, you may want to use this URL in the tutorial:
Here are the main steps in this tutorial [Download demo task file here
]
Task 1: Extract all the URLs of detail pages on the search result pages
- "Go to Web Page"- to open the target web page
- Create a pagination loop - to scrape all the results from multiple pages
- Loop extract detail page URL on each list - to select all the product URLs
- Save and start extraction - to run the task and get data
Task 2: Collect the product information from scraping URLs
- Input a batch of the scraped URLs - to loop open the detail pages
- Extra data - to select the data for extraction
- Start extraction - to run the task and get data
Task 1: Extract all the URLs of detail pages on the search result pages
1. "Go to Web Page" - to open the target web page
- Click "+ Task" to start a new task with Advanced Mode
Advanced Mode is a highly flexible and powerful web scraping mode, with which you can build your own workflows to cater to your needs.
- Paste the URL into the "Extraction URL" box and click "Save URL" to move on
2. Create a pagination loop - to scrape all the results from multiple pages
- Scroll down and click the "Next page" button on the web page
- Click "Loop click next page" on "Action Tips"
Octoparse Version 7.2.2 can detect the AJAX technique and automatically set up AJAX Load. To ensure the web page is fully loaded in the Octoparse built-in browser, we need to set up a time for AJAX timeout. Meantime, please make sure the "Auto Retry" has been unchecked.
- Select an appropriate "AJAX Timeout" in the drop-down menu
- Click "OK" to have the step saved.
3. Loop extract detail page URL on each list - to select all the product URLs
- Click any product title
- Click "Select all" on the "Action Tips"
- Click "Extract URLs from the selected link"
- Rename the fields
4. Save and start extraction - to run the task and get data
- Click “Start Extraction” on the upper left side
- Select “Local Extraction” to run the task on your computer, or select “Cloud Extraction” to run the task in the Cloud (for premium users only)
Export the result after the data extraction has completed. The list of URLs will be used in Task 2.
Task 2: Collect the product information from scraped URLs
1. Input a batch of the scraped URLs - to loop open the detail pages
- Click "+ Task" to start a task using Advanced Mode to build Task 2
- Input batch URLs
There are three ways to batch import URLs to any single task/crawler (up to one million URLs). In this case, we will batch import URLs from local files. For further study, please refer to Batch Import URLs
2. Extra data - to select the data for extraction
- Click on the data you need on the page
- Select "Extract text of the selected element" on the "Action Tips"
- Rename the fields
3. Save and start extraction - to run the task and get data
- Click “Start Extraction” on the upper left side
- Select “Local Extraction” to run the task on your computer, or select “Cloud Extraction” to run the task in the Cloud (for premium users only)
Happy data hunting!
Author: Yanni
Editor: Yvonne