In this tutorial, we are going to show you how to scrape the listing details from BestBuy.
To follow through, you may want to use this URL in the tutorial:
We will scrape data such as product title, model, and reviews from the product details page with Octoparse.
Here are the main steps in this tutorial: [Download task file here
]
- "Go To Web Page" - to open the targeted web page
- Create a pagination loop - to scrape all the reviews from multiple pages
- Create a "Loop Item" to scrape all the reviews on one page
- Extract data - to select the title and model to extract
- Extract data - to select the reviews to extract
- Run extraction - to run your task and get data
1. "Go To Web Page" - to open the targeted web page
- Click "+ Task" to start a new task with Advanced Mode
Advanced Mode is a highly flexible and powerful web scraping mode. For people who want to scrape from websites with complex structures, like Amazon.com, we strongly recommend Advanced Mode to start your data extraction project.
- Paste the URL into the "Input URL" box
- Click "Save URL" to move on
2. Create a pagination loop - to scrape all the reviews from multiple pages
- Scroll down to the bottom of the page
- Click the next page button ">"
- Click "Loop click next page" on the "Action Tips"
- Uncheck the box for "Retry when the page remains unchanged"
- Set "Wait before execution" as 10 seconds (optional according to your network)
- Check the box for "Load the page with AJAX" and set AJAX Timeout
|
Tips! If you want to learn more about AJAX, here is a related tutorial you might need： |
3. Create a "Loop Item"- to scrape all the reviews on one page
- Click the title of the first list on the current page
- Click "Select All" on the Action Tips panel
- Click "Loop click each URL"
- Undo "Retry when the page remains unchanged"
- Set "Wait before execution" for 30 seconds (optional according to your network)
- Check the box for "Load the page with AJAX" and set AJAX Timeout
- Click "Save"
4. Extract data - to select the title and model to extract
- Click on the data you need on the page
- Select "Extract text of the selected element" from the "Action Tips"
- Rename the fields by selecting from the pre-defined list or inputting on your own
- Click "Save"
5. Extract data - to select reviews for extraction
- Click "Customs" on inner browser
- Choose "Click element"
We need to click "Customs" because the reviews are not visible on the inner browser until you click the button to expose them.
- Set "Wait before execution" for 15seconds (optional according to your network)
- Undo "Retry when the page remains unchanged"
- Click "Load the page with AJAX" and set up AJAX Timeout
- Click the first review on inner browser
- Click "Select all"
- Click "Extract text of the selected element" on the "Action Tips"
- Rename the fields by selecting from the pre-defined list or inputting on your own
6. Start extraction - to run the task and get data
- Click "Save"
- Click "Start Extraction" on the upper left side
- Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)
Here is the sample for your information.
Author: Cathy
Editor: Yvonne