Articles in this section

See more

Scrape data from Google Search

  • Updated
Follow

Scraping data from a search engine is a good way to collect information related to one topic. In this tutorial, we are going to show you how to scrape the search results data on Google search.

You can go to "Task Templates" on the home screen of the Octoparse and start with the ready-to-use Google Search Template directly to save your time. With this template, there is no need to configure scraping tasks. For further details, you may check it out here: Task Templatesexternal-link-symbol-1.png

 mceclip0.png

 

You can also use our Advanced Mode to create your own task. To follow through, you may want to use this URL in the tutorial: https://www.google.com/

We will scrape data such as the title, URL, and description from the search results page with Octoparse.

Here are the main steps in this tutorial:[Download demo task file here ]

  1. Open the targeted web page
  2. Auto-detect the web page to create steps to enter text
  3. Modify the settings for the "Click Item"
  4. Auto-detect the search result page to scrape data
  5. Set up wait time to slow down the scraping speed
  6. Save and start to run the task and get data

 

 

1) Open the targeted web page

  • Enter the URL on the home page and click Start

1.png

 

2) Auto-detect the web page to create steps to enter text

  • Click "Auto-detect web page data" and wait for the detection to complete

auto_detect1.gif

  • Choose "Search with keywords" on the Tips panel and you will see instructions to help you set up steps

3.png

a. "Add a search box": click "Settings" and select the search box on the web page

add_search_box.gif

b. "Add keyword(s)": click mceclip0.png and input the keyword(s). One keyword per line.

 input_keywords.gif
 
c. "Click the search button": tick "Click the search button when finishing entering" and select the search button on the web page
 
click_search_button.gif
  • Click "Confirm" to generate the workflow
 

 

3) Modify the settings for the "Click Item"

  • Double-click the "Click Item" to enter the Action Settings panel
  • Tick "Open in a new tab"
  • Extend the AJAX Load timeout

modify_Click_item.gif

 

4) Auto-detect the search result page to scrape data

  • Auto-detect the page again

 auto_detect2.gif

 

  • Click "Create workflow"

4.png

  • Rename the fields or delete fields you don't want

rename_fields.gif

 

 

 5) Set up wait time to slow down the scraping speed
Google search applies an anti-scraping technique and it would show reCAPTCHA to solve. We need to slow down the scraping by setting the wait time.
  • Double-click the Extract Data action
  • Tick "Wait before action" 
  • Select the wait time as 1s-3s
set_up_wait_time.gif
 
 

6) Save and start to run the task and get data

  • Click"Save"
  • Click "Run" on the upper left side
  • Select "Run on your device" to run the task on your computer, or select "Run in the Cloud" to run the task in the Cloud (for premium users only)

5.png

 

Here is the sample output.

mceclip1.png

 

Is this article helpful? Contact us  any time if you need our help!

Author: Yina

Editor: 

 

Related articles