You are browsing a tutorial guide for the latest Octoparse version. If you are running an older version of Octoparse, we strongly recommend you upgrade because it is faster, easier and more robust! Download and upgrade here if you haven't already done so!

Google Advanced Search is a more detailed method of finding information on Google. It uses a variety of Google search operators that consists of specific characters and commands – also known as "advanced operators" – that go beyond a standard Google search.

An example is shown in the picture below:

2.png

By clicking the Advanced Search button at the bottom of the webpage, it will lead to the search result page: https://www.google.com/search?as_q=web+designer&as_epq=&as_oq=&as_eq=&as_nlo=&as_nhi=&lr=lang_en&cr=&as_qdr=w&as_sitesearch=&as_occt=any&safe=images&as_filetype=&tbs=

1.png

This tutorial will show you how to scrape data from Google Advanced Search results with Octoparse using the URL above.

You can also go to "Task Templates" on the main screen of the Octoparse scraping tool and start with the ready-to-use Google Advanced Search Template directly to save time. For further details on task templates, you may check out here.

Here are the main steps in this tutorial: [Download task file here]

  1. Create a Go to Web Page - to open the target website

  2. Create a Pagination - to load more data on the following websites

  3. Create an Extract Data - to extract the search results

  4. Set Wait before Action - to make sure Data is fully loaded

  5. Run the task - to get your desired Data


1. Create a Go to Web Page - to open the target website

  • Enter the target URL into the search bar on the home screen and click Start

10.png

2. Create a Pagination - to load more data on the following websites

  • Click Next at the bottom of the webpage

  • Click Loop click next page

3.png
  • Set AJAX timeout: 7-10s recommended

AJAX.jpg

Note: Sometimes, Google may use Captcha as an anti-scraping measure. To solve the Captcha manually, turn on the Browse mode and follow the instructions.

5.png

If you want to learn more about how Octoparse can resolve the Captcha automatically during the extraction, please check out here: Resolve Captcha


3. Create an Extract Data - to extract the search results

  • Click the Title and Content of the first item on the webpage

  • Click Select All to create a loop of extraction

6.png
  • Click Extract data

11.png
  • Click on the More button next to the data field > Customize XPath

7.png
  • Modify the XPath of the Data field as below:

title: //div[@data-header-feature]/div/a/h3

content: //div[@data-content-feature]/div/span[2]

8.png

4. Set Wait before Action - to make sure data is fully loaded

Wait before action is a function that can be set to every action in the workflow. It will let the task wait before the action is executed.

In this case, it is better to add a Wait before Action for every step in the workflow.

  • Click on each step respectively > Options

  • Set Wait before action: 2-3s recommended

  • Click Apply to save the change

wait_2.jpg

5. Run the task - to get your desired Data

  • Click Save on the upper right to save your task

  • Click Run next to it and wait for a Run Task window to pop up

  • Select Run on your device to run the task on your local device

  • Wait for the task to complete

Here is the sample output from a local run:

9.png

Tip: Local runs are great for task troubleshooting and quick runs. If you are dealing with more complicated tasks, it is recommended that you select Run in the Cloud to run the task in Octopars's cloud-based platform for higher speed. Try out this premium feature by signing up for the 14-day free trial here. You can also schedule your task to run hourly, daily, or weekly and get data delivered to you regularly.

Did this answer your question?