Scrape restaurant info from Grubhub

December 22, 2020 10:37
Updated

In this tutorial, we are going to show you how to scrape restaurant information from Grubhub.

https://www.grubhub.com/search?orderMethod=delivery&locationMode=DELIVERY&facetSet=umamiV2&pageSize=20&hideHateos=true&searchMetrics=true&latitude=40.71277618&longitude=-74.00597382&variationId=0.5-new-gotos&sortSetId=umamiV2&sponsoredSize=3&countOmittingTimes=true

We will scrape data such as the title, address, and phone from each restaurant details page with Octoparse.

Here are the main steps in the tutorial:[Download demo task file here ]

"Go To Web Page" - to open the targeted web page
Create a pagination loop - to scrape all the results from multiple pages
Create a "Loop Item" - to loop click into each restaurant on every page
Extract data - to select data you need to scrape
Save and start extraction - to run your task and get data

1. "Go To Web Page" - to open the targeted web page

Create the task with "Advanced Mode"
Paste the URL into the "Website" box
Click "Save URL" to move on

2. Create a pagination loop - to scrape all the results from multiple pages

Scroll down and click the ">>" button on the web page
Click "Loop click single element" on "Action Tips"
Uncheck "Auto-Retry"
Check "AJAX Load" and set up "AJAX Timeout" as 7 seconds

Set the timeout long enough for the page to load according to your network condition.

Click "Save"

Tips!

AJAX timeout can often be used as a web page timeout for Click Action. For example, when you have a page that takes forever to finish loading, long after the data you need gets loaded, you can conveniently use AJAX timeout to tell Octoparse to move on to the next action when the set time is reached.

If you want to learn more about AJAX, you can enjoy the video tutorial here .

3. Create a "Loop Item" - to loop click into each restaurant on every page

We are now on the second page. When creating a "Loop Item", we should always start with the first item on the first page. Thus, we 'd better go back to the first page.

Click "Go To Web Page" in the workflow.
Select the pagination loop in the workflow

By doing this, we can help Octoparse decide the execution order and generate the Loop Item at the appropriate position in the workflow.

Click the first restaurant item, Octoparse will automatically identify the similar URLs on the page

The first restaurant item is highlighted in green while the others are highlighted in red.

Click "Select All" on the "Action Tips"

All the items are highlighted in green.

Select "Loop click each element"
Uncheck "Auto-Retry"
Uncheck “open the link in new tab"
Check "AJAX Load" and set up "AJAX Timeout" as 7s (optional)
Click on "Loop Item" and set up some wait time as 10s to ensure the web page loads completely (optional)
Click "Save"

4. Extract data - to select the data you need to scrape

Select the data you need on the item page to scrape, such as Name of the restaurant, Address, Opening hours, phone number...
Select "Extract text of the selected element" and rename the "Field name" column if necessary.

Rename the fields by selecting from the pre-defined list or inputting on your own.

Click "OK" to save the result

Normally we can just Click "<" (return to the list page button) to generate a "Click Item" action, but Octoparse fails to do that here. So we need to :

Drop a "Click Item" action into the workflow designer
Click "Customize" and "Customize XPath"
Set the XPath "//BUTTON[contains(@class,'returnToSearch')]" to locate the "<" (return to the list page button)
Uncheck "Auto retry when no response"
Check "Load the page with AJAX" and sets Time out
Click "Save" to move on

Tips!

Modifying XPath in Octoparse works very well with more flexibility and accuracy than the XPath auto-generated.
If you are new to XPath, please learn more from the tutorials here, or enjoy a video tutorial here.

5. Save and start extraction - to run your task and get data

Click "Save"
Click "Start Extraction" on the upper left side
Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)