Octoparse Help
Hello, how can we help?
900000374763
900002131403
  1. Octoparse
  2. 8.1 Beta
  3. Case Tutorials

Scrape business information from Yelp

  • Updated
Follow

Yelp is one of the largest business directory websites on the Internet. Millions of people search for target businesses or leave a review on this website every day. The business info and the reviews are adding valuable information to the platform. In this tutorial, we will show you how to collect business information on Yelp with Octoparse 8.1. 

The easiest way would be using our pre-built Yelp templates. No need to configure scraping tasks but just enter keywords/URLs to get data. For further details, you may check it out here: Task Templatesexternal-link-symbol-1.png

0.png

If you would like to know how to build the task from scratch, you may continue reading the following tutorial or check this video below.

 

To demonstrate, we will use the URL below as an example.  

https://www.yelp.com/search?find_desc=&find_loc=Seattle%2C+WA&ns=1

We will scrape data such as title, star rating, number of reviews, and website with Octoparse.

 

 

Here are the main steps in this tutorial: [Download demo task file here ]

  1. Go To Web Page - open the target web 
  2. Auto-detect the web page - create the workflow
  3. Modify the settings of the Pagination
  4. Click into each detail page to get more information
  5. Set up wait time to slow down the scraping speed
  6. Start extraction - run the task and get data
 

 

1. Go To Web Page - open the target web page

  • Enter the URL on the home page and click "Start"

1.png

 

 

2. Auto-detect the web page - create the workflow

  • Click "Auto-detect web page data" and wait for the detection to complete

2.1.gif

  • Delete the unwanted fields or rename fields if needed on the Data preview

2.2.png

  • Click "Edit" under "Paginate to scrape more pages"
  • Select the next button on the web page
  • Click "Confirm"2.2.gif
  • Click "Create workflow"

2.2.png

 

 

3. Modify the settings of the Pagination

The default XPath for the Pagination works well in most cases, but yelp is quite strict with data scraping and it has a very hard-to-read source code, so we need to revise the XPath for the Pagination.
  • Click open the action settings of the Pagination
  • Enter the XPath: //a[contains(@class,'next-link')]
  • Click "OK" to confirm

3.1.gif

Tips!

If you want to learn more about XPath, please check the following tutorial:

What is XPath and how to use it in Octoparse

 

  • Click open the action settings of "Click to Paginate"
  • Extend the AJAX timeout to 7-10s

3.2.gif

 

If all the data you need could be scraped from the listing page, you can stop here and jump to Start extraction - run the task and get data . If you want to go to each detail page to get more info, follow the steps below.

 

 

4. Click into each detail page to get more information

  • Choose “Click on link(s) to scrape the linked page(s)” on the Tips panel
  • Select "Click on an extracted data field" and select "Title_URL" from the drop-down menu (you can confirm if it's the correct link on the Data Preview)
  • Click "Confirm"

4.1.gif

 

 

5. Extract Data - extract data on the detail pages

  • Select information on the web page
  • Choose "Extract text of the selected element"
  • Repeat the above steps to extract all the data you need

5.1.gif

Tips!

  • If you cannot select the whole address on the page, select part of it firstly, and then click the "Address" tag on the Tips panel

5.2.png
  • Rename the fields is needed

5.3..png

 

Now we need to revise the element XPath for each data field to scrape the information more precisely.

  • Double-click the "Extract Data1" action in the workflow
  • Click the icon mceclip0.png to modify the XPath one by one

mceclip0.png

  • Replace the default XPath with the revised one

4.2.png

No worries! We have prepared frequently used fields' XPath for you. You can just use the element XPath provided below.

  • Website: //p[text()='Business website']/following-sibling::p[1]
  • Phone: //p[text()='Phone number']/following-sibling::p[1]
  • Open hours: //table[contains(@class,'hours-table')]
  • Address: //address
  • Amenities: //h4[text()='Amenities and More']/../../following-sibling::div
  • About the business: //h4[text()='About the Business']/../../following-sibling::div
  • Price range: //span[contains(text(),'$')]
  • Category: //a[contains(@class,'link-color--inherit')]
  • Rating: (//div[contains(@aria-label,'star rating')])[1]

Tips!

The rating info is stored inside the value of the attribute. We can click the Rating areamceclip1.png, choose to extract the text. Then customize the field to scrape the value of "aria-label" attribute.

5.4.png

 

To know more about how to deal with "Extract Data", check the following guides:

  1. Add pre-defined data fieldsexternal-link-symbol-1.png
  2. Extract Dataexternal-link-symbol-1.png
  3. Extract star rating information
 

 

6. Set up wait time to slow down the scraping speed

Yelp website applies an anti-scraping technique and it would block your IP if you scrape too fast. We need to slow down the scraping by setting the wait time.
  • Double-click the "Extract Data1" action
  • Tick "Wait before action" 
  • Select the wait time as 7s-10s

6.1.gif

 

7. Start extraction - run the task and get data 

  • Click"Save"
  • Click "Run" on the upper left side
  • Select "Run on your deviceexternal-link-symbol-1.png" to run the task on your computer, or select "Run task in the Cloudexternal-link-symbol-1.png" to run the task in the Cloud (for premium users only)

6.png

 

Here is the sample output.

mceclip3.png

 

Tutorial en español: Scrapear información comercial de Yelp

También puedes leer más artículos de web scraping en el sitio web oficial

 

Author: Fergus

Editor: Yina

 

Related articles

Take a hand, lend a hand
Join our worldwide community to ask expert users for advice or share your own.
Ask the Community