In this tutorial, we are going to show you how to scrape hotel information on Booking.com.
Also, you can go to "Task Templates" on the main screen of the Octoparse scraping tool, and start with the ready-to-use Booking Template directly to save your time. With this feature, there is no need to configure scraping tasks. For further details, you may check it out here: Task Templates
If you would like to know how to build the task from scratch, you may continue reading the following tutorial.
You may need this URL to follow through in the tutorial: https://www.booking.com/searchresults.en-gb.html?label=gen173nr-1FCAEoggJCAlhYSDNYBGgxiAEBmAEuwgEDYWJuyAEO2AEB6AEB-AELkgIBeagCBA&sid=1964712b1d001062109ae5baf669fd26&checkin_month=12&checkin_monthday=1&checkin_year=2018&checkout_month=12&checkout_monthday=2&checkout_year=2018&class_interval=1&dest_id=20088325&dest_type=city&dtdisc=0&from_sf=1&group_adults=1&group_children=0&inac=0&index_postcard=0&label_click=undef&no_rooms=1&offset=0&postcard=0&raw_dest_type=city&room1=A&sb_price_type=total&src=index&src_elem=sb&ss=New%20York&ss_all=0&ssb=empty&sshis=0&ssne=New%20York&ssne_untouched=New%20York&selected_currency=USD&changed_currency=1&top_currency=1&nflt=
We will scrape data such as hotel names, images, addresses, descriptions, scores, reviews, and star ratings with Octoparse.
Here are the main steps in this tutorial:[Download the demo task click here]
- Go to the web page - open the target web page
- Auto-detect the web page - create a workflow
- Click into each detail link to scrape more information
- Extract Data - extract data on the detail pages
- Set up wait time - slow down the scraping speed
- Modify the XPath of Pagination
- Start extraction- run the task and get data
1. Go to the web page - open the target web page
2. Auto-detect the web page - create a workflow
- Click the "Auto-detect web page" and uncheck the "Add a page scroll" to create a workflow
- Adjust the order for the fields as you want
- Delete and rename the field
With Octoparse 8.4, it is quite convenient to delete the fields that you don't want together after auto-detection. Click the vertical view icon to switch to vertical view to delete and rename the fields. Note that you need to double-click on the field name to rename it.
If the data you need can all be scraped from the listing page, you can just jump to Set up wait time to slow down the scraping speed. If you want to click on each detail link to get more information, please follow the next step.
3. Click into each detail link to scrape more information
- Choose to “Click on link(s) to scrape the linked page(s)” on the Tips panel
- Select "Click on an extracted data field" and select the one you want to click on from the drop-down menu (you can confirm if it's the correct link in the Data Preview)
- Click on "Confirm"
4. Extract Data - extract data on the detail pages
- Select the data you need and click "Extract the text of the element"
- Double click on the field name to rename it if needed
5. Set up wait time - slow down the scraping speed
Booking might block your IP if you scrape it too much, therefore we need to control the scraping speed.
- Click on the "Extract Data1" action
- Tick "Wait before action"
- Set up time as 5s-10s
6. Modify XPath of the Pagination
The auto-generated XPath of Pagination does not always locate the next page button, so we need to modify the XPath.
- Click on Pagination
- Replace the XPath with //button[@aria-label="Next page"]
- Click Apply to save
TIP: Check out this tutorial to learn more about Path: What is XPath and how to use it in Octoparse
7. Start extraction- run the task and get data
- Click Save
- Click Run on the upper left side
- Select Run on your device to run the task on your computer, or select Run task in the Cloud to run the task in the Cloud (for premium users only)
Here is the sample output -