Scrape property data from Realtor.com

August 16, 2021 12:45
Updated

In this tutorial, we are going to show you how to scrape property data from Realtor.com.

We will scrape data from the house detail page and scrape the title, location, price, and rating, etc with Octoparse. To follow through, you may want to use the URL in the tutorial:

https://www.realtor.com/realestateandhomes-search/Tallassee_AL

We'll use 2 tasks to get data in the detail pages.

Here are the main steps in this tutorial:

Task 1: Extract all the URLs of detail pages on the search result pages [Download the demo task file here ]

"Go To Web Page" - open the target web page
Create a pagination loop - scrape all the results from multiple pages
Create a "Loop Item" - to loop extract URLs of all the listings
Start extraction - run the task and get data

Task 2: Collect the product information from scraped URLs [Download the demo task file here ]

Input a batch of the scraped URLs - loop opens the detail pages
Extract data - select the data for extraction
Refine the data fields
Start extraction - run the task and get data

Task 1: Extract the detail page URLs on the search result pages

1. "Go To Web Page" - open the target web page

Enter the example URL and click "Start"

2. Create a Pagination - scrape all the results from multiple pages

Scroll down and click the ">" button on the web page
Click "Loop click single URL" on the Tips panel

Octoparse auto-detects AJAX applied for the click action as 3 seconds. You can modify it based on your local Internet condition (Click to know more about AJAX: Handling AJAX).

Set up "AJAX timeout" as "5" seconds

Double-click the "Pagination" step in the workflow
- Copy and paste the revised XPath for the next page button:
- //a[@aria-label="Go to next page"][not(contains(@class, "disabled"))]

Expand the section of "Before action is performed"
Check "Wait before action" and set the wait time as 2s
Click "OK" to save

3. Create a "Loop Item" - to loop extract URLs of all the listings

Click the image of the 1st item on the list
Click the A tag on the bottom of the Tips panel (A tag defines a hyperlink, which is used to link from one page to another) and press OK
Add a loop Item out of the extract Data showing as below

We can see some items are not selected, so we need to modify the Loop Item.

Click of the "Loop Item"
Change the Loop Mode from "Fixed List" to "Variable list"
Enter XPath //ul[@data-testid='property-list-container']/li into the text box
Click "OK" to save
Go to 'Extract Data' and click the gear icon

Click to modify the URL XPath

Click the 'Relative XPath' and Set the XPath as //a[@rel="noopener"]

Click 'OK' to finish

4. Start extraction - run the task and get data

Click
Click on the upper left side
Select "Run task on your device" to run the task on your computer, or select “Run task in the Cloud” to run the task in the Cloud (for premium users only)

If you're a premium user or trial user, I will suggest you use "Run task in the Cloud" so that you can use the feature of associative tasks (Check this guide What are parent task and child task in Octoparse? for details).

Here is the output data.

Task 2: Collect the property data from scraped property URLs

1. Input a batch of the scraped URLs - loop open the detail pages

In Task 1, we have already got a list of URLs.

Click "+ New" to start a task using Advanced Mode to build Task 2
Choose "Import from task" to get the URLs from Task 1

Tips!

There are 4 ways to input URLs. In this tutorial, we use "Import from task" for demonstration. Please note that this one only works when the parent task is running in the Cloud. If we import from a local run data result, only 100 lines of data will be imported. To know more about importing URLs, check this guide: Batch URL input.

After clicking the "Save" button, you will see a loop item named "Loop URLs" be generated in the workflow.

2. Extract data - select the data for extraction

Click on elements you want to scrape
Choose "Extract text/URL/image URL of the selected element" on the Tips panel

Click to rename the fields

3. Refine the data fields

To avoid data fetched to the wrong column, we will need to Customize element XPath.

Double-click the "Extract Data" step in the workflow to revise the XPath of some data fields
Click the icon to modify XPath
Enter the revised XPath into the text box and click "OK" to save

Here are revised XPaths for some common data fields

Presented_by: //div[contains(text(),'Presented')]/following-sibling::span[2]
Brokered_by: //li[contains(text(),'Brokered')]/following-sibling::li[1]
Price: //span[contains(@class,'price')]
Facilities: //ul[contains(@class,'property')]
Address: //h1[contains(@class,'address')]
Property_type: //span[contains(text(),'Property')]/following-sibling::span[1]
Last_sold: //span[contains(text(),'Last Sold')]/following-sibling::span[1]
Days_on_realtor: //span[contains(text(),'Days on')]/following-sibling::span[1]
Parcel_number: //li[contains(text(),'Parcel')]
Source Listing Status: //li[contains(text(),'Source Listing Status')]

4. Start extraction - run the task and get data

Click "Save" to save the task first
Then, click "Run" on the upper left side
Select "Run task on your device" to run the task on your computer, or select “Run task in the Cloud” to run the task in the Cloud (for premium users only)

Here is the sample output.

Author: Vanny

Editor: Yina