Articles in this section

Scrape job information from indeed

  • Updated
Follow

In this tutorial, we will show you how to build a web scraping with Octoparse to collect job posting information on Indeed.

We can scrape data such as data science, recruitment from job information on Indeed with Octoparse. After design the crawler, the entire scraping process is all automated without coding needed.

 

Let's see how it's done!

 

Before we start, we need to get the URL of the target result page by searching the keywords "DevOps" and "Dallas-Fort Worth, TX".

Then we will get the URL where we need to scrape data:

https://www.indeed.com/jobs?q=devops&l=Dallas-Fort%20Worth%2C%20TX&radius=50

 

We will scrape job titles and descriptions in this tutorial.

Here are the main steps in this tutorial [Download task file here]

  1. "Go To Web Page" - open the target website
  2. Create a pagination - extract multiple web pages
  3. Modify XPath - paginate correctly
  4. Create a "Loop Item" - loop extract each element on each row
  5. Extract data - select data from your target website
  6. Run your task - get data you want

  

1) "Go To Web Page" - open the target website

  • Create your task with "Advanced mode".
  • Paste the URL we just got into the "Website" box and save it to move on.
  • Check "Block Pop-up" to avoid all possible pop-up windows and click "OK" to save

1.gif

Tips!

We always suggest you turn on "Workflow" to get a better picture of what you are doing with the task.

 

2) Create a pagination - extract multiple web pages

  • Scroll down to find the "Next" button
  • Select "A" tag, and click "Loop click the selected link" since it does not automatically locate on the "A" tag of the button

2.gif

 

3) Modify XPath - paginate correctly

XPath is a language that allows you to locate specific elements from a page precisely based on the tags and attributes. So before you get down to write your own XPath, you would need to inspect the HTML structure of the page firstly.

  • Find the correct XPath with Firepath/Firebug extension tool in Firefox browser.

The correct XPath is //span[contains(text(),'Next')][@class="np"]/../..

2.5.png

  • Click pagination loop in your workflow and paste the correct XPath into "Single element" box under "Advanced Options"

2.6.gif

Tips!

The Firebug extension tool is very useful for looking up the elements of an HTML document. (Firebug is now only available for old versions of Firebox. Get the old versions of Firebox here.)

Modifying XPath in Octoparse works very well with more flexibility and accuracy than the XPath auto-generated by clicking elements during the task configuration. So you need to check the "Single element" in the "Loop mode" if you cannot extract data from the next page.

If you are new to XPath, please learn more from the tutorials here, or enjoy a video tutorial here.

 

4) Create a "Loop Item" - loop extract each element on each row

  • Select all information of the first two listing items

Usually when creating a loop, choosing all information instead of only part of the information would be better. Selecting part of the information might cause a problem when you need more information than you have already selected.

  • Click "Extract text from the selected elements" on the "Action Tips" panel

3_loop.gif

 

5) Extract data - select data from your target website

  • Select the job title of the first item and click "Extract text of the selected elements" in "Action Tips" panel
  • Select the description of the first item and click "Extract text of the selected elements" in "Action Tips" panel
  • Delete the first field because it contains all messy text in the first listing item

4_extract.gif

 

  • Extract other data you want and rename the field names if necessary

sample_field.png

For the description field, we need to modify the correct XPath. The correct XPath is .//td[@id='resultsCol']/div[contains(@class,'row')].

  • Select "Description" and click modify.png
  • Select "Customize XPath" and change "Matching XPath" into the correct XPath above
  • Click "OK" to save

xpath.gif

 

6) Save and start extraction - run the task and get data

  • Click"Save"
  • Click "Start Extraction" on the upper left side
  • Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)

run.gif

 

Happy data hunting!

 

Writer: Vanny

Editor: Fergus

Related articles