Articles in this section

See more

Scrape jobs from LinkedIn

  • Updated
Follow

LinkedIn is a good database to find valuable job information. In this tutorial, we are going to introduce how to scrape job information from LinkedIn.com

To follow through, you may want to use the URL in the tutorial:

https://www.linkedin.com/jobs/search/?currentJobId=2011756127&geoId=105080838&keywords=accountant&location=New%20York%2C%20United%20States

We will scrape data such as the job title, company, level, type, function, and industry in Octoparse.

Before that, please make sure that you have downloaded our latest version 8.1 (Check this guide to download News: Octoparse 8.1 Beta Released! ). LinkedIn is no longer compatible with Octoparse 7.3.0. 

The website applies an infinite scroll coupled with a "Show More" to load more reviews. After we scroll the page to the bottom like 6 times, a "show more" button would reveal and if we want to continue to load jobs, we have to click on the button.

 

Here are the main steps in this tutorial.  [Download the demo task from here]

  1. "Go To Web Page" - open the targeted web page
  2.  Auto-detect web page - create a workflow
  3. Loop click "Show More" button - load more reviews
  4. Click into each link to get more detailed information
  5. Extract data - select the data for extraction
  6. Start extraction - run the task and get data
 

 

1. "Go To Web Page" - open the targeted web page

  • Enter the URL on the home page and click Start

mceclip0.png

 

 

2. Auto-detect web page - create a workflow

  • Choose "Auto-detect the web page data"
  • Wait for the detection to complete

2.1.gif

 

  • Check the data fields on the Data Preview, and you can also delete the unwanted fields or rename fields if needed

2.2.png

  • Click "Edit" under the "Add page scroll" option on the Tips panel
  • Set up the wait time as 4-5 seconds (make sure the time is long enough for the page to load new reviews)

2.3.gif

  • Click "Create workflow" on the Tips panel

2.3.png

 

 

3. Loop click "Show More" button - load more jobs

  • Choose "Click on a 'Load More' button" on the Tips panel
  • Select the "See more jobs" button on the web page
  • Set up the number of clicks according to how many jobs you need
  • Click "Confirm"
  • Set up AJAX Load as 5s

3.1.gif

 

 

4. Click into each link to get more detailed information

  • Choose “Click on link(s) to scrape the linked page(s)” on the Tips panel
  • Select "Click on an extracted data field" and select the "resultcard__fullcardlink_URL" from the drop-down menu (you can confirm if it's the correct link on the Data Preview)
  • Click "Confirm"

_.gif

 

  • Click open the action settings of the "Click URLs in the list"
  • Uncheck the "Open in a new tab" option
  • Tick "Load with AJAX" and set up the AJAX timeout as 5-7s
  • Click "OK" to confirm

4.2.gif

 

 

 

5. Extract data - select the data for extraction

  • Click on the data you want to extract on the page
  • Select "Extract the text of the selected element" on the "Tips" panel
  • Repeat the steps until you get all the data needed to be scraped

5.gif

  • Edit the name of data fields if needed5.1.png
 

 

6. Start extraction - run the task and get data 

  • Click "Save"
  • Click "Run" on the upper left side
  • Select "Run task on your device" to run the task on your computer, or select "Run task in the cloud " to run the task in the Cloud (for premium users only)

6.png

Here is the sample output.

mceclip0.png

 

Author: Lesley

Editor: Yina

Related articles