Scrape data on Instagram

In this tutorial, we are going to scrape data from Instagram, including the post content, date, image URL, number of likes and location.

Also, you can go to "Task Templates" on the main screen of the Octoparse scraping tool, and start with the ready-to-use Instagram Template directly to save your time. With this feature, there is no need to configure scraping tasks. For further details, you may check it out here: Task Templates

If you would like to know how to build the task from scratch, you may continue reading the following tutorial.

 

To follow through, you may want to use this URL in the tutorial:

 https://www.instagram.com/izkiz/

 

Here are the main steps in this tutorial:[Download demo task file here ]

 

1. "Go To Web Page" - open the target web page
2. Create a pagination loop - scrape data from multiple posts
3. Extract data - select the data for extraction
4. Customize the data field using RegEx tool - reformat the post date data (Optional)
5. Customize the data field by modifying XPath – improve the accuracy of data fields (Optional)
6. Save and start extraction - run the task and get data

 

 

 

 

 

 

 

 

 

 

 

• Create the task with "Advanced Mode".
• Paste the URL into the "Input URL" box and click "Save URL" to move on
• Click "Setting" to change the default built-in browser

The default built-in browser of Octoparse 7 is incompatible with Instagram. To have our target page loaded normally, we need to modify the browser setting. If you use Octoparse 7.2.2, please have the task saved before modifying the settings

• Switch the default built-in browser to Firefox 45.0.
• Click "Save" to apply the modified setting

Because of the special website structure of Instagram, we need to click one of the posts to return the “>” button, which is the “Next page” button to go to the next post.

• Click the first post and click the "A" tag at the bottom of "Action Tips"

When you select an item with URL, the selected tag would be "A". Normally there’s no need to modify, as Octoparse automatically identifies tags of selected items. But for this case, we need to revise the tag on the bottom of "Action Tips".

• Select "Click the link"
• Uncheck "Auto retry when no response"
• Check "Load the page with AJAX", and set up "AJAX Timeout” for "5" seconds

We have the first post opened now. However, as Instagram loads the content with AJAX, we should set up AJAX Load for the "Click Item" action.

Now, we can create “Pagination”.

• Click the ">" button
• Click "Loop click next page" on the "Action Tips" panel
• Click "Load the page with AJAX" and set up "AJAX timeout" for "5" seconds

Instagram uses AJAX on the ">" button, so we need to set up AJAX Load for "Click to Paginate" action as well.

Tips! To learn more about dealing with AJAX in Octoparse, please refer to Deal with AJAX. You can also go to this video tutorial Octoparse: AJAX 101

 

We are now in the second post. When creating a "Loop Item", we should always start with the first item on the first page. In this case, we should go back to the first post.

• Click "Go To Web Page" in the workflow
• Click "Click Item"

Octoparse would open the first post.

• Click the "Pagination" loop in the workflow

By doing this, we can help Octoparse decide the execution order and generate the "Extract data" step at the appropriate position in the workflow.

Now, let’s start to extract data.

• Select the data you want
• Click "Extract text of the selected element " on the "Action Tips" panel
  • For URL of the image, click "Extract URL of the image"
  • For the post date, select "Extract outer HTML of the selected element" 

• Rename the fields by selecting from the predefined list or inputting on your own
• Click "OK" to save

Tips: To learn more about how to adjust workflow, please refer to Getting to know Octoparse.

 

4.  Customize the data field using RegEx tool - reformat the post date data (Optional)

When extracting the post date data, we may find some data shown in different formats, like "3 days ago". To unify the format, we need to go to "Refine extracted data".

• Select the "post_date" field to be modified
• Click the icon of "Customize data field" 
• Select "Refine extracted data", click"Add step", and then select "Match with Regular Expression"
• Select "Try RegEx Tool"
• Check the box for "Start With" and enter: title="
• Check the box for "End With" and enter:" datetime
• Click "Generate" and "Match"
• Click "Apply" and "OK"
• Click "OK" to save

 

If we check the workflow manually, we may find that the data fields of location and "URL" are blank, which means the data are missing, so we need to go back to the source code of Instagram to find out their XPath expressions.

• Click the row of "URL" to modify the data field
• Click the icon of "Customize data field"
• Click "Customize XPath"
• Paste the revised XPath into the "Matching XPath" text box and click "OK"
  • Revised XPath: //img[@class="FFVAD"]

Follow the above steps to revise the XPath of the data field of "Location".

• Paste the revised XPath into the "Matching XPath" text box and click "OK"
  • Revised XPath: //a[@class="O4GlU"]

Tips! To improve the accuracy of a certain data field, modifying XPath in Octoparse is highly recommended. Here are some related tutorials you might need: How to associate data with nearby text? Data fetched to the incorrect data fields Locate elements with XPath Video: Octoparse: XPath 101

 

• Click "Save"
• Click "Start Extraction" on the upper left side
• Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)

For a premium user, Cloud Extraction is highly recommended. 

Now, we have all the extracted data. Here is the sample output.

 Related Articles:

Scrape tweets from Twitter

Scrape video information from YouTube

Can Octoparse scrape Facebook Community or Groups?

 

 

Was this article helpful? Contact us  any time if you need our help!