Scrape tweets from Twitter

December 15, 2020 01:30
Updated

With Octoparse, you can easily scrape any data you want such as top news, hot topics, worldwide trends, etc. from a variety of social media websites, such as Twitter.

In this tutorial, we will show you how to scrape data from Twitter. Any data seen on the web page can be scraped without coding. If you are interested in the data scraping from these social media websites like Twitter, this tutorial can help you get started.

Also, you can go to "Task Templates" on the main screen of the Octoparse scraping tool, and start with the ready-to-use Twitter Template directly to save your time. With this template, there is no need to configure scraping tasks. For further details, you may check it out here: Task Templates

Tips!

Learn about Octoparse data collection service for large scale data acquisition.

If you would like to know how to build the task from scratch, you may continue reading the following tutorial or check the newly updated tutorial on scraping Twitter with our brand-new version 8.1 beta.

To illustrate, we will take scrape news from Twitter as an example: https://twitter.com/search?q=Latest%20News&src=tyah

By scraping data from Twitter, you can:

Keep updated with the latest trends worldwide
Find out potential customers for your business
Analyze the marketing value of hot topics

Let's get started with the main steps in this tutorial to start your task. [Download demo task file here]

"Go To Web Page" - open the target website
Set "Scroll down" - load more data from the listed page
Create a "Loop Item" and extract data - loop extract each tweet
Set Regular Expression - clean and reformat data if needed (Optional)
Start extraction - run the task and get data

1. "Go to Web Page" - open the targeted web page

Click "+ Task" to start a task using Advanced Mode

Advanced Mode is a highly flexible and powerful web scraping mode.

Paste the URL into the "Website" box and click "Save URL" to move on

Tips!

Please note that this website is the news page from Twitter without login. If you want to extract data behind a login, please refer to the following tutorial:

Extract data behind a login

2. Set "Scroll down" - load more data from the listed page

Check the box for "Scroll down to bottom of the page when finished loading"
Set "Scroll times" as "20" and "Interval" as "3" second (This is for demonstration, and you can set the numbers based on your needs)
Select "Scroll down to the bottom of the page" as the "Scroll way" and click the "OK" button
"Interval" is the time interval between every two scrolls.

For some websites like Twitter, clicking the next page button to paginate is not an option for loading content. To fully load the listings, we need to scroll the page down to the bottom continuously. Theoretically, the higher the number we input for "Scroll times", the more data we can extract.

Tips!

Most social media websites use scroll-down-to-refresh to view more data. Learn details of dealing with infinite scrolling.
We suggest better set a relatively higher value of "Scroll times" if you need more data.

3. Create a "Loop Item" and extract data - loop extract each tweet

Select the 1st tweet block in the list, and then select the 2nd block, make sure the blocks are in the same shape
Click "Extract text of the selected elements" on the "Action Tips" panel

Octoparse will automatically select other similar items and create a "Loop item" list. Remember to select the whole block when you extract data from the list page.

You may find some items undetected, enter this modified XPath of the "Variable List" to improve accuracy when locating elements: //li[@data-item-type='tweet']

Delete the auto-generated data field
Click on the data you want to extract on the 1st block
Click "Extract text of the selected element" on the "Action Tips" panel

Rename the field name column by the predefined names or inputting on your own
Click "OK" to save

Tips!

If you have a task that Octoparse only extracts the first item and keeps producing duplicates, you may need to revise the “Loop Item” you create in the task. In this case, if you only select the name to create a loop and select data outside the name area to extract, Octoparse may extract the 1st item and duplicate. For more details, you can read the following article:

Why does Octoparse only extract the first item and duplicate?

4. Use Regular Expression - clean and reformat data if needed

Regular Expression aims at reformatting data after data extraction in Octoparse. For example, if you want to delete words like "Reply", "Retweet" and "Like" in this case, you can use Regular Expression to get the specific digit value by trimming the strings.

If the result already satisfies your needs, you can just skip this step.

Select the "Reply" row, click the "Customize data field" icon, select the "Refine extracted data" option and click "Add step"
Click "Replace" and paste the "Reply ***" with all space values from extraction data "Reply 856" into the "Replace" box.
Click "OK" to save
For the rows of "Retweet" and "Like", you can just repeat the steps above

Tips!

The value you will enter into the "Replace" box must be copied with all original space value. In this step, *** just means space value.

You can also reformat values in "Retweet" and "Like" rows like this step if needed.

Read more about 8 data re-format options

5. Start data extraction - run your task and get data

Click "Save"
Click "Start Extraction" on the upper left side
Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)

For a premium user, Cloud Extraction is highly recommended.