Scrape video information from YouTube

December 15, 2020 01:32
Updated

In this tutorial, we are going to show you how to scrape video information from YouTube.

Also, you can go to "Task Templates" on the main screen of the Octoparse scraping tool, and start with the ready-to-use YouTube Template directly to save your time. With this feature, there is no need to configure scraping tasks. For further details, you may check it out here: Task Templates

If you would like to know how to build the task from scratch, you may continue reading the following tutorial.

To follow through you might want to use the URL in this tutorial: https://www.youtube.com

Here are the main steps in this tutorial: [Download task file here ]

1. "Go To Web page"- open the targeted web page

Click "+ Task" to start a task using Advanced Mode
Paste the URL into the "Website" box and click "Save URL" to move on

2. Create a "Loop Item" - loop enter searching keywords

We can customize our "Text list" to create a loop search action. Octoparse will automatically enter every keyword in the list into the search box, one line each time.

Drag a "Loop Item" action into the workflow designer
Go to loop mode and select "Text list"
Click "a" to enter the keywords list with one keyword per line. Here we'll enter "Big Data" and "Machine Learning" as examples
Click "OK" and "OK" when you finish entering. Then you can see your keywords in the "Loop Item"
Click on the search box on the homepage in the built-in browser and select "Enter text” on the "Action Tips" panel

When you click on the input field in the built-in browser, Octoparse can detect that you select a search box, the "Enter text" action will automatically appear on the "Action Tips" panel.

Input the first keyword "Big Data" on the "Action Tips"
Click "OK", then the"Enter Text" action will be generated in the workflow
Drag the "Enter Text" action into the"Loop Item. Click on the "Enter Text" action
Go to "Loop Text" and select "Use the text in loop item to fill in the text box" and click "OK" to save
Click the search button of the web page and select "Click button” on "Action Tips", you will notice the "Click Item" action is added into the workflow.

3. Dealing with infinitive scrolling

In this case, pagination is not an option for loading the searching results. We will need to scroll down to the bottom of the page continuously to load all the contents.

Check "Scroll down to bottom of the page when finished loading" under "Advanced Options"
Set "Scroll times" and "Interval" as you need
Select "Scroll down to bottom of the page" as "Scroll way"
Click the "OK" button to save the result

Tips!

Make sure that you input "Scroll times"; otherwise, Octoparse wouldn't perform the scroll-down action. We suggest it is better to set a relatively higher value of "Scroll times" if you need more data.
Most social media websites use scroll-down-to-refresh to view more data. Click here to learn more about: Dealing with infinite scrolling.

4. Create a "Loop Item" - loop extract each item

When you create a list of items to scrape a website, sometimes the list may include several "Ads" items. To exclude the promotional video in this case, we can start building the "Loop Item" from the second row of the products on this page.

Select the second block in the built-in browser

We need to make sure the whole block of the first video item is covered in blue when you curse over your mouse. Only in this way, we could see the whole item block is highlighted in green after clicking, covering all the information like video title, channel name, total reviews...etc.

Click the third and fourth whole video item, until Octoparse identifies all other videos

Octoparse will automatically recognize the other blocks and highlight them in green. (If not, keep clicking on the next one till all of them are selected)

Click " Extract text of the selected element " on the "Action Tips" panel

Tips!

Normally we can just click "Select all sub-elements" on the "Action Tips" panel, but under certain circumstances (like this case), Octoparse only recognizes the sub-elements in the second block but fails to do that in other blocks. Thus, we’ll create a loop at first, and select the data of each block for extracting manually in the next step.

5. Extract data - select data you need to scrape

Click the data you need in the item block which is highlighted in red
Click "Extract text of the selected element" and rename the "Field name" column if necessary

Rename the fields by selecting from the predefined list or inputting on your own.

Click "OK" to save

6. Start extraction - run the task and get data

Click "Save"
Click "Start Extraction" on the upper left side
Select "Local Extraction" to run the task on your computer, or select "Cloud Extraction" to run the task in the Cloud (for premium users only)

For a premium user, Cloud Extraction is highly recommended.