We will introduce how to keep your eyes on the cryptocurrency market with a simple and easy method -- web scraping. I will show you how to extract market data with NO Tech Skills required. You can save scraped data to your database for analysis access. In this tutorial, I will scrape bitcoin as an example. And you can use this method to scrape as many coins as you can.
Step One: Set up a new Task
To set up a new task, we need to get the URL of the target website. Open Coinmarketcap.com with a web browser. Navigate to Botcoin page. Then set up a 12 month Time range from historical data. Then copy this URL. We will use it to build a web crawler.
- Open Octoparse, choose “Advance Mode,” and enter the URL we just prepared into the box.
- Click “Save URL” on the left corner. This will bring you to the Bitcoin historical data page with Octoparse built-in browser.
Step Two: Create an extraction list
- Click the date from the table. Octoparse will find similar elements with the same attributes. The selected item is highlighted in green color. Octoparse should be able to find other similar elements and put them in red color. As you can see, Octoparse finds the date column, but we want Octoparse to select the table row, in this case, choose “TR” at the bottom of the action panel
- Now, Octoparse finds the first row, including all subfields. In this case, follow the guide, choose “Select all sub-elements.” Now the first row has been selected successfully.
- Confirm its selection by clicking the “Select-ALL” Command.
- Now, all data has been select successfully, and you can cross out undesired fields from the action panel. In this case, choose “Extract Data in the Loop.”
- Click “Save” to Save the step
- The action has been added to the workflow box. We have a loop list with 365 listings in the loop list. Octoparse will click through each listing and specific selected data fields.
Step Three, Name the extraction Field
- Now we need to Edit the Fields Name accordingly from the Setting Area. You can choose a name from the drop-down menu. Alternatively, type and rename the fields.
- After that, go ahead and click “Save.”
Step Four, run the task and get data
- After finishing setting up the rules, we can run the task by clicking “start extraction.”
- Then the Select “Local extraction” to run the task.