How to scrape more information which needs to be clicked to show?

Sometimes, you may encounter such a situation that you need to click some items first to display the information hidden behind. How to scrape this kind of data? 

Let's take this website as an example: https://www.nastygal.com/black-croc-structured-mini-backpack/AGG77323.html

In this website, if you want to view data under the fields of "Details and Care" and "Shipping and Returns", you need to click the tags respectively. 

In this case, if we want to extract data under "Details and Care", how are we going to achieve it?

Here are two ways to help you achieve it:

1. Click the tag to display the data first, and then extract data you want
2. If the data can be found in the source code of the web page, you can extract data directly

1. Click the tag to display the data first, and then extract data you want

• Click the "Details and Care" tag
• Select "Click element" on the "Action Tips" panel
• Uncheck "Auto Retry" for the "Click" item and click "Save"
  • Octoparse has detected AJAX automatically. You can input the AJAX timeout based on your network condition.
• Then, select the data and click "Extract text of the selected element" on the "Action Tips" panel
• Click "OK" to save

2. If the data can be found in the source code of the web page, you can extract data directly

• Open the web page with your browser, and then right-click to select the option of "Inspect Element" to view the source code

From the source code, we can find that even though we do not click the "Details and Care" tag, we can still find the corresponding data under the tag.

Now, we can go back to Octoparse and continue scraping.

• Click the switch button  in the top right corner of the built-in browser to switch to the browser mode
• Click the "Details and Care" to display the content
• Click the switch button  again to switch back to the edit mode
• Then, you can select the data and click "Extract text of the selected element" on the "Action Tips" panel
• Click "OK" to save

Author: Vanny

Editor: Yina

Was this article helpful? Contact us any time if you need our help!