Many websites use a "Load More" or "Show More" button to load content continuously. Websites very commonly use this technique to create a better user experience.
Unlike Pagination with a "Next" button, the "Load More" button keeps adding more content onto one web page, making it trickier to scrape. In this article, I will show you how to deal with the "Load More" button in Octoparse.
You may need this example link to follow through:
1. Use Auto-detect to deal with the "Load More" button
Start the Auto-detect process, and you'll find the option to Click on a "Load More" button in the Tips Panel.
Click Check to see if the Load more button has been located correctly. If not, you can click Edit to choose the right button.
Click Edit to set up the number of clicks, which is how many times you want to click on the Load More button.
Click Create workflow to generate the settings
The workflow should look like the picture below:
2. Create a pagination action manually
Select the "Load More" button on the web page and choose Loop click single element
Set up a proper AJAX timeout (what is AJAX?)
1. If you only wish to click the "Load More" button for X times, click the Pagination box, tick "Repeats," and set Repeats to the number X.
2. If you find that the task gets many duplicates during scraping, you can drag the Loop Item out of the Pagination so that Octoparse will start to scrape after loading all the items.