Dealing with pagination (clicking on a "Next" button)

Pagination, also known as paging, is dividing a document into discrete pages, either electronic or printed. Setting up pagination is crucial to getting plenty of data from multiple pages. Common ways of pagination include paging with a "Next" button, a "Load More" button, or an infinite scroll.

This tutorial will introduce how to deal with pagination with a "Next" button in Octoparse.

There are two ways to set up pagination with a "Next" button:

1. Use the auto-detect algorithm to set it up
2. Set up the pagination manually

To follow through the tutorial, you may use the link below:

 https://www.amazon.com/s?k=sunglass&ref=nb_sb_noss_2

1) Use the auto-detect algorithm to set it up

•  Click the "Auto-detect web page data" on the Tips panel. 

• Click the "Check" button to inspect which button is identified as a "Next" button.

The button selected will be highlighted on the page.

If Octoparse fails to recognize the correct next page button, click "Edit" to select the next button manually. 

• Uncheck Add a page scroll and click Create workflow in the Tips panel, and you will see a workflow with pagination created.

• Click Pagination and Click to paginate to see whether Octoparse can move to the next page.

If Octoparse can go to the next page, your pagination has been set up successfully.

2) Set up the pagination manually

You can create the pagination manually if the auto-detection does not find the next page button.

• Click the Next button, choose Loop click next page in the Tips panel. 

You may check whether the pagination works appropriately with the same step in method 1.