Similar to how you will manually scroll down the page, Octoparse does it the same way with the proper settings. Basically, all you need to do is to tell Octoparse which page to scroll, how many times to scroll, and the time interval between every two scrolls.
In this tutorial, we are gonna show you how to deal with infinitive scrolling in Octoparse, you may want to use this URL to follow through https://biomarket.com.ar/product-category/almacen/desayuno/
1) Use the auto-detect algorithm to deal with it
1. Select "Auto-detect web page data" on the Tips panel. Octoparse will start detecting the page data and let's wait for it to finish.
2. Modify the scroll settings
Click "Edit" under "Add a page scroll" and set up the scroll way, repeat times, and wait time as needed. Click "Confirm" to save the settings. Make sure to set up enough scroll-down times and proper intervals between two scrolls.
3. Create the workflow with the settings
Click "Create workflow" on the Tips to generate the workflow with settings.
4. Check if the Loop Item created can locate all the elements
You can go to the settings of the Loop Item to see if all the elements are located
Note to check if the loop mode of the Loop Item is selected as "Variable List". In case it is "Fixed List", please change to "Variable List" and modify the XPath.
Check the tutorial below for more details:
1. Double click the Go to Web Page action(or a Click Item) or click the icon to access the setting menu. Then find the Page scroll-down options by clicking open the "After loading page" section.
2. Check the box for "Scroll down the page after it is loaded", and set up the scroll way, repeat times, and wait time as needed.
Find more details about page scroll-down function at Page scroll-down.
It is easy to set up for infinitive loading but to find the most appropriate settings, you might want to test running the task to see if you’ve assigned enough scroll times and if the scrolling is working with the right pace.
Tutorial en español: Tratar la paginación (desplazamiento infinitivo)
También puedes leer más tutoriales de web scraping en sitio web oficial
If you need any help with task configuration or data collection, submit a ticket to our support team! We'll get back to you soon.