when I am trying to paginate on this site. The paginating is a bit difficult to me, because there is no next button, after display for 20 pages, then click on “..” button to display another


1 comment

  • Scarlett

    For national tool warehouse: https://www.nationaltoolwarehouse.com/OurManufacturers.aspx?m=55045, The reason that the pagination doesn't is you didn't use the right XPath. The correct XPath is "//tr[@class="paging"]//span/../following-sibling::td[1]//a". Your XPath didn't start with the current page, which is why it will go back to the previous page. 

    The current page has a unique identifier called the “span” element.

    You should tell Octoparse to loop click the page after the current page with "span" so you need to locate the "span" first and then move up to its parent element, and then locate its sibling element.

    For the second URL, it's caused by failing to return to the manufacturer selection page. All you need to do is to add a click back action to tell Octoparse when it finishes scraping data from one manufacturer, it should go back to the manufacturer page.

    For the testing purpose. I set the pagination time to 2. You can untick the box when you start extracting data.

    Comment actions Permalink

Please sign in to leave a comment.