he problem is that there is no next element in the page. there is a button says show more. it keeps generating more news in the same page.

Comments

1 comment

  • Fergus

    You can follow these tutorials to solve the issue. In this way, Octoparse will load all the content before starting to extract.
    Infinitive Scroll has setup but no new elements added to the list?
    Dealing with Infinitive Scrolling/Load More

    Or you can use the XPath below as you loop item XPath. 

    //div[@class="topics-sec-block"][position()>last()-1]/div//h2/parent::a

    Make the workflow look like this. In this way, Octoparse will click through the listing page to the detail page to extract the content. Every time when it clicks the load more button.

    0
    Comment actions Permalink

Please sign in to leave a comment.