Hi. How are ya. I am trying to use your program to crawl some news articles. The problem is that there is no next element in the page. there is a button says show more. it keeps generating more news in the same page. So, with a simple a loop, i had A LOT of duplication. I will give you the url to check by yourself. If there is a way to just select the last x elements with each pagination, this will help me a lot. Thanks again.
Here is the link: https://www.aljazeera.com/topics/regions/middleeast.html
Please sign in to leave a comment.