Why do I get so many duplicates?(V8.4)
FollowDuplicates in Octoparse mean the data lines that are the same in all the fields. You can delete the duplicates when exporting the data if there are only a few duplicates.
But if you only get a small number of valid data lines with a lot of duplicates, that can be really frustrating. In this FAQ tutorial, we will help you resolve the duplicates.
Error 1: When scraping multiple pages, Octoparse jumps back to previously scraped pages or keeps scraping the last page.
The auto-generated pagination XPath may not always work well. If you find Octoparse duplicates on several pages over and over again, you will need to make some adjustments.
Solution: Modify the XPath of the pagination to make sure it locates the next page button precisely.
- Click on Pagination
- Enter the new XPath and click Apply to save
Error 2: When scraping multiple pages, the AJAX timeout for pagination is too short to load the next page and Octoparse keeps scraping the current page data.
For pages that are loaded with AJAX, once the timeout is not set long enough, the next page could not be loaded, Octoparse would scrape the current page again, which would produce some duplicates.
Solution: Extend the AJAX timeout to make it long enough for the page to load.
- Click on Click to Paginate
- Select a longer AJAX timeout
Error 3: When scraping a list of items, Octoparse only scrapes the first row of data repeatedly or one data field is the same in all lines.
When looping through a list of items to get data, Octoparse may keep scraping from one item. Or other fields are gathered correctly from each item, but one or two fields are fixed.
This is because the Extract Data action is not associated with the Loop Item action. To associate the Extract Data and Loop Item, two options need to be selected.
1. Extract data in the loop in the Extract Data settings
2. Relative XPath in the setting of the data field
With the two options selected, the Extract Data and Loop Item are associated and Octoparse will scrape the data from each item in the loop.
Solution 1: Re-create the fields
- After checking the Extract data in the loop option, click Loop Item in the workflow, and then click Extract Data
- The first item will be highlighted and we can choose elements from the highlighted area to extract the text
Solution 2: Modify the XPath of the fields directly
- Click on Extract Data
- Click on More and select Customize XPath
- Tick Relative Xpath and enter the correct XPath
Method 1: Click "Extract data" then Customize Xpath
Method 2: Click Extract data, switch to the vertical view, and double click each field to customize Xpath, which is more convenient if you need to modify several Xpaths.
If you need any help with task configuration or data collection, submit a ticket to our support team.
Author: Joy
Editor: Yina
