During data extraction, you may encounter such a situation - there are duplicates in the data output.
Octoparse provides two ways to deal with duplicates if you want to remove them.
- Remove duplicates based on the whole line of data (default)
- Remove duplicates based on specific data fields (manual setting, and supports 8.1.16 and version above)
1. Remove duplicates based on the whole line of data (default)
When data extraction is completed, Octoparse has a default duplicates removal setting.
If all the data fields in a certain line of data are the same as the other lines, then this line of data will be recognized as a duplicate. After removal, Octoparse will only keep the 1st data line extracted.
Example: The 1st and the 4th line of data below have the same value for each data field, so they are duplicates. After Octoparse default removal, only the 1st line of data will be kept.
2. Remove duplicates based on specific data fields (manual setting)
Note: This feature is for Octoparse 8.1.16 and above.
Mechanism
When you're customizing the task workflow, you can set it to remove duplicates based on certain duplicates.
Duplicates removal will be based on a certain or many data fields to check if the values of those selected data fields are the same as the other lines'. This setting will only compare the selected data field(s). As long as the values of selected data fields are the same, the data will be recognized as a duplicate. Other unselected data field(s) will not be considered. After duplicates removal, Octoparse will only keep the 1st data line extracted.
Example 1: If we select "Field2" to compare for duplicates removal, then the 1st, 2nd, 4th line of data have the same value for "Field2". In this case, the 1st, 2nd, 4th line of data will be recognized as duplicates. After removal, Octoparse will only keep the 1st data line extracted.
Example 2: If we select "Field3" and "Field4" to compare for duplicates removal, then the 1st and the 4th line of data have the same value for "Field3" and "Field4". In this case, the 1st and the 4th line of data will be recognized as duplicates. After removal, Octoparse will only keep the 1st data line extracted.
Where to remove duplicates?
Steps:
1. Set up the task and the data fields you need to collect
2. Click the icon 
on the right top corner of the "Data Preview" panel first. Then choose "Remove duplicates" to enter the setting page
3. Select the data field(s) you want to compare for duplicates removal. After selection, click "OK" to apply settings.
Note: Click the icon again. You can see the setting is saved.
Special Note:
For Cloud runs, it will only compare the historical data under the same duplicates removal settings to remove duplicates.
Here is an example.
Let's set your 1st duplicates removal setting as A (e.g. select "Field1" to compare), you'll have the 1st batch of Cloud data.
Then, you go back to your task and modify the setting to B (e.g. select "Field2" to compare), and you'll get the 2nd batch of Cloud data. This batch will not be compared with the 1st batch for duplicates removal.
After that, if you change the setting back to A (e.g. select "Field1" to compare), you'll have the 3rd batch of Cloud data and it will be compared with the 1st batch of Cloud data for duplicates removal. Duplicates will be removed automatically. It won't be compared with the 2nd batch for duplicates removal.
If anything is not clear enough or you have a specific use case but not sure if this feature can help or not, feel free to submit a ticket to reach us for assistance!