I am trying to make a crawler for the website Job Bank (Canada).
The issue i am facing is after extracting the data I can see a lot of duplicate fields.
For example, I do not want any duplicate email addresses while running the crawler. is it possible?
I know I can extract data without the duplicate data's but I need to remove the duplicates while the crawler is running.
I hope I was able to make you understand.
Please sign in to leave a comment.