IP rotation
FollowThere are some websites that might be very sensitive to web scraping and take some serious anti-scraping measures like IP’s blocking to stop any possible scraping activities. Therefore, using multiple IPs is quite useful during data mining.
What does Octoparse offer?
1. Custom proxies
Custom proxies can be added for local extraction only at the moment. More details can be found here: Set up proxies
Please note that Octoparse does not provide proxies. To obtain external proxies, there are many free as well as paid proxy servers available around the web.
2. IP Rotation
The Octoparse Cloud service is supported by thousands of cloud servers, each with a unique IP address. When an extraction task is set to execute in the Cloud, the task will be split into sub-tasks, and each sub-tasks will be run with a Cloud server simultaneously. So requests are performed on the target website through various IP’s, minimizing the chances of being traced and blocked by the target website. The IP pool is constantly being updated.
Why do you want to use Cloud Extraction?
1. Extraction Speed Up
There are 6 to 20 cloud servers scraping the data simultaneously. So the same set of data in the cloud can be scraped 6 to 20 times as fast as with local extraction.
2. Avoid Captcha
More IPs generally mean less likely to be traced/detected, hence less Captcha.
(Know more about the benefits of Octoparse cloud service)
