IP rotation

Some websites can be very sensitive to web scraping and take some serious anti-scraping measures like IP blocking to stop any possible scraping activities. Therefore, IP rotation can be very useful during data mining.

What does Octoparse offer?

1. Custom proxies 

Custom proxies can be added for local extraction only at the moment. More details can be found here: Set up IP proxies

Please note that Octoparse does not provide proxies. To obtain external proxies, there are many free as well as paid proxy servers available around the web. 

2. IP Rotation

The Octoparse Cloud service is supported by thousands of cloud servers, each with a unique IP address. When an extraction task is set to execute in the Cloud, the task will be split into sub-tasks, and each sub-tasks will be run with a Cloud server simultaneously. So requests are performed on the target website through various IPs, minimizing the chances of being traced and blocked by the target website. The IP pool is constantly being updated. 

What can I achieve with cloud extraction?

1. Extraction Speed Up

There are 6 to 20 cloud servers scraping the data simultaneously. So the same set of data in the cloud can be scraped 6 to 20 times as fast as with local extraction.

2. Avoid Captcha

More IPs generally mean less likely to be traced/detected, hence less Captcha. 

(Know more about the benefits of Octoparse cloud service)