By now, you've finished building your first scraping task and know how to run the task to get the data you need. Let's take it to the next level and find out how you can make your daily scraping routines more effective and efficient using task scheduling, auto-data export, and API.
If you are planning on getting data extracted on any regular basis, task scheduling is exactly what you need and can save you a lot of time. You can schedule your task to run once, on a recurring schedule, or even run repeatedly, such as every 1 min, 5 mins, 10 mins, or 30 mins.
1. Find your task on the Dashboard, click Not Set under the Next Run column. The first one is for Cloud schedules and the second one is for local schedules.
2. Choose how often you would like to run the task.
3. For recurring crawls, select the day of the week/day of the month and the time of the day to run your task.
For repeating crawls, select the desired time interval.
4. You can also save the settings for later use. Give the settings a name and click Save. This way, you can always select the saved schedule setting and apply it directly to any other task.
5. After everything's done, click Schedule ON to start running the task on the schedule right away.
6. When a task is scheduled, you'll see the next run time on the Dashboard.
You can easily turn it ON and OFF by clicking the next run time on the Dashboard, there you can choose Schedule ON or Schedule OFF.
Auto-data export (for Cloud data)
Data export to the database can also be automated and scheduled. If you need to export data to your databases on a regular basis, data export scheduling can save you tons of work.
1. Load the cloud data for your task.
2. Click on Export Data
3. Click open Auto-export to database, then select the type of database you have.
4. Complete the information to connect to your database. Click Test connection to test if the database is connected successfully. Then, click Next to proceed and map the data fields, and choose the desired time interval for the export.
5. Lastly, click Next to finish the process.
6. You can find your auto-export tasks in the Database Auto-export Tool
Connect using API
With the Octoparse API, you can run scraping tasks, retrieve extracted data, and even edit your tasks programmatically via coordinating with your own application.
Check the API documentation for details.
Connect using Zapier
You can connect the data scraped with other applications (e.g., Google Drive, Google Sheet, Dropbox) easily without coding by using Zapier. Check this to find out more details: How to Connect Octoparse with Zapier