How do I export data to a database?(Version 8)

January 04, 2021 08:38
Updated

Besides data export to formats such as XLS, CSV, HTML, etc, Octoparse also supports data export to your database (MySQL, SQL Server).

As for data export to the database, there are 2 methods we support:

Export extracted data to your database manually (For local and Cloud data)
Auto-export extracted data to your database with schedule (Database Auto-export Tool) (For Cloud data only)

This article will mainly introduce the 1st method - manual export to your database.

Note:

Our 8 Beta's database export feature supports MySql and SqlServer. If you need to connect with other databases or platforms (such as your own websites), please refer to Octoparse API.
If you need to export data to Oracle, please use the 7 versions.
Octoparse currently does not support data export to cloud databases such as AWS database, Azure, Cloud SQL database, etc.

Export extracted data to your database manually

Here are the prerequisites before we start:

A task with data extracted (both local and Cloud data are okay)
A table with data fields configured in your database to receive data

This tutorial will use the MySql database and a task's local extracted data as an example.

Now, view the local data of the target task.

Click the "Export Data" button
Choose the type of your database
Press "OK"

Fill up the required fields about the target database settings

After filling up the information, click "Test Connection" to see if you can connect to your database.

If the setting is alright, you will receive a "Connected" notice as above. Then, you can click "Next" to move forward.

Now, we need to configure data mappings.

Select an existing table from the Data Table drop-down options

Match the source data fields (data fields extracted) to the target data fields from the designated table
(Optional)Select any source data fields to use as an identifier (primary key)
After data mappings are configured, click "Export" and wait for the data export to finish.

Save Settings:

Once the configuration is completed, it can be saved for later use. You can name the setting first and then press the "Save" button to save it.

Next time, if you want to use your previous database export settings, you can just choose a saved setting to save time.

Export Settings:

You can set up the number of data lines to export per batch. Better to set up a smaller number when the size of one data line is large.

Tick "Ignore errors" and Octoparse won't stop exporting when it encounters errors.

Press the "Completed" button to exit after you receive the "Export Completed" message.

Then, you can go to your database and check if the data is delivered correctly.

Tips!

1. Once a data field is selected as a primary key, subsequent data records with the same primary key will overwrite the existing ones in the database.

2. If you have trouble delivering extracted data to your database, you can also choose to export data to CSV first. Then uploading the CSV file to your database.

Auto-export extracted data to your database with schedule

If you want to set up automatic data export to a database regularly, then you can check this guide for details: Database Auto-export Tool.

Should you encounter any problems with data export to a database, feel free to leave your message.

Author: Vanny

Editor: Yina

Export extracted data to your database manually

Auto-export extracted data to your database with schedule

Related articles