Articles in this section

See more

Lesson 3: Refine your data

  • Updated
Follow
When you are working with data, the second most important thing next to extracting data is cleaning your data.  In this lesson, we will walk you through some practical tips for how you can refine your data set after having the data extracted. 
 
 
Let's cut to the chase now!
 

Rename/move/duplicate/delete a field

As soon as you have the data extracted and shown in Data Preview, you can now look through the data set and start organizing your data. A few typical things you can do to refine your data set include renaming the fields, reordering the columns, duplicating data fields, and deleting the fields that are not required for your project.

To rename a field, click the pencil icon next to the field name, then type in the new name directly. Note that you should only use numbers, letters, and "_" for field names.

renameafile.gif


To move a field, place your cursor at the front of the field and when the hand.png shows up, drag and drop the field to the right spot. 

movefield.gif

 To duplicate a field, click on the show more icon and select "Copy". The selected field will be duplicated automatically. 

duplicateafield.gif

To delete a field, click on the show more icon and select "Delete"

deletefield.gif

Tips!

  • You can also rename/move/duplicate/delete any data fields by going to "Action Settings" for the "Extract Data" action of the workflow. 

extractdatasetting.gif

  • If you have more fields to delete, you can also batch delete the fields. Click on the "Action settings" icon for the "Extract Date" action. On the Setting Panel, click the "Batch delete fields" icon, select the fields you'd like to delete, and then click the "Delete" button. 

batchdelete.gif

 

Clean data

Octoparse provides many different ways for you to clean your data. For example, you can replace a text string, trim extra spaces, add a prefix/suffix, replace a string with RegEx, reformat date/time and more. You can clean any single data field in one or more ways until the data meets your requirements. Some of these may require you to deal with Regular Expression with which you can use the Octoparse RegEx tool for assistance. 

In Data Preview, right click the show more icon for the data field you'd like to clean, select "Clean data".clean_data.png

Click "Add step", and then select what you'd like to do with the data. You can keep working with the data by adding more steps until the data meets your requirements. 

addstep.png

 

  • Replace: replace the specific string(s) in the extracted data with the new string(s) that you want.
  • Replace with Regular Expression: use a specific regular expression to replace the matched string(s) in the extracted data with the string(s) that you want.
  • Match with Regular Expression: use a specific regular expression to pick up the matched string(s) from the extracted data.
  • Trim spaces: remove the unwanted space(s) from the start or/and the end of the data extracted.
  • Add a prefix: add a string/strings to the front of the data extracted.
  • Add a suffix: add a string/strings to the end of the data extracted.
  • Reformat extracted date/time: shift the extracted date/time into one of the 14 built-in formats, or into your own customized format.
  • HTML: convert some specific HTML tags into plain texts automatically. For example, transcode "&gt" into ">" and "&nbsp" into a space.

Tips:

To learn more about reformatting data and RegEx toolexternal-link-symbol-1.png with Octoparse? Check here!

  

Capture HTML code

When auto-detect is used to capture any data from a web page, Octoparse automatically extracts the text and the URL of the elements that you've selected. You can customize the data field and tell Octoparse to extract any HTML code. 

In Data Preview, right click the show more icon and select "Customize field". 

customizefield.png

From the "Customize field" setting panel, select what you'd like to extract. 

customizefield2.png

 

Extract page-level data and date & time

Octoparse offers a number of pre-defined data fields that you can use to capture page-level data, current data & time, or any fixed value conveniently. 

  • Current date & time: the date and time of when the data is extracted from the web page
  • Page-level data: page URL, page title, meta keyword, meta description, and HTML source code
  • Fixed value: any fixed value you define 

Click on the + sign at the upper right hand corner of Data Preview. Select any pre-defined data fields that you'd like to add to the data set. 

 extractmeta.png

Until now, we have gone through all the steps about building and refining the workflow, it's time to start a test run! >> Lesson 4: Test-run the task

 

Artículo en español: Lección 3: Refina tus datos

También puede leer artículos de web scraping en el sitio web oficial.

Related articles