During your web scraping project, you may want to clean the data fields while doing the web scraping. Octoparse offers 8 data cleaning options for turning the extracted data into the format you need.

When should I refine the extracted data?

If you have the desired data format for a specific field, you can use our "Clean Data" function to refine the field within Octoparse. Octoparse would scrape and refine it directly during the scraping process. No need to reformat the field after exporting the data into an excel file.

How to refine the extracted data in Octoparse?

To access these features in Octoparse, you should follow the 4 steps below:

  • Select the data field to refine

  • Click on the "..." icon and select Clean data

  • Click Add Step

  • Select an operation to reformat the data

441.png

Tip:

In programming, a "string" refers to a collection of characters like letters, numerals, symbols, and punctuation marks. For example, " " (space) is a string; "Octoparse" is a string; and "Hello 2 *% World!" is also a string. A string can consist of no character as well. In other words, a string that contains no character is empty. If you replace a word with an empty string, colloquially, it is equal to saying that you delete the word.

You would see the word "string" in many function instructions of Octoparse's data reformat options. Suppose you see the word "string" there. In that case, you can use the corresponding options to deal with various character types in the data extracted, such as letters, words, sentences, numbers, spaces, symbols, and punctuation marks.

9 Data rereformatptions

  1. Replace

  2. Replace with regular expression

  3. Match with regular expression

  4. Trim spaces

  5. Add a prefix

  6. Add suffix

  7. Reformat date/time

  8. Timestamp conversion

  9. HTML transcoding

1. Replace

Function: Replace the specific string/s in the extracted data with the new string/s you want.

68.png

2. Replace with regular expression

Function: Use a specific regular expression to replace the matched string/s in the extracted data with the string/s that you want.

You can learn more about regular expression in W3schools.

67.png

3. Match with regular expression

Function: Use a specific regular expression to pick up the matched string/s from the extracted data.

You can learn more about regular expression in W3schools.

66.png

4. Trim spaces

Function: Remove the unwanted space/s from the start and/or the end of the data extracted.

If you want to delete the spaces amid the data, you can use Replace or Replace with regular expression.

64.png

5. Add a prefix

Function: Add a string or strings to the front of the data extracted.

65.png

6. Add suffix

Function: Add a string to the end of the data extracted.

68.png

7. ReReformatxtracted date/time

Function: Shift the extracted date/time into one of the built-in formats or into your own customized format.

68.png

8. Timestamp conversion

Function: Shift the Unix timestamp into your own customized format.

The Unix timestamp is a sequence of numbers that represents a specific date and time. This function will convert Unix time to a format that we can understand easily.

8.png

9. HTML transcoding

Function: Convert specific HTML tags into plain text automatically. For example, transcode "&" into a "&".

69.png

Tip: All the steps added can be edited and deleted here by clicking the edit and delete icons.

72.png

Octoparse Regex Tool

Octoparse also offers a RegEx Tool to auto-generate the regular expression that you need. Let's have a quick look at how to use Octoparse's RegEx Tool to generate and apply a regular expression. For example, here we want to pick up the numeral of star-rating from the outer HTML extracted.

  • Click Try RegEx Tool

  • Enter the match criteria: start with "src="", end with " " "

  • Click Generate to produce regular expression

  • Click Match to pick up the matched strings

  • Click Apply

  • Click Confirm to save the settings

339.gif

Click the link here for more information about the use of the Regex tool.

Did this answer your question?