Add custom User Agent

October 25, 2021 03:17
Updated

Sometimes you will encounter the problem that some websites cannot be opened in the Octoparse but work well in the browser. This may be related to the browser User Agent (UA), which is a string to tell the target website what kind of device you are accessing the page with. Octoparse offers some default UAs for you to select from but sometimes they don't work. In this tutorial, we will show you how to add your browser's UA to Octoparse to open the target website.

You need to do 2 steps to add a custom UA

1. Get a UA

2. Put the UA in the Octoparse

1. Get a UA

Method One:
Take Chrome (Google) browser as an example. Enter "chrome://version/" in the address bar, and the following information will appear. You can copy the information after [User-Agent] in the yellow box and paste it into the Octoparse.

However, some browsers do not support the about: version method, such as Firefox (Firefox) and Edge browsers. Then the second method below can be used, and the second method is also applicable to the Chrome browser.

Method Two

Open the website and press F12 on the keyboard

Press F12 on the keyboard to open the developer tool console or you can also right-click on the webpage and choose "Inspect" to access the console.

(Firefox)

(Chrome)

Refresh the web page
Click the [Network] option to open the following interface.

(Firefox)

(Chrome)

The Network tab shows various requests sent by the browser.

Click on one of the requests and then scroll the right part to find the User-Agent in the Headers

Headers means the message header. You can find User-Agent information in this part of the content. If any message headers do not contain UA, change the request and continue searching.