Sometimes you will encounter the problem that some websites cannot be opened in the Octoparse but work well in the browser. This may be related to the browser User Agent (UA), which is a string to tell the target website what kind of device you are accessing the page with. Octoparse offers some default UAs for you to select from, but sometimes they don't work. In this tutorial, we will show you how to add your browser's UA to Octoparse to open the target website.

You need to do 2 steps to add a custom UA

  1. Get a UA

  2. Put the UA in the Octoparse


1. Get a UA

Method One:
Take Chrome (Google) browser as an example. Enter "chrome://version/" in the address bar, and the following information will appear. You can copy the information after [User-Agent] in the yellow box and paste it into the Octoparse.

1.png

However, some browsers do not support the about: version method, such as Firefox (Firefox) and Edge browsers. Then the second method below can be used, and the second method is also applicable to the Chrome browser.

Method Two:

  • Open the website and press F12 on the keyboard

Press F12 on the keyboard to open the developer tool console or you can also right-click on the webpage and choose Inspect to access the console.

3.png

Firefox

2.png

Chrome

  • Refresh the web page

  • Click the Network option to open the following interface.

network.png

Firefox

network_Chrome.png

Chrome

The Network tab shows various requests sent by the browser.

  • Click on one of the requests and then scroll the right part to find the User-Agent in the Headers

Headers means the message header. You can find User-Agent information in this part of the content. If any message headers do not contain UA, change the request and continue searching.

Headers.png

Firefox

Chrome.png

Chrome

  • Copy the UA from the user-agent line

6.png

Firefox

user_agent.png

Chrome


2. Put the UA in the Octoparse

  • Open the task, click the setting button to enter the settings page

10.png
  • Click Add custom User Agent

  • Name the User Agent as you want

  • Paste the UA under the User Agent string and click Add to save

8.png
  • Select the added UA from the Browser Ver. box

  • Click Save

9.png
Did this answer your question?