Sometimes you will encounter the problem that some websites cannot be opened in the Octoparse but work well in the browser. This may be related to the browser User Agent (UA), which is a string to tell the target website what kind of device you are accessing the page with. Octoparse offers some default UAs for you to select from, but sometimes they don't work. In this tutorial, we will show you how to add your browser's UA to Octoparse to open the target website.
You need to do 2 steps to add a custom UA
1. Get a UA
Take Chrome (Google) browser as an example. Enter "chrome://version/" in the address bar, and the following information will appear. You can copy the information after [User-Agent] in the yellow box and paste it into the Octoparse.
However, some browsers do not support the about: version method, such as Firefox (Firefox) and Edge browsers. Then the second method below can be used, and the second method is also applicable to the Chrome browser.
- Open the website and press F12 on the keyboard
Press F12 on the keyboard to open the developer tool console or you can also right-click on the webpage and choose Inspect to access the console.
- Refresh the web page
- Click the Network option to open the following interface.
The Network tab shows various requests sent by the browser.
- Click on one of the requests and then scroll the right part to find the User-Agent in the Headers
Headers means the message header. You can find User-Agent information in this part of the content. If any message headers do not contain UA, change the request and continue searching.
- Copy the UA from the user-agent line
2. Put the UA in the Octoparse
- Open the task, click the setting button to enter the settings page
- Click Add custom User Agent
- Name the User Agent as you want
- Paste the UA under the User Agent string and click Add to save
- Select the added UA from the Browser Ver. box
- Click Save