Locate and scrape an element via nearby text
FollowOctoparse tracks data with XPath but data can change location within a web page. To tackle this, we will show you how you can extract data more accurately by associating it with a text nearby.
First, let’s look at an example of when this technique can be useful.
In the example image above, the value for "Brand" is located next to the words "Brand". Similarly, the value for "Item Weight" will always be found next to the words "Item Weight". The same pattern should apply to the rest of the list.
While "Item Weight" might change places from the third row to the fourth row of the list, its associated value should always be found next to it. Therefore, a more consistent way to find and capture the associated values of any elements is really to first look for where the words are, then locate the data next to it. In this example, instead of trying to find the value "10 pounds" directly on the page, we can get it captured more accurately when we relate it to the text of "Item Weight".
Follow the steps below to see how it is done:
1) Click on "10 pounds" to capture the value for "Item Weight". Once extracted, click on the Action Settings icon to check the data field(s)
2) Click "Customize XPath"
3) Find the XPath relating to the text of the target data field
- Now, open the page in Chrome, right-click to inspect the target data
- Notice the actual words of "Item Weight" can be found within the <th> tag while its associated value is found within the <td> tag right below it.
- Once we see the pattern, we can write an XPath to look for the value of "Item Weight" relative to where we will actually find the words: "//th[contains(text(),'Item Weight')]/following-sibling::td[1]" - This XPath expression is telling the program to look for the <th> tag containing the text of "Item Weight" then find the first <td> tag located right below it. And this will give exactly what we want, the associated value of "Item Weight".
- Input the new XPath to the text box for "Matching XPath", click "OK" to save the settings.
Now, Octoparse will always look for the associated value of "Item Weight" according to where the words "Item Weight" are showing on the web page. Apply this technique to similar fields on the list can help reduce the chance of element not found exceptions.
|
Tips! Following-sibling is very often used for finding an element located next to another designated element. Learn more about XPATH |
here!
If you need any help with task configuration or data collection, submit a ticket to our support team! We'll get back to you soon.
Tutorial en español: Localizar y scrapear un elemento a través del texto cercano
También puedes leer más tutoriales de web scraping en sitio web oficial
Author: Kara
Editor: Yina
