Hello, I am trying to create a crawler to do the following. 1. Go to http://costcode.net/Equipment/

Follow

Scarlett

• May 13, 2020 07:04

Hello, I am trying to create a crawler to do the following.
1. Go to http://costcode.net/Equipment/
2. Click the link for each item (each manufacturer) (i.e. ADDCO)
3. Once the manufacturer is selected, then each sub category should be selected (i.e. http://costcode.net/Equipment/addco/brick-fixed-location-variable-message-sign/)
4. Then extract the information including MANUFACTURER, MODEL, DATE ADDED, COST CODE, HP, NOTES, FEMA RATES (all) and the image.
5. Repeat for all other vehicles and manufactruers.

For some reason, mine is not working and I need guidance. Can you assist me?

0

• 

  Scarlett

  • May 13, 2020 07:07
  • Edited

  You have configured three paginations, which is not right.
  Please follow the steps below to configure a new task:
  1. Create a new task and save the URL: http://costcode.net/Equipment/
  
  2. Create a loop item to click each manufacturer: Lesson 5: Getting data - Click through links in a list and capture data from each item page
  Just click the first manufacturer, and select "Select all" on the Action tips:
  ​
  
  
  3. Create a loop to select all the subcategories. Just do as step 2.
  Click the first subcategory, and select "Select all"
  ​

  
  4. Extract the text you need.

  
  
  5. Modify the XPath of the second loop item into the variable list: 
  ​
  
  6. Uncheck the AJAX Load timeout for the click item:

  

  0

  Comment actions Permalink

Please sign in to leave a comment.