Scraping a master and child records

Follow

flanagap

• November 07, 2020 05:50

I am new to web scraping and have an atypical requirement. I would be grateful if someone could point me in the right direction (or tell me it can’t be done). Instead of capturing data on multiple records I would like to capture data for a single record spread over three web pages. The web pages are set for up for a many-to-one relationship but in this case there is always exactly one child record. Page one displays the data for the parent record (which I need to capture) and has a link that takes you to page two. Page two contains a clickable list of the (one) child records. Clicking on the ‘child’ takes you to the detail record for the child record which I also need to capture. Any help would be greatly appreciated.

0

• 

  Brian

  • November 09, 2020 09:18

  Hi flanagap,

  Thank you for reaching out.

  We'll need to check the source code of the webpage to evaluate the feasibility of your target data. Could you please kindly send us the URLs for further investigation? And let us know how to navigate to the target data with screenshots. Thank you for your time.

  Best regards,

  0

  Comment actions Permalink
• 

  flanagap

  • November 09, 2020 15:37

  Thank you for your offer Kara but unfortunately the web site contains confidential data (to which I have been granted access).
  I know this is a lot to ask but is there anything I could look for in the code or can you point me to some reference material? Patrick

  0

  Comment actions Permalink
• 

  Brian

  • November 12, 2020 08:27

  Hi flanagap,

  Thank you for your reply.

  Here are the tutorials that might be helpful:

  https://helpcenter.octoparse.com/hc/en-us/articles/900001523803-Click-each-link-in-a-list-and-scrape-data-from-new-pages

  https://helpcenter.octoparse.com/hc/en-us/articles/360032242511-How-to-scrape-from-the-list-page-and-detail-page-at-the-same-time-

  Best regards,

  0

  Comment actions Permalink

Please sign in to leave a comment.