Crawler count for scraping product pages from search results

Follow

joshuaba

• January 27, 2021 02:42

I have an example:

Lets say I want to crawl all of the product pages details for all products that end up on the search result of "ABC toys" on walmart.ca.

Lets say the search results end up showing 8 products. 

Would the above task count as 9 crawler tasks? So the first crawler task is crawling the search results for "ABC toys" on walmart.ca, and then the remaining 8 tasks are to extract each product page one by one? 

0

• 

  Brian

  • February 01, 2021 11:08

  Hi joshuaba,

  Thank you for reaching out.

  Regarding your question, to scrape the search result of "ABC toys" on walmart.ca, you just need to build one task, since these pages share the same page structure, to learn more, please refer to:

  Dealing-with-pagination-with-a-Next-button-

  Click-each-link-in-a-list-and-scrape-data-from-new-pages

  Best regards,

  0

  Comment actions Permalink

Please sign in to leave a comment.