Crawler count for scraping product pages from search results
I have an example:
Lets say I want to crawl all of the product pages details for all products that end up on the search result of "ABC toys" on walmart.ca.
Lets say the search results end up showing 8 products.
Would the above task count as 9 crawler tasks? So the first crawler task is crawling the search results for "ABC toys" on walmart.ca, and then the remaining 8 tasks are to extract each product page one by one?
-
Hi joshuaba,
Thank you for reaching out.
Regarding your question, to scrape the search result of "ABC toys" on walmart.ca, you just need to build one task, since these pages share the same page structure, to learn more, please refer to:
Dealing-with-pagination-with-a-Next-button-
Click-each-link-in-a-list-and-scrape-data-from-new-pages
Best regards,
Please sign in to leave a comment.
Comments
1 comment