Crawl whole website


1 comment

  • Kara


    Thank you for reaching out.

    Usually to scrape data from one website(or URLs under one domain) will use one task/crawler. Because one task/crawler can only scrape data from pages with a similar page structure. And in one task, we usually need to configure the specific data fields we need to grad, I'm not sure what do you mean by "everything" here, if we are not choosing specific data fields, we can only scape the source code of the whole page(HTML code), or the "text" of the whole page, and the "text" extracted in this way would be quite unorganised and most likely useless.

    Best regards,

    Comment actions Permalink

Please sign in to leave a comment.