How does the concurrent crawler work? Am I able to spin up 100 crawlers in the cloud to view videos for a set time and then refresh the page to view more?


1 comment

  • Fergus

    The standard plan includes 100 tasks(crawlers), which means you can have up to 100 tasks in total on your dashboard.
    If you want to build new tasks after you reach the limit. Please delete some of your tasks and build the new one.

    For standard plan, you can have the unlimited concurrent local run, if you are using cloud extraction, you can have 6 concurrent Cloud Extractions, the number 6 is not referring to 6 tasks, but 6 cloud servers, cloud extraction speed up the scraping process by splitting the task into several sub-tasks to run on multiple cloud servers simultaneously, to learn more, please check: What is "task split" on Cloud Extraction? (Speed up Cloud Extraction) 

    You can also set up schedules for the tasks to run: Run/Schedule tasks in the Cloud

    As for "view videos for a set time and then refresh the page to view more", could you please elaborate a bit more? Any examples or screenshots would be very helpful.

    Comment actions Permalink

Please sign in to leave a comment.