Skip to main content
A crawler is a content source attached to an engine. It produces the documents that end up in the search index — either by fetching pages on a schedule, or by accepting documents you push through the API. An engine can have one or more crawlers (for example, a website crawler plus a feed for product data). Every API path that writes to the index takes a crawlerId: Index Management uses /index/{crawlerId}/documents to push, patch, or delete documents, and the queue endpoint at /crawler/{crawlerId}/paths adds URLs to a fetch queue. Crawlers themselves are created and configured in MyCludo under Crawlers. See Pushing Content for the end-to-end workflow.