> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cludo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawler

> Understand how Cludo crawlers discover, fetch, and feed website or feed content into an engine's index, and how crawl schedules and sources are configured.

A **crawler** is a content source attached to an engine. It produces the documents that end up in the search index — either by fetching pages on a schedule, or by accepting documents you push through the API. An engine can have one or more crawlers (for example, a website crawler plus a feed for product data).

Every API path that writes to the index takes a `crawlerId`: [Index Management](/api-reference/v4/content/index-management) uses `/index/{crawlerId}/documents` to push, patch, or delete documents, and the queue endpoint at `/crawler/{crawlerId}/paths` adds URLs to a fetch queue. Crawlers themselves are created and configured in [MyCludo](https://my.cludo.com) under **Crawlers**.

See [Pushing Content](/guides/pushing-content) for the end-to-end workflow.