> ## Documentation Index > Fetch the complete documentation index at: https://docs.reygrid.com/llms.txt > Use this file to discover all available pages before exploring further. # Crawl a website > Crawls the website from the given URL, following links on the same domain, and indexes each page. Pages already crawled are skipped. The crawl stops at your plan's page or storage limit. The request returns when the crawl is finished. ## OpenAPI ````yaml /openapi.yaml post /agents/{agentId}/sources/website openapi: 3.0.3 info: title: ReyGrid API version: 1.0.3 description: | ReyGrid API ## Persistent user conversations If you want to build a separate conversation system for each user, build a wrapper layer around the ReyGrid API. The wrapper should manage the application's own user IDs and map them to ReyGrid conversations. The wrapper is responsible for: - Managing and identifying users. - Storing the relationship between the application's userId and ReyGrid conversationId. - Persisting conversation IDs in the application's own database. - Reusing the stored conversationId when the same user continues a conversation. - Verifying that conversation IDs belong to the expected user before using them. ReyGrid's userId can also be used as a stable identifier for persistent conversations. However, if the application requires its own user and conversation management, it should maintain this mapping in its own database. servers: - url: https://reygrid.com/api/v1 security: [] paths: /agents/{agentId}/sources/website: post: summary: Crawl a website description: | Crawls the website from the given URL, following links on the same domain, and indexes each page. Pages already crawled are skipped. The crawl stops at your plan's page or storage limit. The request returns when the crawl is finished. operationId: crawlWebsite parameters: - $ref: '#/components/parameters/AgentId' requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/WebsiteSourceRequest' example: url: https://docs.example.com responses: '200': description: Crawl finished content: application/json: schema: $ref: '#/components/schemas/CrawlResponse' example: success: true pagesCrawled: 3 pages: - id: 0f3c9a7e-2b1d-4c6e-8a5f-7d9b1e3c5a7f path: / - id: 9a1b3c5d-7e9f-4a2b-8c4d-6e8f0a2b4c6d path: /pricing - id: 2d4f6a8c-0e2a-4c6e-9a1b-3c5d7e9f1a3b path: /getting-started '400': $ref: '#/components/responses/InvalidBody' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/PlanLimit' '404': $ref: '#/components/responses/NotFound' security: - apiKeyAuth: [] x-codeSamples: - lang: bash label: cURL source: >- curl -X POST https://reygrid.com/api/v1/agents/{AgentId}/sources/website \ -H "Authorization: Bearer $REYGRID_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://docs.example.com"}' - lang: python label: Python SDK source: |- from reygrid import ReyGrid client = ReyGrid(api_key="YOUR_API_KEY") res = client.crawl_website("{AgentId}", "https://docs.example.com") print(res.pagesCrawled, [page.path for page in res.pages]) - lang: javascript label: TypeScript SDK source: >- import { ReyGrid } from "reygrid"; const client = new ReyGrid({ apiKey: "YOUR_API_KEY" }); const { pagesCrawled, pages } = await client.crawlWebsite("{AgentId}", "https://docs.example.com"); console.log(pagesCrawled, pages.map((page) => page.path)); components: parameters: AgentId: name: agentId in: path required: true description: The ID of the agent. schema: type: string schemas: WebsiteSourceRequest: type: object required: - url properties: url: type: string format: uri example: https://docs.example.com CrawlResponse: type: object properties: success: type: boolean pagesCrawled: type: integer pages: type: array items: type: object properties: id: type: string description: Source ID of the page. path: type: string example: /pricing ErrorResponse: type: object required: - success - error properties: success: type: boolean output: type: object additionalProperties: true error: $ref: '#/components/schemas/Error' usage: $ref: '#/components/schemas/ChatUsage' AuthErrorResponse: type: object required: - error - code properties: error: type: string example: Unauthorized code: type: string example: AUTH_REQUIRED Error: type: object required: - code - message properties: code: type: string message: type: string list: type: array description: | Validation error details. Present when error.code is INVALID_BODY. items: $ref: '#/components/schemas/ValidationErrorItem' ChatUsage: type: object required: - credits properties: credits: type: number ValidationErrorItem: type: object additionalProperties: true responses: InvalidBody: description: Invalid body or query (INVALID_BODY) content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: success: false error: code: INVALID_BODY message: Invalid body list: title: - Title too short Unauthorized: description: API key is missing or invalid content: application/json: schema: $ref: '#/components/schemas/AuthErrorResponse' example: error: Unauthorized code: AUTH_REQUIRED PlanLimit: description: The plan's storage or page limit is reached (PLAN_LIMIT_REACHED) content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: success: false error: code: PLAN_LIMIT_REACHED message: You need to upgrade your plan NotFound: description: The agent, source or message doesn't exist, or the key can't access it content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: success: false error: code: SOURCE_NOT_FOUND message: This source does not exist securitySchemes: apiKeyAuth: type: http scheme: bearer bearerFormat: API-Key description: ReyGrid API key sent as a Bearer token. ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.