> ## Documentation Index
> Fetch the complete documentation index at: https://docs.meter.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Discovery Endpoints

> REST API endpoints for site crawling and URL discovery

# Discovery Endpoints

Discover URLs on websites and execute batch scrapes via HTTP.

## Start discovery

Start URL discovery using sitemap, pagination, or link pattern.

```http theme={null}
POST /discover
```

### Request body

<Tabs>
  <Tab title="Sitemap">
    ```json theme={null}
    {
      "discovery": {
        "method": "sitemap",
        "sitemap_url": "https://shop.com/sitemap.xml",
        "url_pattern": "products/*/",
        "max_urls": 1000
      }
    }
    ```
  </Tab>

  <Tab title="Pagination">
    ```json theme={null}
    {
      "discovery": {
        "method": "pagination",
        "url_template": "https://shop.com/products?page={n}",
        "start_index": 1,
        "step": 1,
        "max_pages": 100
      }
    }
    ```
  </Tab>

  <Tab title="Link Pattern">
    ```json theme={null}
    {
      "discovery": {
        "method": "link_pattern",
        "seed_url": "https://news.com",
        "link_pattern": "/article/*/",
        "navigation_pattern": "/category/",
        "max_depth": 2,
        "max_urls": 500
      }
    }
    ```
  </Tab>
</Tabs>

### Discovery parameters

#### Sitemap

| Parameter     | Type    | Required | Description                                          |
| ------------- | ------- | -------- | ---------------------------------------------------- |
| `method`      | string  | Yes      | Must be `"sitemap"`                                  |
| `sitemap_url` | string  | Yes      | URL to sitemap.xml file                              |
| `url_pattern` | string  | No       | Glob pattern to filter URLs                          |
| `max_urls`    | integer | No       | Maximum URLs to discover (default: 1000, max: 10000) |

#### Pagination

| Parameter      | Type    | Required | Description                                         |
| -------------- | ------- | -------- | --------------------------------------------------- |
| `method`       | string  | Yes      | Must be `"pagination"`                              |
| `url_template` | string  | Yes      | URL with `{n}` placeholder                          |
| `url_pattern`  | string  | No       | Glob pattern to filter URLs                         |
| `start_index`  | integer | No       | First page number (default: 1)                      |
| `step`         | integer | No       | Increment between pages (default: 1)                |
| `max_pages`    | integer | No       | Maximum pages to generate (default: 100, max: 1000) |

#### Link Pattern

| Parameter            | Type    | Required | Description                                          |
| -------------------- | ------- | -------- | ---------------------------------------------------- |
| `method`             | string  | Yes      | Must be `"link_pattern"`                             |
| `seed_url`           | string  | Yes      | Starting URL for crawl                               |
| `link_pattern`       | string  | Yes      | Glob pattern for URLs to collect                     |
| `navigation_pattern` | string  | No       | Pattern for pages to visit during crawl              |
| `max_depth`          | integer | No       | How deep to crawl (default: 2, max: 10)              |
| `max_urls`           | integer | No       | Maximum URLs to discover (default: 1000, max: 10000) |

### Response

```json theme={null}
{
  "discovery_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "discovery_method": "sitemap",
  "root_url": "https://shop.com/sitemap.xml"
}
```

### Example

```bash theme={null}
curl -X POST https://api.meter.sh/discover \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "discovery": {
      "method": "sitemap",
      "sitemap_url": "https://shop.com/sitemap.xml",
      "max_urls": 500
    }
  }'
```

## Get discovery status

Get discovery status and results.

```http theme={null}
GET /discover/{discovery_id}
```

### Response

**When pending/running:**

```json theme={null}
{
  "discovery_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "discovery_method": "sitemap",
  "root_url": "https://shop.com/sitemap.xml"
}
```

**When completed:**

```json theme={null}
{
  "discovery_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "completed",
  "discovery_method": "sitemap",
  "root_url": "https://shop.com/sitemap.xml",
  "total_urls": 847,
  "filtered_count": 847,
  "inferred_pattern": "/products/[slug]/",
  "url_patterns": {
    "/products/": 847
  },
  "sample_urls": [
    "https://shop.com/products/widget-a",
    "https://shop.com/products/widget-b"
  ],
  "errors": []
}
```

**When failed:**

```json theme={null}
{
  "discovery_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "failed",
  "discovery_method": "sitemap",
  "root_url": "https://shop.com/sitemap.xml",
  "errors": ["Sitemap not accessible: 404 Not Found"]
}
```

**Status values:** `pending`, `running`, `completed`, `failed`

### Example

```bash theme={null}
curl https://api.meter.sh/discover/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer sk_live_..."
```

## Get discovery URLs

Fetch all discovered URLs with pagination.

```http theme={null}
GET /discover/{discovery_id}/urls?limit=1000&offset=0
```

### Query parameters

| Parameter | Type    | Required | Description                                    |
| --------- | ------- | -------- | ---------------------------------------------- |
| `limit`   | integer | No       | Max URLs to return (default: 1000, max: 10000) |
| `offset`  | integer | No       | Results to skip (default: 0)                   |

### Response

```json theme={null}
{
  "discovery_id": "550e8400-e29b-41d4-a716-446655440000",
  "urls": [
    "https://shop.com/products/widget-a",
    "https://shop.com/products/widget-b",
    "..."
  ],
  "total": 847,
  "limit": 1000,
  "offset": 0
}
```

### Example

```bash theme={null}
# Get first 100 URLs
curl "https://api.meter.sh/discover/550e8400.../urls?limit=100" \
  -H "Authorization: Bearer sk_live_..."

# Get next 100 URLs
curl "https://api.meter.sh/discover/550e8400.../urls?limit=100&offset=100" \
  -H "Authorization: Bearer sk_live_..."
```

## Execute discovery

Execute a one-time batch scrape from discovered URLs.

```http theme={null}
POST /discover/{discovery_id}/execute
```

### Request body

```json theme={null}
{
  "strategy_id": "660e8400-e29b-41d4-a716-446655440000",
  "max_urls": 100,
  "url_filter": ".*widget.*"
}
```

### Parameters

| Parameter     | Type    | Required | Description                       |
| ------------- | ------- | -------- | --------------------------------- |
| `strategy_id` | string  | Yes      | Strategy UUID to use for scraping |
| `max_urls`    | integer | No       | Maximum URLs to process           |
| `url_filter`  | string  | No       | Regex pattern to filter URLs      |

### Response

```json theme={null}
{
  "batch_id": "770e8400-e29b-41d4-a716-446655440000",
  "jobs_queued": 100
}
```

### Example

```bash theme={null}
curl -X POST https://api.meter.sh/discover/550e8400.../execute \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "strategy_id": "660e8400-e29b-41d4-a716-446655440000",
    "max_urls": 50
  }'
```

## Create discovery schedule

Create a recurring schedule from discovered URLs.

```http theme={null}
POST /discover/{discovery_id}/schedule
```

### Request body

<Tabs>
  <Tab title="Interval">
    ```json theme={null}
    {
      "strategy_id": "660e8400-e29b-41d4-a716-446655440000",
      "interval_seconds": 86400,
      "webhook_url": "https://your-app.com/webhooks/meter",
      "max_urls": 500
    }
    ```
  </Tab>

  <Tab title="Cron">
    ```json theme={null}
    {
      "strategy_id": "660e8400-e29b-41d4-a716-446655440000",
      "cron_expression": "0 9 * * *",
      "webhook_url": "https://your-app.com/webhooks/meter",
      "max_urls": 500
    }
    ```
  </Tab>
</Tabs>

### Parameters

| Parameter          | Type    | Required | Description                      |
| ------------------ | ------- | -------- | -------------------------------- |
| `strategy_id`      | string  | Yes      | Strategy UUID to use             |
| `interval_seconds` | integer | No\*     | Seconds between runs             |
| `cron_expression`  | string  | No\*     | Cron schedule expression         |
| `webhook_url`      | string  | No       | URL for completion notifications |
| `max_urls`         | integer | No       | Maximum URLs per run             |
| `url_filter`       | string  | No       | Regex pattern to filter URLs     |

\*Either `interval_seconds` or `cron_expression` is required.

### Response

```json theme={null}
{
  "schedule_id": "880e8400-e29b-41d4-a716-446655440000",
  "strategy_id": "660e8400-e29b-41d4-a716-446655440000",
  "urls": ["https://shop.com/products/widget-a", "..."],
  "schedule_type": "interval",
  "interval_seconds": 86400,
  "enabled": true,
  "webhook_url": "https://your-app.com/webhooks/meter",
  "next_run_at": "2025-01-16T10:30:00Z",
  "created_at": "2025-01-15T10:30:00Z"
}
```

### Example

```bash theme={null}
curl -X POST https://api.meter.sh/discover/550e8400.../schedule \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "strategy_id": "660e8400-e29b-41d4-a716-446655440000",
    "interval_seconds": 86400,
    "webhook_url": "https://your-app.com/webhooks/meter"
  }'
```

## List discoveries

List all discoveries for the authenticated user.

```http theme={null}
GET /discoveries?status={status}&limit=20&offset=0
```

### Query parameters

| Parameter | Type    | Required | Description                         |
| --------- | ------- | -------- | ----------------------------------- |
| `status`  | string  | No       | Filter by status                    |
| `limit`   | integer | No       | Max results (default: 20, max: 100) |
| `offset`  | integer | No       | Results to skip (default: 0)        |

### Response

Array of discovery objects (same format as Get discovery).

### Example

```bash theme={null}
# All discoveries
curl https://api.meter.sh/discoveries \
  -H "Authorization: Bearer sk_live_..."

# Only completed
curl "https://api.meter.sh/discoveries?status=completed" \
  -H "Authorization: Bearer sk_live_..."
```

## Delete discovery

Delete a discovery and its associated URLs.

```http theme={null}
DELETE /discover/{discovery_id}
```

### Response

```json theme={null}
{
  "message": "Discovery deleted",
  "discovery_id": "550e8400-e29b-41d4-a716-446655440000"
}
```

### Example

```bash theme={null}
curl -X DELETE https://api.meter.sh/discover/550e8400... \
  -H "Authorization: Bearer sk_live_..."
```

## Polling for completion

Since discovery runs asynchronously, poll until status is `completed` or `failed`:

```javascript theme={null}
async function waitForDiscovery(discoveryId) {
  while (true) {
    const response = await fetch(
      `https://api.meter.sh/discover/${discoveryId}`,
      {
        headers: {
          Authorization: `Bearer ${process.env.METER_API_KEY}`,
        },
      }
    );

    const discovery = await response.json();

    if (discovery.status === "completed") {
      return discovery;
    } else if (discovery.status === "failed") {
      throw new Error(discovery.errors.join(", "));
    }

    // Wait 2 seconds before next check
    await new Promise((resolve) => setTimeout(resolve, 2000));
  }
}
```

## Error responses

| Status | Description                                           |
| ------ | ----------------------------------------------------- |
| `400`  | Invalid request (see discovery-specific errors below) |
| `401`  | Invalid or missing API key                            |
| `404`  | Discovery or strategy not found                       |
| `500`  | Internal server error                                 |
| `503`  | Service temporarily unavailable                       |

### Discovery-specific errors

| Status | Error                     | Description                        |
| ------ | ------------------------- | ---------------------------------- |
| `400`  | Invalid url\_filter regex | The regex pattern is invalid       |
| `400`  | Discovery not ready       | Tried to execute before completion |
| `400`  | No URLs match the filter  | Filter excluded all URLs           |

See [REST API Errors](/api-reference/rest/errors) for detailed error handling.

## Next steps

<CardGroup cols={2}>
  <Card title="Site Crawling Guide" icon="book" href="/guides/site-crawling">
    Step-by-step crawling guide
  </Card>

  <Card title="Site Crawling Concepts" icon="lightbulb" href="/concepts/site-crawling">
    Understand how site crawling works
  </Card>

  <Card title="Schedule Endpoints" icon="clock" href="/api-reference/rest/schedules">
    Manage recurring scrapes
  </Card>

  <Card title="Job Endpoints" icon="gears" href="/api-reference/rest/jobs">
    Track batch job results
  </Card>
</CardGroup>

## Need help?

Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh)