> ## Documentation Index
> Fetch the complete documentation index at: https://docs.meter.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Strategies

> AI-generated extraction plans that define how to scrape a website

# Strategies

A **strategy** is a reusable extraction plan that tells Meter how to extract data from a webpage. Think of it like a recipe: you create it once by describing what you want, and Meter's AI figures out the exact selectors and extraction logic.

## What is a strategy?

A strategy contains:

* **Extraction method**: Either CSS Path (for traditional HTML) or API Path (for JavaScript-heavy sites)
* **Field definitions** mapping selectors or API responses to your data fields
* **Extraction metadata** like item containers, scopes, or API endpoints
* **Output schema** (optional): Defines the exact JSON structure for results. See [Output Schemas](/concepts/output-schemas)
* **Filter config** (optional): Post-extraction filtering conditions. See [Filtering](/concepts/filtering)
* **Strategy group** (optional): Group membership for bulk management. See [Strategy Groups](/concepts/strategy-groups)

Meter automatically detects which extraction method works best for each site—you don't need to choose. Once created, a strategy can be reused unlimited times across similar pages—**no LLM costs after initial generation**.

## How strategies are generated

Meter uses AI to analyze your target webpage and generate precise extraction strategies:

1. **You provide**: A URL and plain-English description of what to extract
2. **Meter analyzes**: The page structure, HTML patterns, and content layout
3. **AI generates**: CSS selectors and extraction rules optimized for that page
4. **You get**: A reusable strategy plus preview data showing what was extracted

This approach combines the intelligence of AI setup with the speed and reliability of traditional scraping.

### Example

```python theme={null}
from meter_sdk import MeterClient

client = MeterClient(api_key="sk_live_...")

# Generate a strategy
result = client.generate_strategy(
    url="https://news.ycombinator.com",
    description="Extract post titles and scores",
    name="HN Front Page"
)

# Check the preview
print(f"Extracted {len(result['preview_data'])} items")
for item in result['preview_data'][:3]:
    print(item)

# Output:
# {'title': 'Launch HN: ...', 'score': 42}
# {'title': 'Ask HN: ...', 'score': 15}
# ...
```

## Extraction methods

Meter automatically selects the best extraction method for each site. You describe what you want, and Meter figures out how to get it.

### CSS Path extraction

For traditional HTML pages, Meter generates CSS selectors that target the content you need.

**Best for:**

* Static HTML sites
* Server-rendered pages
* Sites with stable DOM structure
* Blogs, news sites, and content pages

CSS Path extraction is fast and reliable for sites where content is present in the initial HTML response.

### API Path extraction

For JavaScript-heavy sites, Meter automatically discovers the underlying APIs that power the page and extracts data directly from them.

<Steps>
  <Step title="Automatic detection">
    Meter identifies when a page relies on JavaScript to load its content.
  </Step>

  <Step title="API discovery">
    The data source APIs are automatically identified—no reverse engineering required.
  </Step>

  <Step title="Authentication handled">
    Any required tokens or session data are handled automatically.
  </Step>

  <Step title="Direct extraction">
    Data is extracted directly from API responses—cleaner and more reliable than parsing the DOM.
  </Step>
</Steps>

**Best for:**

* Single-page applications (React, Vue, Angular)
* Financial data sites
* Dynamic dashboards
* Sites with client-side rendering

<Tip>
  API Path extraction often returns cleaner, more structured data than DOM scraping—and it's more resilient to UI changes.
</Tip>

## Automatic token handling

JavaScript-heavy sites often require authentication tokens to access their APIs. Meter handles this automatically—you don't need to worry about the details.

<AccordionGroup>
  <Accordion title="CSRF Tokens">
    Many sites protect their APIs with CSRF tokens. Meter detects and includes these tokens automatically, so your extractions work without manual configuration.
  </Accordion>

  <Accordion title="Session Cookies">
    Session state is maintained across the extraction process. If a site requires cookies to access its APIs, Meter handles this for you.
  </Accordion>

  <Accordion title="Custom Headers">
    API keys, authorization headers, and other required headers are automatically included in requests.
  </Accordion>

  <Accordion title="Multi-API Chains">
    Some sites require multiple API calls in sequence. Meter handles these dependencies and chains requests in the correct order.
  </Accordion>
</AccordionGroup>

## Real-world example: Financial data

Consider extracting stock quotes from a financial data site. When you visit the page, you see prices updating in real-time—but the HTML source shows almost nothing. The data is loaded via JavaScript from a hidden API.

With traditional scraping, you would need to:

1. Reverse-engineer the API endpoints
2. Figure out the authentication requirements
3. Handle CSRF tokens and session management
4. Parse the JSON response format

**With Meter**, you simply describe what you want: "Extract stock symbol, current price, and daily change." Meter automatically:

* Discovers the quote API endpoint
* Captures any required authentication tokens
* Extracts the structured data from API responses
* Returns clean, normalized data

The result is faster extraction, cleaner data, and a strategy that's resilient to UI redesigns—because you're hitting the same API the site uses internally.

## Strategy lifecycle

```mermaid theme={null}
graph LR
    A[Generate] --> B{Preview OK?}
    B -->|Yes| C[Use in Jobs]
    B -->|No| D[Refine]
    D --> B
    C --> E[Monitor Results]
    E --> F{Still Working?}
    F -->|Yes| C
    F -->|No| D
```

1. **Generate**: Create strategy with AI
2. **Preview**: Check the `preview_data` to verify extraction
3. **Refine** (optional): Provide feedback if something's missing
4. **Use**: Run jobs with the strategy
5. **Monitor**: Check if results are still accurate over time

## Refining strategies

If the initial extraction isn't perfect, refine it with feedback:

```python theme={null}
# Initial generation
result = client.generate_strategy(
    url="https://shop.com/products",
    description="Extract product info",
    name="Product Scraper"
)

# Check preview - oops, missing images
print(result['preview_data'])  # No 'image' field

# Refine with feedback
refined = client.refine_strategy(
    strategy_id=result['strategy_id'],
    feedback="Also extract product images"
)

# Check again
print(refined['preview_data'])  # Now has 'image' field
```

<Tip>
  Refinement uses cached HTML from the initial generation, so it's fast and doesn't re-fetch the page.
</Tip>

## When to create new strategies

Create a new strategy when:

<CardGroup cols={2}>
  <Card title="Different Site Structure" icon="sitemap">
    Each website layout needs its own strategy
  </Card>

  <Card title="Different Data Fields" icon="table">
    Different extraction requirements need different strategies
  </Card>

  <Card title="Major Site Redesign" icon="paintbrush">
    If a site changes its HTML structure significantly
  </Card>

  <Card title="Different Page Types" icon="file">
    Product pages vs. category pages need separate strategies
  </Card>
</CardGroup>

## Reusing strategies

You can reuse the same strategy across:

* **Multiple URLs** on the same site (e.g., different products)
* **Pagination** (if the structure is consistent)
* **Similar pages** (if they share HTML structure)

```python theme={null}
# Generate once
strategy = client.generate_strategy(
    url="https://shop.com/product/123",
    description="Extract name, price, description"
)
strategy_id = strategy['strategy_id']

# Reuse for different products
job1 = client.create_job(strategy_id, "https://shop.com/product/123")
job2 = client.create_job(strategy_id, "https://shop.com/product/456")
job3 = client.create_job(strategy_id, "https://shop.com/product/789")
```

## Strategy management

### Listing strategies

```python theme={null}
# Get all strategies
strategies = client.list_strategies(limit=20)

for strategy in strategies:
    print(f"{strategy['name']}: {strategy['strategy_id']}")
```

### Getting strategy details

```python theme={null}
strategy = client.get_strategy(strategy_id)

print(f"Name: {strategy['name']}")
print(f"Description: {strategy['description']}")
print(f"Created: {strategy['created_at']}")
print(f"Preview: {strategy['preview_data']}")
```

### Deleting strategies

```python theme={null}
# Delete a strategy (also deletes associated jobs and schedules)
client.delete_strategy(strategy_id)
```

<Warning>
  Deleting a strategy also deletes all associated jobs and schedules. This action cannot be undone.
</Warning>

## Best practices

<AccordionGroup>
  <Accordion title="Use descriptive names">
    Give strategies clear names that describe their purpose:

    **Good**: `"HN Front Page - Titles and Scores"`
    **Bad**: `"Strategy 1"`

    This helps when managing multiple strategies.
  </Accordion>

  <Accordion title="Test with refinement">
    Always check `preview_data` before creating jobs:

    ```python theme={null}
    result = client.generate_strategy(...)

    # Verify all required fields are present
    required_fields = {'title', 'price', 'image'}
    actual_fields = set(result['preview_data'][0].keys())

    if not required_fields.issubset(actual_fields):
        missing = required_fields - actual_fields
        client.refine_strategy(
            strategy_id=result['strategy_id'],
            feedback=f"Also extract: {', '.join(missing)}"
        )
    ```
  </Accordion>

  <Accordion title="Be specific in descriptions">
    Provide clear, specific extraction instructions:

    **Good**: "Extract product name, price with currency, main image URL, and stock availability from the product grid"

    **Bad**: "Get products"

    Specific descriptions lead to better strategies on the first try.
  </Accordion>

  <Accordion title="Monitor strategy health">
    Strategies can break if sites change their HTML:

    ```python theme={null}
    # Check recent jobs for failures
    jobs = client.list_jobs(
        strategy_id=strategy_id,
        status='failed',
        limit=5
    )

    if len(jobs) > 0:
        print(f"Strategy {strategy_id} may need updating")
    ```
  </Accordion>
</AccordionGroup>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Strategy generation fails">
    **Possible causes**:

    * URL is not accessible
    * Page requires authentication
    * Description is too vague

    **Solutions**:

    * Verify the URL loads in a browser
    * For auth-required pages, contact support
    * Make your description more specific
  </Accordion>

  <Accordion title="Missing fields in preview">
    **Problem**: Some expected fields aren't in `preview_data`

    **Solution**: Use refinement:

    ```python theme={null}
    client.refine_strategy(
        strategy_id=strategy_id,
        feedback="Also extract the product SKU and brand name"
    )
    ```
  </Accordion>

  <Accordion title="Strategy stops working">
    **Problem**: Jobs that worked before now fail or return incorrect data

    **Cause**: Website HTML structure changed

    **Solutions**:

    1. Generate a new strategy for the updated site
    2. Update your jobs to use the new strategy
    3. Delete the old strategy
  </Accordion>

  <Accordion title="API Path extraction returns empty data">
    **Problem**: Meter detected an API but returns no data

    **Possible causes**:

    * The API requires authentication that expired
    * The site changed its API endpoints
    * Rate limiting is blocking requests

    **Solutions**:

    * Generate a fresh strategy to capture new authentication tokens
    * If the site has changed significantly, the strategy may need regeneration
    * For rate-limited sites, reduce scrape frequency
  </Accordion>
</AccordionGroup>

## Next steps

<CardGroup cols={2}>
  <Card title="Create Jobs" icon="gears" href="/concepts/jobs">
    Learn how to run scrapes using your strategies
  </Card>

  <Card title="Set Up Schedules" icon="clock" href="/concepts/schedules">
    Automate scraping with recurring schedules
  </Card>

  <Card title="Python SDK Reference" icon="code" href="/api-reference/python/strategies">
    Explore all strategy methods in the SDK
  </Card>

  <Card title="REST API Reference" icon="brackets-curly" href="/api-reference/rest/strategies">
    View strategy endpoints in the REST API
  </Card>
</CardGroup>

## Need help?

Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh)