# MeterClient Source: https://docs.meter.sh/api-reference/python/client Main client class for interacting with the Meter API # MeterClient The `MeterClient` class is the main interface for all Meter API operations. It handles authentication, request management, and provides methods for strategies, jobs, and schedules. ## Constructor ```python theme={null} MeterClient(api_key: str, base_url: str = "https://api.meter.sh") ``` ### Parameters | Parameter | Type | Required | Description | | ---------- | ----- | -------- | ---------------------------------------------- | | `api_key` | `str` | Yes | Your Meter API key (starts with `sk_live_`) | | `base_url` | `str` | No | API base URL (default: `https://api.meter.sh`) | ### Example ```python theme={null} from meter_sdk import MeterClient import os # Recommended: Load from environment client = MeterClient(api_key=os.getenv("METER_API_KEY")) # With custom base URL (for development) client = MeterClient( api_key=os.getenv("METER_API_KEY"), base_url="http://localhost:8000" ) ``` ## Context Manager The client can be used as a context manager for automatic resource cleanup: ```python theme={null} with MeterClient(api_key="sk_live_...") as client: strategies = client.list_strategies() # Client automatically closes HTTP connections on exit ``` ## Strategy Methods ### generate\_strategy() Generate a new extraction strategy using AI. ```python theme={null} generate_strategy( url: str, description: str, name: str, force_api: bool = False, output_schema: Optional[Dict[str, Any]] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | --------------- | ------ | -------- | ----------------------------------------------------------------------------- | | `url` | `str` | Yes | Target webpage URL to analyze | | `description` | `str` | Yes | Plain English description of what to extract | | `name` | `str` | Yes | Human-readable name for this strategy | | `force_api` | `bool` | No | Force API-based capture instead of CSS extraction (default: False) | | `output_schema` | `Dict` | No | Desired output JSON structure. See [Output Schemas](/concepts/output-schemas) | **Returns:** `Dict` with fields: * `strategy_id` (str): UUID of the created strategy * `strategy` (dict): The extraction strategy (CSS selectors, fields) * `preview_data` (list): Sample extracted data (first 5-10 items) * `attempts` (int): Number of generation attempts (usually 1) * `scraper_type` (str): Type of scraper used - `'css'` or `'api'` * `api_parameters` (dict, optional): Available URL parameters for API-based strategies **Example:** ```python theme={null} result = client.generate_strategy( url="https://news.ycombinator.com", description="Extract post titles and scores", name="HN Front Page" ) strategy_id = result["strategy_id"] print(f"Created strategy: {strategy_id}") print(f"Preview: {result['preview_data'][:3]}") ``` **Example with API capture:** ```python theme={null} # Force API-based capture for sites with underlying APIs result = client.generate_strategy( url="https://api-heavy-site.com/products", description="Extract product listings", name="Product API Scraper", force_api=True ) print(f"Scraper type: {result['scraper_type']}") # 'api' or 'css' # For API strategies, check available parameters if result.get('api_parameters'): print(f"Available parameters: {result['api_parameters']}") # e.g., {'page': 1, 'limit': 20, 'sort': 'price'} ``` **Raises:** `MeterError` if generation fails When `force_api=True`, Meter will attempt to identify and capture underlying API calls instead of using CSS selectors. This is useful for sites that load data dynamically via JavaScript APIs. *** ### refine\_strategy() Refine an existing strategy with feedback. ```python theme={null} refine_strategy( strategy_id: str, feedback: str ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ------------------------------------- | | `strategy_id` | `str` | Yes | UUID of the strategy to refine | | `feedback` | `str` | Yes | Description of what to improve or add | **Returns:** `Dict` with same fields as `generate_strategy()` **Example:** ```python theme={null} refined = client.refine_strategy( strategy_id="550e8400-e29b-41d4-a716-446655440000", feedback="Also extract the product images and SKU" ) print(f"Refined preview: {refined['preview_data']}") ``` Refinement uses cached HTML from initial generation, so it's fast and doesn't re-fetch the page. *** ### list\_strategies() List all strategies for the authenticated user. ```python theme={null} list_strategies( limit: int = 20, offset: int = 0 ) -> List[Dict] ``` **Parameters:** | Parameter | Type | Required | Description | | --------- | ----- | -------- | ---------------------------------------------------- | | `limit` | `int` | No | Maximum number of strategies to return (default: 20) | | `offset` | `int` | No | Number of strategies to skip (default: 0) | **Returns:** `List[Dict]` where each dict contains: * `id` (str): Strategy UUID * `name` (str): Strategy name * `description` (str): Extraction description * `url` (str): Original URL used for generation * `preview_data` (list): Sample extracted data * `created_at` (str): ISO timestamp * `updated_at` (str): ISO timestamp **Example:** ```python theme={null} # Get first 20 strategies strategies = client.list_strategies() for strategy in strategies: print(f"{strategy['name']}: {strategy['strategy_id']}") # Pagination page_2 = client.list_strategies(limit=20, offset=20) ``` *** ### get\_strategy() Get details for a specific strategy. ```python theme={null} get_strategy(strategy_id: str) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | -------------------- | | `strategy_id` | `str` | Yes | UUID of the strategy | **Returns:** `Dict` with full strategy details (same fields as `list_strategies()` items) **Example:** ```python theme={null} strategy = client.get_strategy("550e8400-e29b-41d4-a716-446655440000") print(f"Name: {strategy['name']}") print(f"Created: {strategy['created_at']}") print(f"Preview: {strategy['preview_data']}") ``` **Raises:** `MeterError` with 404 if strategy not found *** ### delete\_strategy() Delete a strategy and all associated jobs and schedules. ```python theme={null} delete_strategy(strategy_id: str) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ------------------------------ | | `strategy_id` | `str` | Yes | UUID of the strategy to delete | **Returns:** `Dict` with confirmation message **Example:** ```python theme={null} result = client.delete_strategy("550e8400-e29b-41d4-a716-446655440000") print(result) # {'message': 'Strategy deleted successfully'} ``` This action is irreversible. All associated jobs and schedules will also be deleted. *** ## Job Methods ### create\_job() Create a new scrape job using a strategy. ```python theme={null} create_job( strategy_id: str, url: Optional[str] = None, urls: Optional[List[str]] = None, parameters: Optional[Dict[str, Any]] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----------- | ----------- | ---------------------------------------------------------- | | `strategy_id` | `str` | Yes | UUID of the strategy to use | | `url` | `str` | Conditional | Single URL to scrape (use `url` OR `urls`, not both) | | `urls` | `List[str]` | Conditional | List of URLs to scrape as a batch | | `parameters` | `Dict` | No | Override API parameters for this job (API strategies only) | **Returns:** `Dict` with fields: * `job_id` (str): UUID of the created job (single URL) * `batch_id` (str): Batch UUID for tracking progress (multiple URLs) * `status` (str): Job status (usually "pending") * `strategy_id` (str): Strategy UUID * `url` (str): Target URL * `parameters` (dict, optional): Parameters used for this job * `created_at` (str): ISO timestamp **Example:** ```python theme={null} job = client.create_job( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/page" ) print(f"Job created: {job['job_id']}") print(f"Status: {job['status']}") ``` **Example with API parameters:** ```python theme={null} # For API-based strategies, override parameters at runtime job = client.create_job( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/api/products", parameters={ "page": 2, "limit": 50, "category": "electronics" } ) ``` **Example with batch URLs:** ```python theme={null} # Scrape multiple URLs in a single batch job = client.create_job( strategy_id="550e8400-e29b-41d4-a716-446655440000", urls=[ "https://example.com/products/1", "https://example.com/products/2", "https://example.com/products/3" ] ) print(f"Batch created: {job['batch_id']}") ``` You must provide either `url` or `urls`, but not both. The `parameters` option only applies to API-based strategies (where `scraper_type` is `'api'`). *** ### execute\_job() Create and execute a scrape job synchronously. Returns results directly without polling. ```python theme={null} execute_job( strategy_id: str, url: str, parameters: Optional[Dict[str, Any]] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ------ | -------- | ---------------------------------------------------------- | | `strategy_id` | `str` | Yes | UUID of the strategy to use | | `url` | `str` | Yes | URL to scrape | | `parameters` | `Dict` | No | Override API parameters for this job (API strategies only) | **Returns:** `Dict` with completed job details including `results` **Example:** ```python theme={null} # Simple synchronous scrape result = client.execute_job( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products" ) print(f"Got {result['item_count']} items") for item in result['results']: print(item) ``` **Example with API parameters:** ```python theme={null} result = client.execute_job( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/api/products", parameters={"page": 2, "limit": 50} ) ``` **Raises:** `MeterError` if job fails or times out This endpoint blocks until the job completes (up to 1 hour timeout). Use `create_job()` + `wait_for_job()` for more control over polling behavior, or `create_job()` alone for fire-and-forget jobs. *** ### get\_job() Get status and results for a job. ```python theme={null} get_job(job_id: str) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | --------- | ----- | -------- | --------------- | | `job_id` | `str` | Yes | UUID of the job | **Returns:** `Dict` with fields: * `job_id` (str): Job UUID * `status` (str): "pending", "running", "completed", or "failed" * `results` (list): Extracted data (only if status is "completed") * `item_count` (int): Number of items extracted * `content_hash` (str): Hash for change detection * `structural_signature` (dict): Structural fingerprint * `error` (str): Error message (only if status is "failed") * `started_at` (str): ISO timestamp * `completed_at` (str): ISO timestamp * `created_at` (str): ISO timestamp **Example:** ```python theme={null} job = client.get_job("660e8400-e29b-41d4-a716-446655440000") if job['status'] == 'completed': print(f"Extracted {job['item_count']} items") for item in job['results']: print(item) elif job['status'] == 'failed': print(f"Job failed: {job['error']}") else: print(f"Job is {job['status']}") ``` *** ### wait\_for\_job() Wait for a job to complete, polling automatically. ```python theme={null} wait_for_job( job_id: str, poll_interval: float = 1.0, timeout: Optional[float] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | --------------- | ------- | -------- | -------------------------------------------------- | | `job_id` | `str` | Yes | UUID of the job to wait for | | `poll_interval` | `float` | No | Seconds between status checks (default: 1.0) | | `timeout` | `float` | No | Maximum seconds to wait (default: None = infinite) | **Returns:** `Dict` with completed job details (same as `get_job()`) **Example:** ```python theme={null} from meter_sdk import MeterError # Wait indefinitely completed = client.wait_for_job("660e8400-e29b-41d4-a716-446655440000") print(f"Done! {completed['item_count']} items") # With timeout try: completed = client.wait_for_job( "660e8400-e29b-41d4-a716-446655440000", poll_interval=2.0, timeout=300.0 # 5 minutes ) except MeterError as e: print(f"Timeout or error: {e}") ``` **Raises:** `MeterError` if timeout exceeded or job fails *** ### list\_jobs() List jobs with optional filtering. ```python theme={null} list_jobs( strategy_id: Optional[str] = None, status: Optional[str] = None, limit: int = 20, offset: int = 0 ) -> List[Dict] ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ------------------------------------------------------------- | | `strategy_id` | `str` | No | Filter by strategy UUID | | `status` | `str` | No | Filter by status: "pending", "running", "completed", "failed" | | `limit` | `int` | No | Maximum jobs to return (default: 20) | | `offset` | `int` | No | Number of jobs to skip (default: 0) | **Returns:** `List[Dict]` of job summaries **Example:** ```python theme={null} # All jobs all_jobs = client.list_jobs(limit=50) # Jobs for specific strategy strategy_jobs = client.list_jobs( strategy_id="550e8400-e29b-41d4-a716-446655440000" ) # Only failed jobs failed = client.list_jobs(status="failed", limit=10) # Combined filters recent_completed = client.list_jobs( strategy_id="550e8400-e29b-41d4-a716-446655440000", status="completed", limit=5 ) ``` *** ### compare\_jobs() Compare two jobs to detect changes. ```python theme={null} compare_jobs( job_id: str, other_job_id: str ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | -------------- | ----- | -------- | ------------------------------- | | `job_id` | `str` | Yes | First job UUID | | `other_job_id` | `str` | Yes | Second job UUID to compare with | **Returns:** `Dict` with fields: * `content_hash_match` (bool): True if content hashes match * `structural_match` (bool): True if structure matches * `semantic_similarity` (float): Similarity score 0.0-1.0 (planned feature) * `changes` (list): Detected structural changes **Example:** ```python theme={null} comparison = client.compare_jobs( "660e8400-e29b-41d4-a716-446655440000", "770e8400-e29b-41d4-a716-446655440000" ) print(f"Content match: {comparison['content_hash_match']}") print(f"Structural match: {comparison['structural_match']}") if not comparison['content_hash_match']: print("Content has changed!") for change in comparison.get('changes', []): print(f" - {change}") ``` *** ### get\_strategy\_history() Get timeline of all jobs for a strategy. ```python theme={null} get_strategy_history(strategy_id: str) -> List[Dict] ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ------------- | | `strategy_id` | `str` | Yes | Strategy UUID | **Returns:** `List[Dict]` where each dict contains: * `job_id` (str): Job UUID * `status` (str): Job status * `item_count` (int): Items extracted * `has_changes` (bool): True if content changed vs. previous job * `created_at` (str): ISO timestamp **Example:** ```python theme={null} history = client.get_strategy_history("550e8400-e29b-41d4-a716-446655440000") for entry in history: status_icon = "✓" if entry['status'] == 'completed' else "✗" change_icon = "📝" if entry['has_changes'] else "—" print(f"{status_icon} {entry['created_at']}: {entry['item_count']} items {change_icon}") ``` *** ## Schedule Methods ### create\_schedule() Create a new recurring schedule. ```python theme={null} create_schedule( strategy_id: str, url: Optional[str] = None, urls: Optional[List[str]] = None, interval_seconds: Optional[int] = None, cron_expression: Optional[str] = None, webhook_url: Optional[str] = None, webhook_metadata: Optional[Dict[str, Any]] = None, webhook_secret: Optional[str] = None, webhook_type: Optional[str] = None, parameters: Optional[Dict[str, Any]] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------------ | ----------- | ----------- | ---------------------------------------------------------------------------------------------- | | `strategy_id` | `str` | Yes | Strategy UUID to use | | `url` | `str` | Conditional | Single URL to scrape (use `url` OR `urls`, not both) | | `urls` | `List[str]` | Conditional | List of URLs to scrape on each run | | `interval_seconds` | `int` | Conditional | Interval in seconds (required if no cron) | | `cron_expression` | `str` | Conditional | Cron expression (required if no interval) | | `webhook_url` | `str` | No | Webhook URL for notifications | | `webhook_metadata` | `Dict` | No | Custom JSON metadata included in every webhook payload | | `webhook_secret` | `str` | No | Secret for `X-Webhook-Secret` header. Auto-generated if not provided when `webhook_url` is set | | `webhook_type` | `str` | No | `'standard'` or `'slack'`. Auto-detected from URL if not specified | | `parameters` | `Dict` | No | Default API parameter overrides for all scheduled runs (API strategies only) | **Returns:** `Dict` with schedule details **Example:** ```python theme={null} # Interval-based schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", interval_seconds=3600 # Every hour ) # Cron-based schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", cron_expression="0 9 * * *" # Daily at 9 AM ) # With webhook schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter" ) print(f"Schedule created: {schedule['schedule_id']}") print(f"Next run: {schedule['next_run_at']}") ``` **Example with API parameters:** ```python theme={null} # For API-based strategies, set default parameters for all runs schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/api/jobs", interval_seconds=3600, parameters={ "category": "engineering", "location": "remote", "limit": 100 } ) ``` **Example with multiple URLs:** ```python theme={null} # Monitor multiple pages on a schedule schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", urls=[ "https://example.com/products/electronics", "https://example.com/products/clothing", "https://example.com/products/home" ], interval_seconds=3600 ) ``` You must provide either `url` or `urls`, but not both. You must also provide either `interval_seconds` or `cron_expression`, but not both. *** ### list\_schedules() List all schedules for the authenticated user. ```python theme={null} list_schedules() -> List[Dict] ``` **Returns:** `List[Dict]` of schedules **Example:** ```python theme={null} schedules = client.list_schedules() for schedule in schedules: print(f"Schedule {schedule['schedule_id']}:") print(f" Type: {schedule['schedule_type']}") print(f" Enabled: {schedule['enabled']}") print(f" Next run: {schedule['next_run_at']}") ``` *** ### update\_schedule() Update an existing schedule. ```python theme={null} update_schedule( schedule_id: str, enabled: Optional[bool] = None, url: Optional[str] = None, urls: Optional[List[str]] = None, interval_seconds: Optional[int] = None, cron_expression: Optional[str] = None, webhook_url: Optional[str] = None, webhook_metadata: Optional[Dict[str, Any]] = None, webhook_secret: Optional[str] = None, webhook_type: Optional[str] = None, parameters: Optional[Dict[str, Any]] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------------ | ----------- | -------- | --------------------------------------------------- | | `schedule_id` | `str` | Yes | Schedule UUID | | `enabled` | `bool` | No | Enable/disable schedule | | `url` | `str` | No | Update to single URL | | `urls` | `List[str]` | No | Update to multiple URLs | | `interval_seconds` | `int` | No | New interval in seconds | | `cron_expression` | `str` | No | New cron expression | | `webhook_url` | `str` | No | New webhook URL (or None to remove) | | `webhook_metadata` | `Dict` | No | Update custom JSON metadata for webhook payloads | | `webhook_secret` | `str` | No | Update webhook secret | | `webhook_type` | `str` | No | Update webhook type: `'standard'` or `'slack'` | | `parameters` | `Dict` | No | Update API parameter defaults (API strategies only) | **Returns:** `Dict` with updated schedule details **Example:** ```python theme={null} # Disable schedule client.update_schedule(schedule_id, enabled=False) # Change interval client.update_schedule(schedule_id, interval_seconds=7200) # Update webhook client.update_schedule( schedule_id, webhook_url="https://new-domain.com/webhooks" ) # Remove webhook client.update_schedule(schedule_id, webhook_url=None) # Update API parameters client.update_schedule( schedule_id, parameters={"category": "new-category", "limit": 200} ) ``` Setting `url` will clear `urls`, and vice versa. *** ### delete\_schedule() Delete a schedule (stops future jobs). ```python theme={null} delete_schedule(schedule_id: str) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ----------------------- | | `schedule_id` | `str` | Yes | Schedule UUID to delete | **Returns:** `Dict` with confirmation message **Example:** ```python theme={null} result = client.delete_schedule("880e8400-e29b-41d4-a716-446655440000") print(result) # {'message': 'Schedule deleted successfully'} ``` *** ### get\_schedule\_changes() Get unseen changes for a schedule (pull-based change detection). ```python theme={null} get_schedule_changes( schedule_id: str, mark_seen: bool = True, filter: Optional[str] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ------ | -------- | --------------------------------------------- | | `schedule_id` | `str` | Yes | Schedule UUID | | `mark_seen` | `bool` | No | Mark returned changes as seen (default: True) | | `filter` | `str` | No | Lucene-style keyword filter for results | **Returns:** `Dict` with fields: * `schedule_id` (str): Schedule UUID * `changes` (list): Jobs with changes (full job details) * `count` (int): Number of changed jobs * `marked_seen` (bool): Whether changes were marked as seen **Example:** ```python theme={null} # Get and mark changes as seen changes = client.get_schedule_changes( "880e8400-e29b-41d4-a716-446655440000", mark_seen=True ) if changes['count'] > 0: print(f"Found {changes['count']} jobs with changes") for change in changes['changes']: print(f"Job {change['job_id']}: {change['item_count']} items") # Process change['results'] # Preview without marking as seen preview = client.get_schedule_changes( "880e8400-e29b-41d4-a716-446655440000", mark_seen=False ) ``` **Example with keyword filtering:** ```python theme={null} # Filter for items containing both keywords (AND) changes = client.get_schedule_changes( schedule_id, filter="+python +remote" ) # Filter for items containing either keyword (OR) changes = client.get_schedule_changes( schedule_id, filter="python javascript" ) # Exclude items with a keyword changes = client.get_schedule_changes( schedule_id, filter="+engineer -manager" ) # Exact phrase matching changes = client.get_schedule_changes( schedule_id, filter='"machine learning"' ) ``` Use `mark_seen=False` to preview changes without affecting state. The `filter` parameter filters individual items within job results. *** ### regenerate\_webhook\_secret() Regenerate the webhook secret for a schedule. The old secret is immediately invalidated. ```python theme={null} regenerate_webhook_secret(schedule_id: str) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ------------- | | `schedule_id` | `str` | Yes | Schedule UUID | **Returns:** `Dict` with `schedule_id` and the new `webhook_secret` **Example:** ```python theme={null} result = client.regenerate_webhook_secret("880e8400-e29b-41d4-a716-446655440000") new_secret = result["webhook_secret"] print(f"New secret: {new_secret}") # Update your webhook handler with the new secret ``` **Raises:** `MeterError` if schedule has no webhook URL configured The new secret is returned only once. Store it securely and update your webhook handler before the next delivery. *** ## Workflow Methods For workflow methods (`create_workflow`, `run_workflow`, `wait_for_workflow`, etc.), see the dedicated [Workflow Methods](/api-reference/python/workflows) reference. *** ## Strategy Group Methods Manage collections of strategies with shared schedules and output schemas. See [Strategy Groups](/concepts/strategy-groups) for concepts. ### create\_strategy\_group() Create a new strategy group. ```python theme={null} create_strategy_group( name: str, description: Optional[str] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ------------------------ | | `name` | `str` | Yes | Name for the group | | `description` | `str` | No | Description of the group | **Returns:** `Dict` with `id`, `name`, `description`, `strategy_count`, `created_at`, `updated_at` **Example:** ```python theme={null} group = client.create_strategy_group( name="E-commerce Monitors", description="Product price tracking across 50 stores" ) print(f"Group ID: {group['id']}") ``` *** ### list\_strategy\_groups() List all strategy groups. ```python theme={null} list_strategy_groups( limit: int = 50, offset: int = 0 ) -> List[Dict] ``` **Parameters:** | Parameter | Type | Required | Description | | --------- | ----- | -------- | ---------------------------- | | `limit` | `int` | No | Max results (default: 50) | | `offset` | `int` | No | Results to skip (default: 0) | **Returns:** `List[Dict]` of strategy groups with strategy counts **Example:** ```python theme={null} groups = client.list_strategy_groups() for g in groups: print(f"{g['name']}: {g['strategy_count']} strategies") ``` *** ### get\_strategy\_group() Get group details including member strategies. ```python theme={null} get_strategy_group(group_id: str) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ---------- | ----- | -------- | ------------------- | | `group_id` | `str` | Yes | Strategy group UUID | **Returns:** `Dict` with `id`, `name`, `description`, `strategies` (list), `created_at`, `updated_at` **Example:** ```python theme={null} detail = client.get_strategy_group("aa0e8400-e29b-41d4-a716-446655440000") for s in detail["strategies"]: print(f" - {s['name']}") ``` *** ### update\_strategy\_group() Update a group's name or description. ```python theme={null} update_strategy_group( group_id: str, name: Optional[str] = None, description: Optional[str] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ------------------- | | `group_id` | `str` | Yes | Strategy group UUID | | `name` | `str` | No | New name | | `description` | `str` | No | New description | **Returns:** `Dict` with updated group details *** ### delete\_strategy\_group() Delete a group. Strategies become ungrouped (not deleted). ```python theme={null} delete_strategy_group(group_id: str) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ---------- | ----- | -------- | ------------------- | | `group_id` | `str` | Yes | Strategy group UUID | **Returns:** `Dict` with confirmation message *** ### add\_strategies\_to\_group() Add existing strategies to a group. ```python theme={null} add_strategies_to_group( group_id: str, strategy_ids: List[str] ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | -------------- | ----------- | -------- | --------------------- | | `group_id` | `str` | Yes | Strategy group UUID | | `strategy_ids` | `List[str]` | Yes | Strategy UUIDs to add | **Returns:** `Dict` with confirmation message **Example:** ```python theme={null} client.add_strategies_to_group( group_id=group["id"], strategy_ids=["550e8400-...", "660e8400-..."] ) ``` *** ### remove\_strategy\_from\_group() Remove a strategy from a group without deleting it. ```python theme={null} remove_strategy_from_group( group_id: str, strategy_id: str ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ----------------------- | | `group_id` | `str` | Yes | Strategy group UUID | | `strategy_id` | `str` | Yes | Strategy UUID to remove | **Returns:** `Dict` with confirmation message *** ### apply\_group\_schedule() Apply a schedule to all strategies in a group. ```python theme={null} apply_group_schedule( group_id: str, interval_seconds: Optional[int] = None, cron_expression: Optional[str] = None, webhook_url: Optional[str] = None, webhook_secret: Optional[str] = None, webhook_type: Optional[str] = None, webhook_metadata: Optional[Dict[str, Any]] = None ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | ------------------ | ------ | ----------- | ----------------------------------------------------------- | | `group_id` | `str` | Yes | Strategy group UUID | | `interval_seconds` | `int` | Conditional | Run every N seconds (min: 60) | | `cron_expression` | `str` | Conditional | Cron expression | | `webhook_url` | `str` | No | Webhook URL for notifications | | `webhook_secret` | `str` | No | Webhook secret (auto-generated if not provided) | | `webhook_type` | `str` | No | `'standard'`, `'slack'`, `'slack_workflow'`, or `'discord'` | | `webhook_metadata` | `Dict` | No | Custom JSON metadata for webhook payloads | **Returns:** `Dict` with `message`, `created` count, `updated` count **Example:** ```python theme={null} client.apply_group_schedule( group_id=group["id"], interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter" ) ``` Provide either `interval_seconds` or `cron_expression`, not both. *** ### delete\_group\_schedules() Delete all schedules for strategies in a group. ```python theme={null} delete_group_schedules(group_id: str) -> Dict ``` *** ### toggle\_group\_schedules() Enable or disable all schedules in a group. ```python theme={null} toggle_group_schedules( group_id: str, enabled: bool ) -> Dict ``` **Example:** ```python theme={null} # Pause all group schedules client.toggle_group_schedules(group["id"], enabled=False) # Resume client.toggle_group_schedules(group["id"], enabled=True) ``` *** ### apply\_group\_schema() Apply an output schema to all strategies in a group. Triggers async regeneration. ```python theme={null} apply_group_schema( group_id: str, output_schema: Dict[str, Any] ) -> Dict ``` **Parameters:** | Parameter | Type | Required | Description | | --------------- | ------ | -------- | ------------------------------------- | | `group_id` | `str` | Yes | Strategy group UUID | | `output_schema` | `Dict` | Yes | JSON schema defining output structure | **Returns:** `Dict` with `message` and `strategy_count` **Example:** ```python theme={null} client.apply_group_schema( group_id=group["id"], output_schema={ "title": "string", "price": "number", "in_stock": "boolean" } ) ``` *** ### get\_schema\_progress() Poll progress of a group schema regeneration. ```python theme={null} get_schema_progress(group_id: str) -> Dict ``` *** ### test\_group\_webhook() Send a test webhook using a group's webhook configuration. ```python theme={null} test_group_webhook( group_id: str, webhook_url: str, webhook_secret: Optional[str] = None, webhook_type: Optional[str] = None, webhook_metadata: Optional[Dict[str, Any]] = None ) -> Dict ``` **Returns:** `Dict` with `success`, `status_code`, `message` *** ## Error Handling All methods raise `MeterError` on API errors. See [Error Handling](/api-reference/python/errors) for details. ```python theme={null} from meter_sdk import MeterClient, MeterError client = MeterClient(api_key="sk_live_...") try: strategy = client.generate_strategy(url, description, name) except MeterError as e: print(f"Error: {e}") # Handle error appropriately ``` ## Next steps Learn about error handling and exceptions See the SDK in action with examples Deep dive into strategy concepts Understanding job lifecycle ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Error Handling Source: https://docs.meter.sh/api-reference/python/errors Exception handling and error codes in the Meter Python SDK # Error Handling The Meter SDK uses exceptions to signal errors. All API-related errors raise `MeterError`, which you should catch and handle appropriately. ## MeterError The base exception for all Meter SDK errors. ```python theme={null} from meter_sdk import MeterError class MeterError(Exception): """Base exception for Meter SDK errors""" pass ``` ### Catching errors ```python theme={null} from meter_sdk import MeterClient, MeterError client = MeterClient(api_key="sk_live_...") try: strategy = client.generate_strategy( url="https://example.com", description="Extract data", name="Test Strategy" ) except MeterError as e: print(f"API error: {e}") # Handle error appropriately ``` ## Common error scenarios ### 401 Unauthorized **Cause:** Invalid or missing API key ```python theme={null} try: client = MeterClient(api_key="invalid_key") strategies = client.list_strategies() except MeterError as e: # "Invalid or missing API key" print(f"Authentication failed: {e}") ``` **Solutions:** * Verify your API key is correct * Check that the key hasn't been deleted * Ensure you're using the full key (starts with `sk_live_`) ### 400 Bad Request **Cause:** Invalid request parameters ```python theme={null} try: # Missing required parameter strategy = client.generate_strategy( url="https://example.com", description="", # Empty description name="Test" ) except MeterError as e: # "Invalid request parameters" print(f"Validation error: {e}") ``` **Solutions:** * Check all required parameters are provided * Verify parameter types are correct * Ensure values are valid (e.g., URL is well-formed) ### 404 Not Found **Cause:** Resource doesn't exist ```python theme={null} try: strategy = client.get_strategy("invalid-uuid") except MeterError as e: # "Strategy not found" print(f"Resource not found: {e}") ``` **Solutions:** * Verify the UUID is correct * Check the resource hasn't been deleted * Ensure you have permission to access the resource ### 429 Too Many Requests **Cause:** Rate limit exceeded on strategy generation ```python theme={null} try: strategy = client.generate_strategy(url, description, name) except MeterError as e: # "Please slow down - limits on strategy generation" print(f"Rate limited: {e}") ``` **Solutions:** * Wait before retrying (the API returns a `Retry-After` header with seconds to wait) * Reduce request frequency * Use exponential backoff ### 500 Internal Server Error **Cause:** Server-side error ```python theme={null} try: job = client.create_job(strategy_id, url) except MeterError as e: # "Internal server error" print(f"Server error: {e}") ``` **Solutions:** * Retry the request after a short delay * If persistent, contact support * Check API status page for incidents ### 503 Service Unavailable **Cause:** LLM service temporarily unavailable ```python theme={null} try: strategy = client.generate_strategy(url, description, name) except MeterError as e: # "Service temporarily unavailable" print(f"Service unavailable: {e}") ``` **Solutions:** * Retry the request after 30-60 seconds * This is usually temporary ## Timeout errors When waiting for jobs with a timeout: ```python theme={null} from meter_sdk import MeterError try: completed = client.wait_for_job( job_id, timeout=60.0 # 1 minute ) except MeterError as e: # "Job timed out after 60 seconds" print(f"Timeout: {e}") # Check job status manually job = client.get_job(job_id) if job['status'] == 'failed': print(f"Job failed: {job['error']}") else: print(f"Job still {job['status']}, wait longer") ``` ## Best practices ### Always catch MeterError ```python theme={null} from meter_sdk import MeterClient, MeterError client = MeterClient(api_key="sk_live_...") try: # API calls strategy = client.generate_strategy(url, description, name) job = client.create_job(strategy['strategy_id'], url) result = client.wait_for_job(job['job_id']) except MeterError as e: # Log error logger.error(f"Meter API error: {e}") # Handle appropriately send_alert(f"Scraping failed: {e}") ``` ### Implement retry logic For transient errors (429, 500, 503, network issues): ```python theme={null} import time from meter_sdk import MeterError def generate_strategy_with_retry(client, url, description, name, max_retries=3): """Generate strategy with retry logic""" for attempt in range(max_retries): try: return client.generate_strategy(url, description, name) except MeterError as e: error_str = str(e).lower() # Rate limited - wait longer (respect Retry-After) if "429" in str(e) or "slow down" in error_str: wait_time = 60 # Default from Retry-After header print(f"Rate limited. Waiting {wait_time}s...") time.sleep(wait_time) continue # Server errors - exponential backoff if "500" in str(e) or "503" in str(e) or "unavailable" in error_str: if attempt < max_retries - 1: wait_time = 2 ** attempt print(f"Retry {attempt + 1}/{max_retries} in {wait_time}s") time.sleep(wait_time) else: raise # Max retries exceeded else: raise # Non-retryable error ``` ### Graceful degradation Handle errors without crashing: ```python theme={null} from meter_sdk import MeterError def fetch_latest_data(client, schedule_id): """Fetch changes with fallback""" try: changes = client.get_schedule_changes(schedule_id) return changes['changes'] except MeterError as e: logger.warning(f"Failed to fetch changes: {e}") # Fallback: use cached data return load_from_cache(schedule_id) ``` ### Log errors for debugging ```python theme={null} import logging from meter_sdk import MeterError logger = logging.getLogger(__name__) try: strategy = client.generate_strategy(url, description, name) except MeterError as e: # Log with context logger.error( "Strategy generation failed", extra={ "url": url, "description": description, "error": str(e) } ) raise ``` ## Error response format API errors return JSON with details: ```json theme={null} { "detail": "Error message describing what went wrong" } ``` The SDK extracts this message and includes it in the `MeterError` exception. ## Handling specific error types ### Strategy generation failures ```python theme={null} try: strategy = client.generate_strategy(url, description, name) except MeterError as e: if "not accessible" in str(e).lower(): print("URL cannot be accessed. Check if it's public.") elif "invalid url" in str(e).lower(): print("URL format is invalid") else: print(f"Generation failed: {e}") ``` ### Job failures ```python theme={null} job = client.get_job(job_id) if job['status'] == 'failed': error = job['error'] if "selector not found" in error.lower(): print("Website structure changed. Regenerate strategy.") elif "timeout" in error.lower(): print("Website took too long to respond") else: print(f"Job failed: {error}") ``` ## Error monitoring Set up error tracking for production: ```python theme={null} import sentry_sdk from meter_sdk import MeterError sentry_sdk.init(dsn="your-sentry-dsn") try: strategy = client.generate_strategy(url, description, name) except MeterError as e: # Automatically captured by Sentry sentry_sdk.capture_exception(e) raise ``` ## Next steps Explore all SDK methods See error handling in practice Learn SDK best practices View REST API error codes ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Installation Source: https://docs.meter.sh/api-reference/python/installation Install and set up the Meter Python SDK # Python SDK Installation The Meter Python SDK provides a clean, Pythonic interface for all Meter API operations. ## Requirements * Python 3.8 or later * pip or uv for package management ## Install with pip ```bash theme={null} pip install meter-sdk ``` ## Install with uv ```bash theme={null} uv add meter-sdk ``` ## Install from source For development or to use the latest unreleased features: ```bash theme={null} git clone https://github.com/yourusername/meter-sdk cd meter-sdk pip install -e . ``` ## Verify installation ```python theme={null} from meter_sdk import MeterClient print("Meter SDK installed successfully!") ``` ## Set up authentication Store your API key as an environment variable: ```bash theme={null} export METER_API_KEY="sk_live_your_key_here" ``` For persistent configuration, add to your `~/.bashrc`, `~/.zshrc`, or `.env` file: ```bash .env theme={null} METER_API_KEY=sk_live_your_key_here ``` Load environment variables in Python: ```python theme={null} from dotenv import load_dotenv import os load_dotenv() # Load from .env file api_key = os.getenv("METER_API_KEY") ``` ## Quick test Verify your setup with a quick test: ```python theme={null} from meter_sdk import MeterClient import os # Initialize client client = MeterClient(api_key=os.getenv("METER_API_KEY")) # List your strategies (should return empty list if you haven't created any) strategies = client.list_strategies() print(f"Found {len(strategies)} strategies") # Success! You're ready to use Meter ``` ## Dependencies The SDK has minimal dependencies: * `requests`: HTTP client for API calls * `typing-extensions`: Type hints for older Python versions All dependencies are installed automatically. ## Updating Keep your SDK up to date: ```bash theme={null} pip install --upgrade meter-sdk ``` ## Troubleshooting **Solution**: Install the package: ```bash theme={null} pip install meter-sdk ``` If using a virtual environment, ensure it's activated. **Problem**: Package installed but import fails **Solutions**: * Check you're in the correct Python environment: `which python` * Verify installation: `pip show meter-sdk` * Try reinstalling: `pip uninstall meter-sdk && pip install meter-sdk` **Problem**: Dependency conflicts with other packages **Solution**: Use a virtual environment: ```bash theme={null} python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install meter-sdk ``` ## Next steps Generate your first strategy in 5 minutes Explore the MeterClient API Learn about API key management See real-world SDK usage ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Job Methods Source: https://docs.meter.sh/api-reference/python/jobs Execute scrapes and retrieve results # Job Methods Job methods allow you to execute scrapes, check status, and retrieve results. See the [MeterClient reference](/api-reference/python/client#job-methods) for complete method signatures. ## Quick reference | Method | Description | | ------------------------ | ------------------------------------ | | `create_job()` | Create a new scrape job (async) | | `execute_job()` | Create and run a job synchronously | | `get_job()` | Get job status and results | | `wait_for_job()` | Wait for job completion with polling | | `list_jobs()` | List jobs with filtering | | `compare_jobs()` | Compare two jobs for changes | | `get_strategy_history()` | Get timeline of jobs for a strategy | ## Common workflows ### Create and wait ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") # Create job job = client.create_job( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products" ) # Wait for completion completed = client.wait_for_job(job['job_id'], timeout=300) # Process results for item in completed['results']: print(item) ``` ### Jobs with API parameters For API-based strategies, override parameters at runtime: ```python theme={null} # Override API parameters for this specific job job = client.create_job( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/api/jobs", parameters={ "page": 2, "limit": 100, "category": "engineering", "location": "remote" } ) completed = client.wait_for_job(job['job_id']) print(f"Found {completed['item_count']} items") ``` ### Batch jobs Scrape multiple URLs in a single request: ```python theme={null} # Create batch job for multiple URLs job = client.create_job( strategy_id="550e8400-e29b-41d4-a716-446655440000", urls=[ "https://example.com/products/1", "https://example.com/products/2", "https://example.com/products/3" ] ) # Batch jobs return batch_id for tracking print(f"Batch ID: {job['batch_id']}") ``` ### Poll manually ```python theme={null} import time job = client.create_job(strategy_id, url) while True: status = client.get_job(job['job_id']) if status['status'] == 'completed': print(f"Done! {status['item_count']} items") break elif status['status'] == 'failed': print(f"Failed: {status['error']}") break print(f"Status: {status['status']}") time.sleep(2) ``` ### Compare for changes ```python theme={null} # Get last two jobs jobs = client.list_jobs(strategy_id=strategy_id, limit=2) if len(jobs) >= 2: comparison = client.compare_jobs(jobs[0]['job_id'], jobs[1]['job_id']) if not comparison['content_hash_match']: print("Content has changed!") else: print("No changes detected") ``` ### Monitor failures ```python theme={null} # Check for recent failures failed = client.list_jobs( strategy_id=strategy_id, status='failed', limit=5 ) if len(failed) > 0: print(f"Warning: {len(failed)} recent failures") for job in failed: print(f" - {job['job_id']}: {job['error']}") ``` ## See also Complete parameter and return type documentation Understand job lifecycle and status Job endpoints in the REST API Learn about job comparison ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Schedule Methods Source: https://docs.meter.sh/api-reference/python/schedules Automate scraping with recurring schedules # Schedule Methods Schedule methods allow you to set up automated, recurring scrapes. See the [MeterClient reference](/api-reference/python/client#schedule-methods) for complete method signatures. ## Quick reference | Method | Description | | ----------------------------- | ---------------------------------------- | | `create_schedule()` | Create a recurring schedule | | `list_schedules()` | List all schedules | | `update_schedule()` | Modify schedule settings | | `delete_schedule()` | Delete a schedule | | `get_schedule_changes()` | Get unseen changes (pull-based) | | `regenerate_webhook_secret()` | Regenerate webhook secret for a schedule | ## Common workflows ### Interval-based monitoring ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") # Run every hour schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", interval_seconds=3600 ) print(f"Schedule created: {schedule['schedule_id']}") print(f"Next run: {schedule['next_run_at']}") ``` ### Cron-based monitoring ```python theme={null} # Daily at 9 AM schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", cron_expression="0 9 * * *" ) # Weekdays at 8 AM schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", cron_expression="0 8 * * 1-5" ) ``` ### With webhook ```python theme={null} schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter" ) # Store the auto-generated webhook secret (shown only once) print(f"Webhook secret: {schedule.get('webhook_secret')}") ``` ### With webhook metadata and type Attach custom metadata to every webhook payload and specify the delivery format: ```python theme={null} # Standard webhook with metadata schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter", webhook_metadata={"project": "price-monitor", "env": "prod"}, webhook_type="standard" ) # Slack webhook (auto-detected from URL) schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/products", interval_seconds=3600, webhook_url="https://hooks.slack.com/services/T.../B.../xxx" ) ``` ### Regenerate webhook secret If a webhook secret is compromised, regenerate it: ```python theme={null} result = client.regenerate_webhook_secret(schedule_id) new_secret = result["webhook_secret"] # Update your webhook handler with the new secret ``` ### With API parameters For API-based strategies, set default parameters for all scheduled runs: ```python theme={null} schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", url="https://example.com/api/listings", interval_seconds=3600, parameters={ "category": "electronics", "sort": "newest", "limit": 100 } ) # Update parameters later client.update_schedule( schedule['schedule_id'], parameters={"category": "clothing", "limit": 200} ) ``` ### Multiple URLs Monitor multiple pages on a single schedule: ```python theme={null} schedule = client.create_schedule( strategy_id="550e8400-e29b-41d4-a716-446655440000", urls=[ "https://example.com/products/electronics", "https://example.com/products/clothing", "https://example.com/products/home" ], interval_seconds=3600 ) ``` ### Pull-based change detection ```python theme={null} import time while True: # Check for changes every hour changes = client.get_schedule_changes( schedule_id="880e8400-e29b-41d4-a716-446655440000", mark_seen=True ) if changes['count'] > 0: print(f"Processing {changes['count']} changes") for change in changes['changes']: # Process change['results'] update_database(change['results']) time.sleep(3600) ``` ### Keyword filtering Filter results to only include items matching specific keywords: ```python theme={null} # Filter for articles mentioning both "jfk" AND "tariff" changes = client.get_schedule_changes( schedule_id="880e8400-e29b-41d4-a716-446655440000", filter="+jfk +tariff" ) # Filter for articles mentioning "jfk" OR "elon" changes = client.get_schedule_changes( schedule_id="880e8400-e29b-41d4-a716-446655440000", filter="jfk elon" ) # Filter for articles with "jfk" but NOT "biden" changes = client.get_schedule_changes( schedule_id="880e8400-e29b-41d4-a716-446655440000", filter="+jfk -biden" ) # Exact phrase matching changes = client.get_schedule_changes( schedule_id="880e8400-e29b-41d4-a716-446655440000", filter='"elon musk"' ) ``` The filter applies to individual items within results. Only matching items are returned. Jobs with zero matching items are excluded entirely. ### Manage schedules ```python theme={null} # List all schedules schedules = client.list_schedules() for schedule in schedules: print(f"{schedule['schedule_id']}: {schedule['enabled']}") # Disable temporarily client.update_schedule(schedule_id, enabled=False) # Change interval client.update_schedule(schedule_id, interval_seconds=7200) # Delete client.delete_schedule(schedule_id) ``` ### Pause and resume ```python theme={null} # Pause during maintenance client.update_schedule(schedule_id, enabled=False) # Do maintenance work regenerate_strategy() # Resume client.update_schedule(schedule_id, enabled=True) ``` ## See also Complete parameter and return type documentation Understand schedule types and timing Schedule endpoints in the REST API Guide to using get\_schedule\_changes() ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Strategy Methods Source: https://docs.meter.sh/api-reference/python/strategies Generate, refine, and manage extraction strategies # Strategy Methods Strategy methods allow you to create and manage AI-generated extraction strategies. See the [MeterClient reference](/api-reference/python/client#strategy-methods) for complete method signatures. ## Quick reference | Method | Description | | --------------------- | ------------------------------------------- | | `generate_strategy()` | Generate a new extraction strategy using AI | | `refine_strategy()` | Improve an existing strategy with feedback | | `list_strategies()` | List all strategies | | `get_strategy()` | Get details for a specific strategy | | `delete_strategy()` | Delete a strategy and associated data | ## Common workflows ### Generate and refine ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") # Generate initial strategy result = client.generate_strategy( url="https://shop.com/products", description="Extract product name, price, and image", name="Product Scraper" ) # Check preview print(result['preview_data'][:3]) # Refine if needed if 'sku' not in result['preview_data'][0]: refined = client.refine_strategy( strategy_id=result['strategy_id'], feedback="Also extract the product SKU" ) print(refined['preview_data'][:3]) ``` ### API-based scraping For sites that load data via JavaScript APIs, use `force_api=True` to capture the underlying API: ```python theme={null} # Force API-based capture result = client.generate_strategy( url="https://api-heavy-site.com/listings", description="Extract all listing data", name="Listings API Scraper", force_api=True ) # Check scraper type print(f"Scraper type: {result['scraper_type']}") # 'api' or 'css' # For API strategies, available parameters are returned if result.get('api_parameters'): print(f"Available parameters: {result['api_parameters']}") # e.g., {'page': 1, 'limit': 20, 'category': 'all'} ``` API-based strategies capture underlying API calls instead of using CSS selectors. This is useful for dynamic sites where data is loaded via JavaScript. ### List and filter ```python theme={null} # Get all strategies strategies = client.list_strategies() # Find by name product_strategies = [ s for s in strategies if 'product' in s['name'].lower() ] # Most recent recent = client.list_strategies(limit=5, offset=0) ``` ### Batch operations ```python theme={null} # Delete old strategies strategies = client.list_strategies(limit=100) for strategy in strategies: # Delete if created more than 30 days ago if is_old(strategy['created_at']): client.delete_strategy(strategy['strategy_id']) print(f"Deleted {strategy['name']}") ``` ## See also Complete parameter and return type documentation Understand strategy lifecycle and best practices Strategy endpoints in the REST API Generate your first strategy ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Workflow Methods Source: https://docs.meter.sh/api-reference/python/workflows Build and run DAG-based scraping pipelines # Workflow Methods Workflow methods allow you to build multi-step scraping pipelines where the output of one scraper feeds into the next. See the [Workflows concept page](/concepts/workflows) for an introduction. ## Quick reference ### Workflow classes | Class | Description | | -------------- | ------------------------------------------ | | `Workflow` | Define a DAG pipeline with nodes and edges | | `WorkflowNode` | A single scraping step in the workflow | | `Filter` | Build filter conditions for edges | ### Client methods | Method | Description | | ---------------------------- | ---------------------------------------------- | | `create_workflow()` | Create a workflow from a Workflow object | | `run_workflow()` | Run a workflow (create + run, or run existing) | | `wait_for_workflow()` | Poll a workflow run until completion | | `get_workflow()` | Get workflow details | | `list_workflows()` | List all workflows | | `delete_workflow()` | Delete a workflow | | `get_workflow_run()` | Get details of a specific run | | `list_workflow_runs()` | List run history | | `get_workflow_output()` | Get latest run's results | | `schedule_workflow()` | Schedule a workflow on interval or cron | | `list_workflow_schedules()` | List schedules for a workflow | | `update_workflow_schedule()` | Update a workflow schedule | | `delete_workflow_schedule()` | Delete a workflow schedule | ## Workflow class ```python theme={null} from meter_sdk.workflow import Workflow ``` ### Constructor ```python theme={null} Workflow(name: str, description: Optional[str] = None) ``` | Parameter | Type | Required | Description | | ------------- | ----- | -------- | ------------------------------------- | | `name` | `str` | Yes | Workflow name | | `description` | `str` | No | Description of what the workflow does | ### start() Create the starting node with static URLs. ```python theme={null} workflow.start( name: str, strategy_id: str, urls: List[str], parameters: Optional[Dict[str, Any]] = None ) -> WorkflowNode ``` | Parameter | Type | Required | Description | | ------------- | ----------- | -------- | ------------------------------------------ | | `name` | `str` | Yes | Unique node identifier within the workflow | | `strategy_id` | `str` | Yes | Strategy UUID to use for scraping | | `urls` | `List[str]` | Yes | List of URLs to scrape | | `parameters` | `Dict` | No | Static API parameter overrides | **Returns:** `WorkflowNode` — chain with `.then()` for downstream nodes ```python theme={null} workflow = Workflow("Job Scraper") index = workflow.start("index", strategy_id, urls=["https://jobs.com"]) ``` ### to\_dict() Serialize the workflow for API requests. You typically don't need to call this directly — `create_workflow()` and `run_workflow()` handle serialization. ```python theme={null} workflow.to_dict() -> Dict[str, Any] ``` ## WorkflowNode class ### then() Chain a downstream node that receives data from this node. ```python theme={null} node.then( name: str, strategy_id: str, url_field: Optional[str] = None, parameter_config: Optional[Dict[str, Any]] = None, parameters: Optional[Dict[str, Any]] = None, filter: Optional[Dict[str, Any]] = None ) -> WorkflowNode ``` | Parameter | Type | Required | Description | | ------------------ | ------ | -------- | --------------------------------------------------------------------------------------------------------- | | `name` | `str` | Yes | Unique node identifier | | `strategy_id` | `str` | Yes | Strategy UUID for the downstream scraper | | `url_field` | `str` | No | Field name in upstream results containing URLs to scrape. If provided, sets input type to `upstream_urls` | | `parameter_config` | `Dict` | No | Map upstream result fields to strategy parameters | | `parameters` | `Dict` | No | Static API parameter overrides | | `filter` | `Dict` | No | Filter condition (use `Filter` class to build) | **Returns:** `WorkflowNode` — the new downstream node (for further chaining) If `url_field` is provided, the node uses `upstream_urls` input type (extracts URLs from upstream results). If neither `url_field` nor `parameter_config` is provided, it uses `upstream_data` input type (passes full upstream results as context). ```python theme={null} # Extract URLs from upstream results details = index.then("details", detail_strategy_id, url_field="job_url") # With filtering tech_jobs = index.then( "tech_jobs", detail_strategy_id, url_field="job_url", filter=Filter.contains("category", "engineering") ) # Chain further sub_details = details.then("sub_details", sub_strategy_id, url_field="company_url") ``` ## Filter class ```python theme={null} from meter_sdk.workflow import Filter ``` Build filter conditions for workflow edges. All string operators accept an optional `case_sensitive` parameter (default: `False`). ### String operators ```python theme={null} Filter.contains(field: str, value: str, case_sensitive: bool = False) -> dict Filter.not_contains(field: str, value: str, case_sensitive: bool = False) -> dict Filter.equals(field: str, value: str, case_sensitive: bool = False) -> dict Filter.not_equals(field: str, value: str, case_sensitive: bool = False) -> dict Filter.regex_match(field: str, pattern: str, case_sensitive: bool = False) -> dict ``` ### Existence operators ```python theme={null} Filter.exists(field: str) -> dict Filter.not_exists(field: str) -> dict ``` ### Comparison operators ```python theme={null} Filter.gt(field: str, value: str) -> dict Filter.lt(field: str, value: str) -> dict ``` ### Logical combinators ```python theme={null} Filter.all(*conditions) -> dict # AND — all conditions must match Filter.any(*conditions) -> dict # OR — at least one must match ``` ### Examples ```python theme={null} from meter_sdk.workflow import Filter # Single condition f = Filter.contains("url", "/products/") # AND: all conditions must match f = Filter.all( Filter.contains("category", "electronics"), Filter.gt("price", "50"), Filter.exists("in_stock") ) # OR: at least one must match f = Filter.any( Filter.equals("status", "sale"), Filter.equals("status", "clearance") ) # Case-sensitive match f = Filter.equals("sku", "ABC-123", case_sensitive=True) # Pass to then() details = index.then("details", strategy_id, url_field="link", filter=f) ``` ## Client methods ### create\_workflow() Create a workflow from a `Workflow` object. ```python theme={null} client.create_workflow(workflow: Workflow) -> Dict ``` | Parameter | Type | Required | Description | | ---------- | ---------- | -------- | ------------------------------------------ | | `workflow` | `Workflow` | Yes | Workflow object defining the DAG structure | **Returns:** Created workflow details including `id` ```python theme={null} workflow = Workflow("My Workflow") index = workflow.start("index", strategy_id, urls=["https://example.com"]) details = index.then("details", detail_strategy_id, url_field="link") created = client.create_workflow(workflow) print(f"Workflow ID: {created['id']}") ``` *** ### run\_workflow() Run a workflow. Accepts either a `Workflow` object (creates then runs) or a workflow ID string. ```python theme={null} client.run_workflow( workflow_or_id: Union[Workflow, str], force: bool = False, wait: bool = True, timeout: float = 3600 ) -> Dict ``` | Parameter | Type | Required | Description | | ---------------- | ------------------- | -------- | -------------------------------------------------------- | | `workflow_or_id` | `Workflow` or `str` | Yes | Workflow object (creates then runs) or workflow ID | | `force` | `bool` | No | Force re-run, skipping change detection (default: False) | | `wait` | `bool` | No | Block until completion (default: True) | | `timeout` | `float` | No | Maximum seconds to wait (default: 3600) | **Returns:** Completed run details including `status` and `node_executions` (if `wait=True`) or run details with status (if `wait=False`). Use `get_workflow_output()` to fetch results. **Raises:** `MeterError` if run fails or times out ```python theme={null} # Create and run in one step workflow = Workflow("My Scraper") index = workflow.start("index", strategy_id, urls=["https://example.com"]) result = client.run_workflow(workflow) # Run an existing workflow by ID result = client.run_workflow("workflow-uuid") # Fire and forget (don't wait) run = client.run_workflow("workflow-uuid", wait=False) print(f"Run started: {run['id']}") # Force re-run (skip change detection) result = client.run_workflow("workflow-uuid", force=True) ``` *** ### wait\_for\_workflow() Poll a workflow run until it completes. ```python theme={null} client.wait_for_workflow( workflow_id: str, run_id: str, poll_interval: float = 5.0, timeout: float = 3600 ) -> Dict ``` | Parameter | Type | Required | Description | | --------------- | ------- | -------- | --------------------------------------- | | `workflow_id` | `str` | Yes | Workflow UUID | | `run_id` | `str` | Yes | Run UUID | | `poll_interval` | `float` | No | Seconds between polls (default: 5.0) | | `timeout` | `float` | No | Maximum seconds to wait (default: 3600) | **Returns:** Completed run with results **Raises:** `MeterError` if run fails or times out ```python theme={null} # Start a run without waiting run = client.run_workflow("workflow-uuid", wait=False) # Wait for it later completed = client.wait_for_workflow("workflow-uuid", run["id"], timeout=600) ``` *** ### get\_workflow() Get workflow details including nodes and edges. ```python theme={null} client.get_workflow(workflow_id: str) -> Dict ``` *** ### list\_workflows() List all workflows. ```python theme={null} client.list_workflows( limit: int = 50, offset: int = 0 ) -> List[Dict] ``` *** ### delete\_workflow() Delete a workflow and all associated runs and schedules. ```python theme={null} client.delete_workflow(workflow_id: str) -> Dict ``` This deletes the workflow, all run history, and any associated schedules. *** ### get\_workflow\_run() Get details of a specific workflow run, including node execution results. ```python theme={null} client.get_workflow_run(workflow_id: str, run_id: str) -> Dict ``` **Returns:** Run details including `status` and `node_executions` *** ### list\_workflow\_runs() List run history for a workflow. ```python theme={null} client.list_workflow_runs( workflow_id: str, limit: int = 20, offset: int = 0 ) -> List[Dict] ``` *** ### get\_workflow\_output() Get the latest completed run's results. ```python theme={null} client.get_workflow_output( workflow_id: str, flat: bool = False ) -> Dict ``` | Parameter | Type | Required | Description | | ------------- | ------ | -------- | --------------------------------------------------------------------------- | | `workflow_id` | `str` | Yes | Workflow UUID | | `flat` | `bool` | No | Return flat per-URL results instead of grouped by strategy (default: False) | **Returns:** Latest run output. Default format uses `final_results_by_url_grouped`; with `flat=True` uses `final_results_by_url`. Also includes `status`, `changed_since_previous`, `run_id`, `workflow_id`, `completed_at`. **Raises:** `MeterError` if no completed runs exist ```python theme={null} # Grouped by URL and strategy (default) output = client.get_workflow_output("workflow-uuid") for url, strategies in output["final_results_by_url_grouped"].items(): for strategy, items in strategies.items(): print(f"{strategy}: {len(items)} items") # Flat per-URL results output = client.get_workflow_output("workflow-uuid", flat=True) for url, items in output["final_results_by_url"].items(): print(f"{url}: {len(items)} items") ``` *** ## Scheduling methods ### schedule\_workflow() Schedule a workflow to run on an interval or cron expression. ```python theme={null} client.schedule_workflow( workflow_id: str, interval_seconds: Optional[int] = None, cron_expression: Optional[str] = None, webhook_url: Optional[str] = None, webhook_metadata: Optional[Dict[str, Any]] = None, webhook_secret: Optional[str] = None, webhook_type: Optional[str] = None ) -> Dict ``` | Parameter | Type | Required | Description | | ------------------ | ------ | ----------- | ------------------------------------------------------ | | `workflow_id` | `str` | Yes | Workflow UUID | | `interval_seconds` | `int` | Conditional | Run every N seconds (or use `cron_expression`) | | `cron_expression` | `str` | Conditional | Cron expression (e.g., `"0 9 * * *"`) | | `webhook_url` | `str` | No | Webhook URL to receive results | | `webhook_metadata` | `Dict` | No | Custom JSON metadata included in every webhook payload | | `webhook_secret` | `str` | No | Secret for `X-Webhook-Secret` header | | `webhook_type` | `str` | No | `'standard'` or `'slack'` (default: `'standard'`) | ```python theme={null} # Run every hour client.schedule_workflow(workflow_id, interval_seconds=3600) # Daily at 9 AM with webhook client.schedule_workflow( workflow_id, cron_expression="0 9 * * *", webhook_url="https://your-app.com/webhook", webhook_metadata={"project": "my-project"} ) ``` *** ### list\_workflow\_schedules() ```python theme={null} client.list_workflow_schedules(workflow_id: str) -> List[Dict] ``` *** ### update\_workflow\_schedule() ```python theme={null} client.update_workflow_schedule( workflow_id: str, schedule_id: str, enabled: Optional[bool] = None, interval_seconds: Optional[int] = None, cron_expression: Optional[str] = None, webhook_url: Optional[str] = None, webhook_metadata: Optional[Dict[str, Any]] = None, webhook_secret: Optional[str] = None, webhook_type: Optional[str] = None ) -> Dict ``` *** ### delete\_workflow\_schedule() ```python theme={null} client.delete_workflow_schedule(workflow_id: str, schedule_id: str) -> Dict ``` *** ## Complete example ```python theme={null} from meter_sdk import MeterClient from meter_sdk.workflow import Workflow, Filter client = MeterClient(api_key="sk_live_...") # Step 1: Build the workflow workflow = Workflow("E-commerce Scraper", description="Scrape product index then detail pages") # Root node: scrape the product listing index = workflow.start( "product_index", index_strategy_id, urls=["https://shop.com/products"] ) # Downstream: only follow electronics links electronics = index.then( "electronics", detail_strategy_id, url_field="product_url", filter=Filter.contains("category", "electronics") ) # Step 2: Run the workflow run = client.run_workflow(workflow) # Step 3: Get results output = client.get_workflow_output(run["workflow_id"]) for url, strategies in output["final_results_by_url_grouped"].items(): print(f"\n{url}:") for strategy, items in strategies.items(): print(f" {strategy} ({len(items)} items):") for item in items[:3]: print(f" {item}") # Step 4: Schedule for daily runs client.schedule_workflow( run["workflow_id"], cron_expression="0 9 * * *", webhook_url="https://your-app.com/webhook" ) ``` ## See also Understand workflow architecture and patterns Workflow endpoints in the REST API Full client method documentation Learn about the strategies workflows use ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Account Endpoints Source: https://docs.meter.sh/api-reference/rest/account Account-level state — strategy quota and plan tier # Account Endpoints Read-only endpoints for account-level state. Use these when you need quota or plan information without making a creation call. ## Get strategy quota Return current strategy creation quota over the rolling 30-day window. This endpoint is a pure read — it does **not** consume quota. ```http theme={null} GET /api/account/quota ``` ### Response ```json theme={null} { "strategy_quota": { "used": 12, "limit": 100, "tier": "pro", "reset_at": "2026-06-14T09:21:00Z" } } ``` | Field | Type | Description | | ------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `strategy_quota.used` | integer | Strategies counted in the rolling 30-day window (active + created-then-deleted). | | `strategy_quota.limit` | integer | Plan limit. `-1` indicates an unlimited (enterprise) plan. | | `strategy_quota.tier` | string | `free` \| `hobby` \| `pro` \| `enterprise`. | | `strategy_quota.reset_at` | string \| null | ISO 8601 UTC timestamp of when the next quota slot frees (oldest counted strategy + 30 days). Omitted when the plan is unlimited or when there is no usage in the window. | ### Example ```bash theme={null} curl https://api.meter.sh/api/account/quota \ -H "Authorization: Bearer sk_live_..." ``` The same numbers are also returned as response headers (`X-Strategy-Quota-Used`, `-Limit`, `-Reset`, `-Tier`) on every strategy-creation call. See [Strategy quota headers](/api-reference/rest/strategies#strategy-quota-headers) for the header-based variant — convenient when you'd already be making a creation request and want to avoid an extra round-trip. ## Error responses | Status | Description | | ------ | -------------------------- | | `401` | Invalid or missing API key | | `500` | Internal server error | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Create strategies (and see quota headers on the response) Learn about strategies and quota ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # REST API Authentication Source: https://docs.meter.sh/api-reference/rest/authentication Authenticate HTTP requests to the Meter API # REST API Authentication All Meter API requests require authentication using an API key in the `Authorization` header. ## Authentication header Include your API key using the Bearer authentication scheme: ``` Authorization: Bearer sk_live_your_key_here ``` ## Example requests ### Using curl ```bash theme={null} curl https://api.meter.sh/api/strategies \ -H "Authorization: Bearer sk_live_your_key_here" \ -H "Content-Type: application/json" ``` ### Using JavaScript (fetch) ```javascript theme={null} const response = await fetch('https://api.meter.sh/api/strategies', { headers: { 'Authorization': `Bearer ${process.env.METER_API_KEY}`, 'Content-Type': 'application/json' } }); const data = await response.json(); ``` ### Using Python (requests) ```python theme={null} import requests import os headers = { 'Authorization': f'Bearer {os.getenv("METER_API_KEY")}', 'Content-Type': 'application/json' } response = requests.get('https://api.meter.sh/api/strategies', headers=headers) data = response.json() ``` ## Getting an API key See the main [Authentication](/authentication) guide for instructions on obtaining and managing API keys. ## Error responses ### 401 Unauthorized Missing or invalid API key: ```json theme={null} { "detail": "Invalid or missing API key" } ``` **Solutions:** * Verify your API key is correct * Ensure the `Authorization` header is included * Check the key hasn't been deleted ### 403 Forbidden Valid key but insufficient permissions: ```json theme={null} { "detail": "You do not have permission to access this resource" } ``` **Solutions:** * Verify you're accessing your own resources * Check the resource exists ## Best practices Never hardcode API keys in source code Always use HTTPS, never HTTP Generate new keys periodically Track API calls in your dashboard ## Next steps Learn how to create and manage API keys Explore the REST API Start making API calls ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Discovery Endpoints Source: https://docs.meter.sh/api-reference/rest/discovery REST API endpoints for site crawling and URL discovery # Discovery Endpoints Discover URLs on websites and execute batch scrapes via HTTP. ## Start discovery Start URL discovery using sitemap, pagination, or link pattern. ```http theme={null} POST /discover ``` ### Request body ```json theme={null} { "discovery": { "method": "sitemap", "sitemap_url": "https://shop.com/sitemap.xml", "url_pattern": "products/*/", "max_urls": 1000 } } ``` ```json theme={null} { "discovery": { "method": "pagination", "url_template": "https://shop.com/products?page={n}", "start_index": 1, "step": 1, "max_pages": 100 } } ``` ```json theme={null} { "discovery": { "method": "link_pattern", "seed_url": "https://news.com", "link_pattern": "/article/*/", "navigation_pattern": "/category/", "max_depth": 2, "max_urls": 500 } } ``` ### Discovery parameters #### Sitemap | Parameter | Type | Required | Description | | ------------- | ------- | -------- | ---------------------------------------------------- | | `method` | string | Yes | Must be `"sitemap"` | | `sitemap_url` | string | Yes | URL to sitemap.xml file | | `url_pattern` | string | No | Glob pattern to filter URLs | | `max_urls` | integer | No | Maximum URLs to discover (default: 1000, max: 10000) | #### Pagination | Parameter | Type | Required | Description | | -------------- | ------- | -------- | --------------------------------------------------- | | `method` | string | Yes | Must be `"pagination"` | | `url_template` | string | Yes | URL with `{n}` placeholder | | `url_pattern` | string | No | Glob pattern to filter URLs | | `start_index` | integer | No | First page number (default: 1) | | `step` | integer | No | Increment between pages (default: 1) | | `max_pages` | integer | No | Maximum pages to generate (default: 100, max: 1000) | #### Link Pattern | Parameter | Type | Required | Description | | -------------------- | ------- | -------- | ---------------------------------------------------- | | `method` | string | Yes | Must be `"link_pattern"` | | `seed_url` | string | Yes | Starting URL for crawl | | `link_pattern` | string | Yes | Glob pattern for URLs to collect | | `navigation_pattern` | string | No | Pattern for pages to visit during crawl | | `max_depth` | integer | No | How deep to crawl (default: 2, max: 10) | | `max_urls` | integer | No | Maximum URLs to discover (default: 1000, max: 10000) | ### Response ```json theme={null} { "discovery_id": "550e8400-e29b-41d4-a716-446655440000", "status": "pending", "discovery_method": "sitemap", "root_url": "https://shop.com/sitemap.xml" } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/discover \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "discovery": { "method": "sitemap", "sitemap_url": "https://shop.com/sitemap.xml", "max_urls": 500 } }' ``` ## Get discovery status Get discovery status and results. ```http theme={null} GET /discover/{discovery_id} ``` ### Response **When pending/running:** ```json theme={null} { "discovery_id": "550e8400-e29b-41d4-a716-446655440000", "status": "pending", "discovery_method": "sitemap", "root_url": "https://shop.com/sitemap.xml" } ``` **When completed:** ```json theme={null} { "discovery_id": "550e8400-e29b-41d4-a716-446655440000", "status": "completed", "discovery_method": "sitemap", "root_url": "https://shop.com/sitemap.xml", "total_urls": 847, "filtered_count": 847, "inferred_pattern": "/products/[slug]/", "url_patterns": { "/products/": 847 }, "sample_urls": [ "https://shop.com/products/widget-a", "https://shop.com/products/widget-b" ], "errors": [] } ``` **When failed:** ```json theme={null} { "discovery_id": "550e8400-e29b-41d4-a716-446655440000", "status": "failed", "discovery_method": "sitemap", "root_url": "https://shop.com/sitemap.xml", "errors": ["Sitemap not accessible: 404 Not Found"] } ``` **Status values:** `pending`, `running`, `completed`, `failed` ### Example ```bash theme={null} curl https://api.meter.sh/discover/550e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." ``` ## Get discovery URLs Fetch all discovered URLs with pagination. ```http theme={null} GET /discover/{discovery_id}/urls?limit=1000&offset=0 ``` ### Query parameters | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------------------------- | | `limit` | integer | No | Max URLs to return (default: 1000, max: 10000) | | `offset` | integer | No | Results to skip (default: 0) | ### Response ```json theme={null} { "discovery_id": "550e8400-e29b-41d4-a716-446655440000", "urls": [ "https://shop.com/products/widget-a", "https://shop.com/products/widget-b", "..." ], "total": 847, "limit": 1000, "offset": 0 } ``` ### Example ```bash theme={null} # Get first 100 URLs curl "https://api.meter.sh/discover/550e8400.../urls?limit=100" \ -H "Authorization: Bearer sk_live_..." # Get next 100 URLs curl "https://api.meter.sh/discover/550e8400.../urls?limit=100&offset=100" \ -H "Authorization: Bearer sk_live_..." ``` ## Execute discovery Execute a one-time batch scrape from discovered URLs. ```http theme={null} POST /discover/{discovery_id}/execute ``` ### Request body ```json theme={null} { "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "max_urls": 100, "url_filter": ".*widget.*" } ``` ### Parameters | Parameter | Type | Required | Description | | ------------- | ------- | -------- | --------------------------------- | | `strategy_id` | string | Yes | Strategy UUID to use for scraping | | `max_urls` | integer | No | Maximum URLs to process | | `url_filter` | string | No | Regex pattern to filter URLs | ### Response ```json theme={null} { "batch_id": "770e8400-e29b-41d4-a716-446655440000", "jobs_queued": 100 } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/discover/550e8400.../execute \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "max_urls": 50 }' ``` ## Create discovery schedule Create a recurring schedule from discovered URLs. ```http theme={null} POST /discover/{discovery_id}/schedule ``` ### Request body ```json theme={null} { "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "interval_seconds": 86400, "webhook_url": "https://your-app.com/webhooks/meter", "max_urls": 500 } ``` ```json theme={null} { "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "cron_expression": "0 9 * * *", "webhook_url": "https://your-app.com/webhooks/meter", "max_urls": 500 } ``` ### Parameters | Parameter | Type | Required | Description | | ------------------ | ------- | -------- | -------------------------------- | | `strategy_id` | string | Yes | Strategy UUID to use | | `interval_seconds` | integer | No\* | Seconds between runs | | `cron_expression` | string | No\* | Cron schedule expression | | `webhook_url` | string | No | URL for completion notifications | | `max_urls` | integer | No | Maximum URLs per run | | `url_filter` | string | No | Regex pattern to filter URLs | \*Either `interval_seconds` or `cron_expression` is required. ### Response ```json theme={null} { "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "urls": ["https://shop.com/products/widget-a", "..."], "schedule_type": "interval", "interval_seconds": 86400, "enabled": true, "webhook_url": "https://your-app.com/webhooks/meter", "next_run_at": "2025-01-16T10:30:00Z", "created_at": "2025-01-15T10:30:00Z" } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/discover/550e8400.../schedule \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "interval_seconds": 86400, "webhook_url": "https://your-app.com/webhooks/meter" }' ``` ## List discoveries List all discoveries for the authenticated user. ```http theme={null} GET /discoveries?status={status}&limit=20&offset=0 ``` ### Query parameters | Parameter | Type | Required | Description | | --------- | ------- | -------- | ----------------------------------- | | `status` | string | No | Filter by status | | `limit` | integer | No | Max results (default: 20, max: 100) | | `offset` | integer | No | Results to skip (default: 0) | ### Response Array of discovery objects (same format as Get discovery). ### Example ```bash theme={null} # All discoveries curl https://api.meter.sh/discoveries \ -H "Authorization: Bearer sk_live_..." # Only completed curl "https://api.meter.sh/discoveries?status=completed" \ -H "Authorization: Bearer sk_live_..." ``` ## Delete discovery Delete a discovery and its associated URLs. ```http theme={null} DELETE /discover/{discovery_id} ``` ### Response ```json theme={null} { "message": "Discovery deleted", "discovery_id": "550e8400-e29b-41d4-a716-446655440000" } ``` ### Example ```bash theme={null} curl -X DELETE https://api.meter.sh/discover/550e8400... \ -H "Authorization: Bearer sk_live_..." ``` ## Polling for completion Since discovery runs asynchronously, poll until status is `completed` or `failed`: ```javascript theme={null} async function waitForDiscovery(discoveryId) { while (true) { const response = await fetch( `https://api.meter.sh/discover/${discoveryId}`, { headers: { Authorization: `Bearer ${process.env.METER_API_KEY}`, }, } ); const discovery = await response.json(); if (discovery.status === "completed") { return discovery; } else if (discovery.status === "failed") { throw new Error(discovery.errors.join(", ")); } // Wait 2 seconds before next check await new Promise((resolve) => setTimeout(resolve, 2000)); } } ``` ## Error responses | Status | Description | | ------ | ----------------------------------------------------- | | `400` | Invalid request (see discovery-specific errors below) | | `401` | Invalid or missing API key | | `404` | Discovery or strategy not found | | `500` | Internal server error | | `503` | Service temporarily unavailable | ### Discovery-specific errors | Status | Error | Description | | ------ | ------------------------- | ---------------------------------- | | `400` | Invalid url\_filter regex | The regex pattern is invalid | | `400` | Discovery not ready | Tried to execute before completion | | `400` | No URLs match the filter | Filter excluded all URLs | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Step-by-step crawling guide Understand how site crawling works Manage recurring scrapes Track batch job results ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # REST API Errors Source: https://docs.meter.sh/api-reference/rest/errors HTTP error codes and responses in the Meter API # REST API Errors The Meter API uses standard HTTP status codes and returns JSON error messages. ## HTTP status codes | Code | Meaning | Description | | ----- | --------------------- | ---------------------------------------- | | `200` | OK | Request succeeded | | `201` | Created | Resource created successfully | | `400` | Bad Request | Invalid request parameters | | `401` | Unauthorized | Invalid or missing API key | | `403` | Forbidden | Valid key but insufficient permissions | | `404` | Not Found | Resource doesn't exist | | `422` | Unprocessable Entity | Request valid but semantically incorrect | | `429` | Too Many Requests | Rate limit exceeded | | `500` | Internal Server Error | Server-side error | | `503` | Service Unavailable | Temporary service issue | ## Error response format All errors return JSON with a `detail` field: ```json theme={null} { "detail": "Error message describing what went wrong" } ``` ## Common errors ### 401 Unauthorized **Cause:** Invalid or missing API key ```json theme={null} { "detail": "Invalid or missing API key" } ``` **Solutions:** * Verify `Authorization` header is included * Check API key is correct * Ensure key hasn't been deleted ### 400 Bad Request **Cause:** Invalid request parameters ```json theme={null} { "detail": "Invalid URL format" } ``` Common causes: * Missing required fields * Invalid field types * Malformed JSON * Invalid UUIDs **Solutions:** * Check request body matches expected format * Verify all required fields are present * Ensure JSON is valid ### 404 Not Found **Cause:** Resource doesn't exist ```json theme={null} { "detail": "Strategy not found" } ``` **Solutions:** * Verify the UUID is correct * Check the resource hasn't been deleted * Ensure you have permission to access it ### 422 Unprocessable Entity **Cause:** Request is valid but contains semantic errors ```json theme={null} { "detail": "Invalid URL format" } ``` **Solutions:** * Check field values meet semantic requirements (e.g., valid URL format) * Verify data relationships are valid ### 500 Internal Server Error **Cause:** Server-side error ```json theme={null} { "detail": "Internal server error" } ``` **Solutions:** * Retry the request after a delay * If persistent, contact support * Check API status page ## Handling errors ### JavaScript ```javascript theme={null} const response = await fetch('https://api.meter.sh/api/strategies', { headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' } }); if (!response.ok) { const error = await response.json(); console.error(`API error (${response.status}): ${error.detail}`); if (response.status === 401) { // Handle authentication error } else if (response.status === 404) { // Handle not found } else if (response.status === 429) { // Handle rate limit - check Retry-After header const retryAfter = response.headers.get('Retry-After') || '60'; console.log(`Rate limited. Retry after ${retryAfter} seconds.`); } } const data = await response.json(); ``` ### Python ```python theme={null} import requests import time response = requests.get( 'https://api.meter.sh/api/strategies', headers={'Authorization': f'Bearer {api_key}'} ) if not response.ok: error = response.json() print(f"API error ({response.status_code}): {error['detail']}") if response.status_code == 401: # Handle authentication error pass elif response.status_code == 404: # Handle not found pass elif response.status_code == 429: # Handle rate limit - check Retry-After header retry_after = int(response.headers.get('Retry-After', 60)) print(f"Rate limited. Retry after {retry_after} seconds.") time.sleep(retry_after) data = response.json() ``` ### curl ```bash theme={null} response=$(curl -s -w "\n%{http_code}" https://api.meter.sh/api/strategies \ -H "Authorization: Bearer sk_live_...") http_code=$(echo "$response" | tail -n 1) body=$(echo "$response" | sed '$d') if [ "$http_code" != "200" ]; then echo "Error ($http_code): $body" fi ``` ## Best practices For 429, 500, and 503 errors, retry with appropriate backoff: ```javascript theme={null} async function fetchWithRetry(url, options, maxRetries = 3) { for (let i = 0; i < maxRetries; i++) { try { const response = await fetch(url, options); // Rate limited - respect Retry-After header if (response.status === 429) { const retryAfter = response.headers.get('Retry-After') || '60'; await new Promise(r => setTimeout(r, parseInt(retryAfter) * 1000)); continue; } // Server errors - exponential backoff if ((response.status === 500 || response.status === 503) && i < maxRetries - 1) { await new Promise(r => setTimeout(r, 2 ** i * 1000)); continue; } return response; } catch (error) { if (i === maxRetries - 1) throw error; await new Promise(r => setTimeout(r, 2 ** i * 1000)); } } } ``` Always check HTTP status codes before parsing responses: ```javascript theme={null} const response = await fetch(url, options); // Check before parsing JSON if (!response.ok) { const error = await response.json(); throw new Error(error.detail); } const data = await response.json(); ``` Include request details when logging errors: ```javascript theme={null} if (!response.ok) { const error = await response.json(); console.error('API request failed', { status: response.status, detail: error.detail, url: response.url, method: options.method }); } ``` ## Rate limiting Strategy generation endpoints (`/api/strategies/generate`, `/api/watch`) are rate limited to prevent overloading the underlying LLM service. ### 429 Too Many Requests When you hit rate limits, the API returns: ``` HTTP/1.1 429 Too Many Requests Retry-After: 60 Content-Type: application/json { "detail": "Please slow down - limits on strategy generation" } ``` The `Retry-After` header tells you how many seconds to wait before retrying. ### Handling rate limits **JavaScript:** ```javascript theme={null} const response = await fetch(url, options); if (response.status === 429) { const retryAfter = response.headers.get('Retry-After') || '60'; const waitSeconds = parseInt(retryAfter, 10); console.log(`Rate limited. Waiting ${waitSeconds} seconds...`); await new Promise(r => setTimeout(r, waitSeconds * 1000)); // Retry the request return fetch(url, options); } ``` **Python:** ```python theme={null} response = requests.post(url, headers=headers, json=data) if response.status_code == 429: retry_after = int(response.headers.get('Retry-After', 60)) print(f"Rate limited. Waiting {retry_after} seconds...") time.sleep(retry_after) # Retry the request response = requests.post(url, headers=headers, json=data) ``` ### 503 Service Unavailable If the LLM service is temporarily unavailable: ```json theme={null} { "detail": "Service temporarily unavailable. Please try again later." } ``` ## Next steps Error handling in the Python SDK Learn about the REST API Understand authentication errors ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Job Endpoints Source: https://docs.meter.sh/api-reference/rest/jobs REST API endpoints for executing and managing scrape jobs # Job Endpoints Execute scrapes and retrieve results via HTTP. ## Create job Create a new scrape job using a strategy. ```http theme={null} POST /api/jobs ``` ### Request body ```json theme={null} { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products" } ``` | Field | Type | Required | Description | | ------------- | ------ | ----------- | ---------------------------------------------------------- | | `strategy_id` | string | Yes | Strategy UUID | | `url` | string | Conditional | Single URL to scrape (use `url` OR `urls`, not both) | | `urls` | array | Conditional | List of URLs to scrape as a batch | | `parameters` | object | No | Override API parameters for this job (API strategies only) | ### Response (single URL) ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "status": "pending", "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "created_at": "2025-01-15T10:30:00Z" } ``` ### Response (batch URLs) When using `urls`, the response includes a `batch_id` for tracking: ```json theme={null} { "batch_id": "770e8400-e29b-41d4-a716-446655440000", "job_count": 3, "status": "pending", "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "created_at": "2025-01-15T10:30:00Z" } ``` ### Example ```bash theme={null} # Single URL curl -X POST https://api.meter.sh/api/jobs \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products" }' # Batch URLs curl -X POST https://api.meter.sh/api/jobs \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "urls": [ "https://example.com/products/1", "https://example.com/products/2" ] }' # With API parameters curl -X POST https://api.meter.sh/api/jobs \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/api/products", "parameters": {"page": 2, "limit": 50} }' ``` ## Execute job (synchronous) Create a job and wait for completion. Returns results directly without polling. ```http theme={null} POST /api/jobs/execute ``` ### Request body ```json theme={null} { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "parameters": {"page": 1, "limit": 50} } ``` | Field | Type | Required | Description | | ------------- | ------ | -------- | --------------------------------------------- | | `strategy_id` | string | Yes | Strategy UUID | | `url` | string | Yes | URL to scrape | | `parameters` | object | No | Override API parameters (API strategies only) | ### Response ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "status": "completed", "results": [ {"name": "Product A", "price": "$19.99"}, {"name": "Product B", "price": "$29.99"} ], "item_count": 2, "error": null, "started_at": "2025-01-15T10:30:01Z", "completed_at": "2025-01-15T10:30:08Z", "created_at": "2025-01-15T10:30:00Z" } ``` This endpoint blocks until the job completes (up to 1 hour timeout). Use the async `POST /api/jobs` endpoint for long-running scrapes or when you don't need immediate results. ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/jobs/execute \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products" }' ``` ### Error handling If the job fails, the response will include the error: ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "status": "failed", "results": null, "error": "Connection timeout: target site did not respond", "completed_at": "2025-01-15T10:30:15Z" } ``` ## Get job Get job status and results. ```http theme={null} GET /api/jobs/{job_id} ``` ### Response ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "status": "completed", "results": [ {"name": "Product A", "price": "$19.99"}, {"name": "Product B", "price": "$29.99"} ], "item_count": 2, "content_hash": "7f3d9a2b4c1e...", "structural_signature": {...}, "started_at": "2025-01-15T10:30:05Z", "completed_at": "2025-01-15T10:30:12Z", "created_at": "2025-01-15T10:30:00Z" } ``` **Status values:** `pending`, `running`, `completed`, `failed` ### Example ```bash theme={null} curl https://api.meter.sh/api/jobs/660e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." ``` ## List jobs List jobs with optional filtering. ```http theme={null} GET /api/jobs?strategy_id={id}&status={status}&limit=20&offset=0 ``` ### Query parameters | Parameter | Type | Required | Description | | ------------- | ------- | -------- | ---------------------------- | | `strategy_id` | string | No | Filter by strategy UUID | | `status` | string | No | Filter by status | | `limit` | integer | No | Max results (default: 20) | | `offset` | integer | No | Results to skip (default: 0) | ### Response Array of job objects (same format as Get job). ### Example ```bash theme={null} # All jobs curl https://api.meter.sh/api/jobs \ -H "Authorization: Bearer sk_live_..." # Filter by strategy curl https://api.meter.sh/api/jobs?strategy_id=550e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." # Only failed jobs curl https://api.meter.sh/api/jobs?status=failed \ -H "Authorization: Bearer sk_live_..." ``` ## Compare jobs Compare two jobs to detect changes. ```http theme={null} POST /api/jobs/compare ``` ### Request body ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "other_job_id": "770e8400-e29b-41d4-a716-446655440000" } ``` ### Response ```json theme={null} { "content_hash_match": false, "structural_match": true, "semantic_similarity": 0.95, "changes": [ "Item count changed: 10 -> 12", "Field 'price' changed in 3 items" ] } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/jobs/compare \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "job_id": "660e8400-e29b-41d4-a716-446655440000", "other_job_id": "770e8400-e29b-41d4-a716-446655440000" }' ``` ## Compare manifest Compare a manifest of known items against a job's scrape results using fuzzy matching. Identifies items that were added, removed, or still present. ```http theme={null} POST /api/jobs/{job_id}/compare-manifest ``` ### Request body ```json theme={null} { "manifest": [ {"name": "Acme Corp"}, {"name": "Beta Industries"}, {"name": "Gamma Solutions"} ], "match_fields": ["name"], "threshold": 80 } ``` | Field | Type | Required | Description | | -------------- | ------ | -------- | ------------------------------------------------------------------- | | `manifest` | array | Yes | List of known items (objects with at least the `match_fields` keys) | | `match_fields` | array | Yes | Field name(s) to fuzzy-match on (e.g., `["name"]`) | | `threshold` | number | No | Minimum match score 0-100 (default: 80) | ### Response ```json theme={null} { "matched": [ { "manifest_item": {"name": "Acme Corp"}, "scraped_item": {"name": "Acme Corporation", "website": "acme.com"}, "score": 90.0, "matched_on": "name" }, { "manifest_item": {"name": "Gamma Solutions"}, "scraped_item": {"name": "Gamma Solutions Inc", "website": "gamma.com"}, "score": 95.0, "matched_on": "name" } ], "added": [ {"name": "Delta Partners", "website": "delta.com"} ], "removed": [ {"name": "Beta Industries"} ], "summary": { "matched": 2, "added": 1, "removed": 1, "manifest_count": 3, "scraped_count": 3 }, "threshold_used": 80.0, "match_fields_used": ["name"], "job_id": "660e8400-e29b-41d4-a716-446655440000" } ``` | Field | Description | | --------- | ----------------------------------------------------------------------- | | `matched` | Items found in both manifest and scrape results, with confidence scores | | `added` | Items found in scrape results but **not** in the manifest | | `removed` | Items in the manifest but **not** found in scrape results | | `summary` | Count summary of matched, added, removed, and totals | ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/jobs/660e8400-e29b-41d4-a716-446655440000/compare-manifest \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "manifest": [ {"name": "Acme Corp"}, {"name": "Beta Industries"} ], "match_fields": ["name"], "threshold": 80 }' ``` Use `POST /api/strategies/{strategy_id}/compare-manifest` instead if you want to automatically compare against the latest results without specifying a job ID. See [Strategy Endpoints](/api-reference/rest/strategies#compare-manifest). ## Get strategy history Get timeline of all jobs for a strategy. ```http theme={null} GET /api/strategies/{strategy_id}/history ``` ### Response ```json theme={null} [ { "job_id": "660e8400-e29b-41d4-a716-446655440000", "status": "completed", "item_count": 12, "has_changes": true, "created_at": "2025-01-15T10:30:00Z" }, { "job_id": "770e8400-e29b-41d4-a716-446655440000", "status": "completed", "item_count": 10, "has_changes": false, "created_at": "2025-01-15T09:30:00Z" } ] ``` ### Example ```bash theme={null} curl https://api.meter.sh/api/strategies/550e8400-e29b-41d4-a716-446655440000/history \ -H "Authorization: Bearer sk_live_..." ``` ## Polling for completion Use `POST /api/jobs/execute` instead if you want synchronous behavior without polling. Since jobs created with `POST /api/jobs` run asynchronously, poll the Get job endpoint until status is `completed` or `failed`: ```javascript theme={null} async function waitForJob(jobId) { while (true) { const response = await fetch(`https://api.meter.sh/api/jobs/${jobId}`, { headers: { 'Authorization': `Bearer ${process.env.METER_API_KEY}` } }); const job = await response.json(); if (job.status === 'completed') { return job.results; } else if (job.status === 'failed') { throw new Error(job.error); } // Wait 2 seconds before next check await new Promise(resolve => setTimeout(resolve, 2000)); } } ``` ## Error responses | Status | Description | | ------ | --------------------------------------------- | | `400` | Invalid request (missing strategy\_id or url) | | `401` | Invalid or missing API key | | `404` | Job or strategy not found | | `500` | Internal server error | | `503` | Service temporarily unavailable | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Automate job execution Use the Python SDK with built-in polling Learn about job lifecycle ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # REST API Overview Source: https://docs.meter.sh/api-reference/rest/overview Introduction to the Meter REST API # REST API Overview The Meter REST API provides direct HTTP access to all Meter features. Use it when the Python SDK isn't available or when you need language-agnostic integration. ## Base URL ``` https://api.meter.sh ``` All API requests should be made to this base URL. ## Authentication Include your API key in the `Authorization` header using the Bearer scheme: ```bash theme={null} curl https://api.meter.sh/api/strategies \ -H "Authorization: Bearer sk_live_your_key_here" \ -H "Content-Type: application/json" ``` See [Authentication](/api-reference/rest/authentication) for details. ## Request format All POST and PATCH requests must include `Content-Type: application/json` and send JSON-encoded request bodies. **Example:** ```bash theme={null} curl -X POST https://api.meter.sh/api/strategies/generate \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "description": "Extract product names and prices", "name": "Product Scraper" }' ``` ## Response format All responses are JSON-encoded with appropriate HTTP status codes: * `200 OK`: Successful request * `201 Created`: Resource created successfully * `400 Bad Request`: Invalid request parameters * `401 Unauthorized`: Invalid or missing API key * `404 Not Found`: Resource not found * `500 Internal Server Error`: Server error **Success response:** ```json theme={null} { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "strategy": {...}, "preview_data": [...] } ``` **Error response:** ```json theme={null} { "detail": "Error message describing what went wrong" } ``` ## API endpoints ### Strategies | Method | Endpoint | Description | | ------ | ----------------------------- | ------------------------------------------ | | POST | `/api/strategies/generate` | Generate a new strategy | | POST | `/api/strategies/{id}/refine` | Refine an existing strategy | | GET | `/api/strategies` | List all strategies | | GET | `/api/strategies/{id}` | Get strategy details | | GET | `/api/strategies/audit` | Audit log of strategy create/delete events | | DELETE | `/api/strategies/{id}` | Delete a strategy | ### Account | Method | Endpoint | Description | | ------ | -------------------- | ---------------------------------------- | | GET | `/api/account/quota` | Current strategy quota state (read-only) | ### Jobs | Method | Endpoint | Description | | ------ | ------------------------------ | -------------------------- | | POST | `/api/jobs` | Create a new job (async) | | POST | `/api/jobs/execute` | Execute job (synchronous) | | GET | `/api/jobs/{id}` | Get job status and results | | GET | `/api/jobs` | List jobs (with filtering) | | POST | `/api/jobs/compare` | Compare two jobs | | GET | `/api/strategies/{id}/history` | Get strategy job history | ### Schedules | Method | Endpoint | Description | | ------ | ----------------------------- | ------------------ | | POST | `/api/schedules` | Create a schedule | | GET | `/api/schedules` | List all schedules | | PATCH | `/api/schedules/{id}` | Update a schedule | | DELETE | `/api/schedules/{id}` | Delete a schedule | | GET | `/api/schedules/{id}/changes` | Get unseen changes | ### Watch (Simplified) | Method | Endpoint | Description | | ------ | ------------ | ------------------------------------------------ | | POST | `/api/watch` | Create a watch (strategy + schedule in one call) | ### Discovery | Method | Endpoint | Description | | ------ | ------------------------- | ------------------------- | | POST | `/discover` | Start URL discovery | | GET | `/discover/{id}` | Get discovery status | | GET | `/discover/{id}/urls` | Get discovered URLs | | POST | `/discover/{id}/execute` | Execute batch scrape | | POST | `/discover/{id}/schedule` | Create recurring schedule | | GET | `/discoveries` | List all discoveries | | DELETE | `/discover/{id}` | Delete a discovery | ### Strategy Groups | Method | Endpoint | Description | | ------ | -------------------------------------------- | ----------------------------------- | | POST | `/api/strategy-groups` | Create a strategy group | | GET | `/api/strategy-groups` | List strategy groups | | GET | `/api/strategy-groups/{id}` | Get group details with strategies | | PATCH | `/api/strategy-groups/{id}` | Update group name/description | | DELETE | `/api/strategy-groups/{id}` | Delete a group | | POST | `/api/strategy-groups/{id}/strategies` | Add strategies to group | | DELETE | `/api/strategy-groups/{id}/strategies/{sid}` | Remove strategy from group | | POST | `/api/strategy-groups/{id}/schedule` | Apply schedule to all group members | | DELETE | `/api/strategy-groups/{id}/schedule` | Delete all group schedules | | PATCH | `/api/strategy-groups/{id}/schedule/toggle` | Enable/disable group schedules | | PATCH | `/api/strategy-groups/{id}/schema` | Apply output schema to group | | GET | `/api/strategy-groups/{id}/schema/progress` | Poll schema regeneration progress | | POST | `/api/strategy-groups/{id}/webhook/test` | Test group webhook | ### Workflows | Method | Endpoint | Description | | ------ | ---------------------------------------- | ------------------------ | | POST | `/api/workflows` | Create a workflow | | GET | `/api/workflows` | List workflows | | GET | `/api/workflows/{id}` | Get workflow details | | PUT | `/api/workflows/{id}` | Update workflow metadata | | DELETE | `/api/workflows/{id}` | Delete a workflow | | POST | `/api/workflows/{id}/nodes` | Add a node | | PUT | `/api/workflows/{id}/nodes/{nid}` | Update a node | | DELETE | `/api/workflows/{id}/nodes/{nid}` | Delete a node | | POST | `/api/workflows/{id}/edges` | Add an edge | | DELETE | `/api/workflows/{id}/edges/{eid}` | Delete an edge | | POST | `/api/workflows/{id}/run` | Run a workflow | | GET | `/api/workflows/{id}/runs/{rid}` | Get run details | | GET | `/api/workflows/{id}/runs` | List runs | | GET | `/api/workflows/{id}/runs/latest/output` | Get latest output | | POST | `/api/workflows/{id}/runs/{rid}/cancel` | Cancel a run | | POST | `/api/workflows/{id}/schedules` | Create workflow schedule | | GET | `/api/workflows/{id}/schedules` | List workflow schedules | | PATCH | `/api/workflows/{id}/schedules/{sid}` | Update workflow schedule | | DELETE | `/api/workflows/{id}/schedules/{sid}` | Delete workflow schedule | ### Webhooks | Method | Endpoint | Description | | ------ | -------------------- | --------------------- | | POST | `/api/webhooks/test` | Test webhook delivery | See detailed endpoint documentation: * [Strategy endpoints](/api-reference/rest/strategies) * [Job endpoints](/api-reference/rest/jobs) * [Schedule endpoints](/api-reference/rest/schedules) * [Strategy group endpoints](/api-reference/rest/strategy-groups) * [Workflow endpoints](/api-reference/rest/workflows) * [Watch endpoint](/api-reference/rest/watch) * [Discovery endpoints](/api-reference/rest/discovery) * [Webhooks](/api-reference/rest/webhooks) ## Rate limiting Rate limit headers are included in responses: ``` X-RateLimit-Limit: 1000 X-RateLimit-Remaining: 999 X-RateLimit-Reset: 1640995200 ``` ## Pagination List endpoints support pagination with `limit` and `offset` query parameters: ```bash theme={null} # Get first 20 results curl "https://api.meter.sh/api/strategies?limit=20&offset=0" \ -H "Authorization: Bearer sk_live_..." # Get next 20 results curl "https://api.meter.sh/api/strategies?limit=20&offset=20" \ -H "Authorization: Bearer sk_live_..." ``` ## Idempotency Strategy generation supports an optional `Idempotency-Key` request header. Retries with the same key within 24 hours replay the original response without re-running the LLM or consuming additional quota. See [Strategy generation idempotency](/api-reference/rest/strategies#idempotency) for the full semantics and examples. Other POST endpoints do not currently support `Idempotency-Key` — calling them twice will create two resources. ## CORS The API does not currently support CORS. For browser-based applications, proxy requests through your backend. ## Webhooks Configure webhooks when creating schedules to receive real-time notifications: ```bash theme={null} curl -X POST https://api.meter.sh/api/schedules \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "interval_seconds": 3600, "webhook_url": "https://your-app.com/webhooks/meter" }' ``` See the [Webhooks Guide](/guides/webhooks) for implementation details. ## Next steps Learn about API key authentication Explore strategy API endpoints Read live quota and plan tier Explore job API endpoints Explore schedule API endpoints One-step URL monitoring setup Site crawling and URL discovery Manage strategy groups Test webhook delivery ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Schedule Endpoints Source: https://docs.meter.sh/api-reference/rest/schedules REST API endpoints for managing recurring scrape schedules # Schedule Endpoints Create and manage automated, recurring scrapes via HTTP. ## Create schedule Create a new recurring schedule. ```http theme={null} POST /api/schedules ``` ### Request body (interval-based) ```json theme={null} { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "interval_seconds": 3600, "webhook_url": "https://your-app.com/webhooks/meter" } ``` ### Request body (cron-based) ```json theme={null} { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "cron_expression": "0 9 * * *", "webhook_url": "https://your-app.com/webhooks/meter" } ``` | Field | Type | Required | Description | | ------------------ | ------- | ----------- | ---------------------------------------------------------------------------------------------- | | `strategy_id` | string | Yes | Strategy UUID | | `url` | string | Conditional | Single URL to scrape (use `url` OR `urls`, not both) | | `urls` | array | Conditional | List of URLs to scrape (use `url` OR `urls`, not both) | | `interval_seconds` | integer | Conditional | Interval in seconds (minimum: 60) | | `cron_expression` | string | Conditional | Cron expression | | `webhook_url` | string | No | Webhook URL for notifications | | `webhook_metadata` | object | No | Custom JSON metadata included in every webhook payload | | `webhook_secret` | string | No | Secret for `X-Webhook-Secret` header. Auto-generated if not provided when `webhook_url` is set | | `webhook_type` | string | No | `standard` or `slack`. Auto-detected from URL if not specified | | `parameters` | object | No | Default API parameter overrides for all scheduled runs (API strategies only) | Provide either `interval_seconds` or `cron_expression`, not both. Provide either `url` or `urls`, not both. ### Response ```json theme={null} { "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "urls": null, "schedule_type": "interval", "interval_seconds": 3600, "cron_expression": null, "enabled": true, "webhook_url": "https://your-app.com/webhooks/meter", "webhook_metadata": null, "webhook_secret": "whsec_a1b2c3...", "webhook_type": "standard", "parameters": null, "next_run_at": "2025-01-15T11:30:00Z", "last_run_at": null, "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ``` The `webhook_secret` is only returned in the create response. It is not included in subsequent GET responses. ```` ### Example ```bash # Interval-based curl -X POST https://api.meter.sh/api/schedules \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "interval_seconds": 3600 }' # Cron-based curl -X POST https://api.meter.sh/api/schedules \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com/products", "cron_expression": "0 9 * * *" }' ```` ## List schedules Get all schedules for the authenticated user. ```http theme={null} GET /api/schedules ``` ### Response Array of schedule objects (same format as Create schedule response). ### Example ```bash theme={null} curl https://api.meter.sh/api/schedules \ -H "Authorization: Bearer sk_live_..." ``` ## Update schedule Update an existing schedule. ```http theme={null} PATCH /api/schedules/{schedule_id} ``` ### Request body All fields are optional. Include only fields to update: ```json theme={null} { "enabled": false, "interval_seconds": 7200, "webhook_url": "https://new-domain.com/webhooks", "webhook_metadata": {"project": "updated-project"}, "webhook_type": "standard" } ``` | Field | Type | Description | | ------------------ | ------- | ------------------------------------------------ | | `enabled` | boolean | Enable/disable the schedule | | `url` | string | Update to a single URL | | `urls` | array | Update to multiple URLs | | `interval_seconds` | integer | New interval in seconds | | `cron_expression` | string | New cron expression | | `webhook_url` | string | New webhook URL (or `null` to remove) | | `webhook_metadata` | object | Update custom JSON metadata for webhook payloads | | `webhook_secret` | string | Update webhook secret | | `webhook_type` | string | Update webhook type: `standard` or `slack` | | `parameters` | object | Update API parameter defaults | ### Response Updated schedule object. ### Example ```bash theme={null} # Disable schedule curl -X PATCH https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{"enabled": false}' # Change interval curl -X PATCH https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{"interval_seconds": 7200}' ``` ## Delete schedule Delete a schedule (stops future jobs). ```http theme={null} DELETE /api/schedules/{schedule_id} ``` ### Response ```json theme={null} { "message": "Schedule deleted successfully" } ``` ### Example ```bash theme={null} curl -X DELETE https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." ``` ## Regenerate webhook secret Generate a new webhook secret for a schedule. The old secret is immediately invalidated. ```http theme={null} POST /api/schedules/{schedule_id}/webhook-secret/regenerate ``` ### Response ```json theme={null} { "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "webhook_secret": "whsec_new_secret_here..." } ``` The new secret is returned only once. Store it securely and update your webhook handler before the next delivery. ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000/webhook-secret/regenerate \ -H "Authorization: Bearer sk_live_..." ``` ## Get schedule changes Get unseen changes for a schedule (pull-based change detection). ```http theme={null} GET /api/schedules/{schedule_id}/changes?mark_seen=true&filter=+keyword ``` ### Query parameters | Parameter | Type | Required | Description | | ----------- | ------- | -------- | -------------------------------------------- | | `mark_seen` | boolean | No | Mark changes as seen (default: true) | | `filter` | string | No | Lucene-style keyword filter for result items | ### Keyword filter syntax The `filter` parameter uses Lucene-style syntax to filter individual result items: | Syntax | Meaning | Example | | ---------- | -------------- | ---------------------------------- | | `+keyword` | Required (AND) | `+rubio +tariff` - items with both | | `keyword` | Optional (OR) | `rubio elon` - items with either | | `-keyword` | Excluded (NOT) | `-bitcoin` - items without | | `"phrase"` | Exact phrase | `"elon musk"` - exact match | The filter applies to individual items within results, not entire jobs. Jobs with zero matching items are excluded from the response. ### Response ```json theme={null} { "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "changes": [ { "job_id": "660e8400-e29b-41d4-a716-446655440000", "status": "completed", "results": [...], "item_count": 12, "content_hash": "7f3d9a2b4c1e...", "completed_at": "2025-01-15T10:30:12Z", "seen": true } ], "count": 1, "marked_seen": true, "filter_applied": "+rubio +tariff" } ``` ### Example ```bash theme={null} # Get and mark as seen curl https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000/changes \ -H "Authorization: Bearer sk_live_..." # Preview without marking curl https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000/changes?mark_seen=false \ -H "Authorization: Bearer sk_live_..." # Filter for items containing both "rubio" AND "tariff" curl "https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000/changes?filter=%2Brubio+%2Btariff" \ -H "Authorization: Bearer sk_live_..." # Filter for items containing "rubio" OR "elon" curl "https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000/changes?filter=rubio+elon" \ -H "Authorization: Bearer sk_live_..." # Filter for items with "rubio" but NOT "biden" curl "https://api.meter.sh/api/schedules/880e8400-e29b-41d4-a716-446655440000/changes?filter=%2Brubio+-biden" \ -H "Authorization: Bearer sk_live_..." ``` ## Webhook payload When a schedule has a webhook URL, Meter POSTs to it after each job: ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "status": "completed", "results": [...], "item_count": 12, "has_changes": true, "content_hash": "7f3d9a2b4c1e...", "completed_at": "2025-01-15T10:30:12Z", "delivery_reason": "first_run" } ``` `delivery_reason` is `first_run` for the first successful delivery for a given `schedule + url`, and `content_changed` for subsequent deliveries triggered by a content change. See [Webhook payload formats](/api-reference/rest/webhooks#webhook-payload-formats) for the full field reference, and the [Webhooks Guide](/guides/webhooks) for implementation details. ## Error responses | Status | Description | | ------ | ------------------------------------------------------------------ | | `400` | Invalid request (invalid cron expression, missing required fields) | | `401` | Invalid or missing API key | | `404` | Schedule or strategy not found | | `500` | Internal server error | | `503` | Service temporarily unavailable | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Implement webhook endpoints Use the changes endpoint Use the Python SDK Learn about schedules ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Strategy Endpoints Source: https://docs.meter.sh/api-reference/rest/strategies REST API endpoints for managing extraction strategies # Strategy Endpoints Create and manage AI-generated extraction strategies via HTTP. ## Generate strategy Generate a new extraction strategy using AI. ```http theme={null} POST /api/strategies/generate ``` ### Request body ```json theme={null} { "url": "https://example.com/products", "description": "Extract product names and prices", "name": "Product Scraper", "output_schema": {"title": "string", "price": "number"}, "filter_config": { "mode": "all", "conditions": [ {"field": "price", "operator": "gt", "value": "10"} ] } } ``` | Field | Type | Required | Description | | ------------------- | ------ | -------- | ----------------------------------------------------------------------------- | | `url` | string | Yes | Target webpage URL | | `description` | string | Yes | What to extract | | `name` | string | Yes | Strategy name | | `output_schema` | object | No | Desired output JSON structure. See [Output Schemas](/concepts/output-schemas) | | `filter_config` | object | No | Post-extraction filter. See [Filtering](/concepts/filtering) | | `strategy_group_id` | UUID | No | Add the strategy to this group on creation | ### Response ```json theme={null} { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "strategy": { "container": {...}, "fields": {...} }, "preview_data": [ {"name": "Product A", "price": "$19.99"}, {"name": "Product B", "price": "$29.99"} ], "attempts": 1 } ``` ### Response headers This endpoint returns [strategy quota headers](#strategy-quota-headers) on every response (success **and** the `403` thrown when the quota is exceeded), so clients can render quota state without an extra round-trip. ### Idempotency Pass an `Idempotency-Key: ` header to make retries safe — the server caches the response for **24 hours** and replays it on subsequent requests with the same key. Cached replays do **not** run a new LLM job and do **not** consume additional quota. | Scenario | Result | | ----------------------------------------- | -------------------------------------------------------------------------------- | | First request with a given key | Runs normally; response cached for 24h | | Retry with same key + same body | Returns cached response with `Idempotent-Replayed: true` header | | Retry with same key + **different** body | `422 Idempotency-Key reused with a different request body.` | | Retry while the original is still running | `409 Idempotent request still in progress. Retry shortly.` plus `Retry-After: 5` | | No header sent | No change in behavior — existing clients are unaffected | Notes: * Keys are **scoped per user / API key**, so two accounts cannot collide on the same string. * Use a client-generated unique value (e.g. `uuidgen`, or a stable hash of `URL + description + run-id`). Two semantically-equivalent retries must reuse the same key. * Pair this with your own retry-on-5xx logic to avoid duplicate strategies (and quota burn) on transient timeouts. * The 24h TTL is fixed and not configurable. ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/strategies/generate \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://news.ycombinator.com", "description": "Extract post titles and scores", "name": "HN Front Page" }' ``` Idempotent retry — first call runs, second call replays the cached response: ```bash theme={null} KEY=$(uuidgen) # First request — runs normally, returns the new strategy. curl -X POST https://api.meter.sh/api/strategies/generate \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $KEY" \ -d '{ "url": "https://news.ycombinator.com", "description": "Extract post titles and scores", "name": "HN Front Page" }' # Retry with the same key + same body — replays the cached response. # Response includes the header `Idempotent-Replayed: true`. No quota is consumed. curl -X POST https://api.meter.sh/api/strategies/generate \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $KEY" \ -d '{ "url": "https://news.ycombinator.com", "description": "Extract post titles and scores", "name": "HN Front Page" }' ``` The same `Idempotency-Key` is also honored on `POST /api/strategies/generate/stream`. ## Refine strategy Improve an existing strategy with feedback. ```http theme={null} POST /api/strategies/{strategy_id}/refine ``` ### Request body ```json theme={null} { "feedback": "Also extract product images and SKU" } ``` ### Response Same as generate strategy response with updated preview data. ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/strategies/550e8400-e29b-41d4-a716-446655440000/refine \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{"feedback": "Also extract product images"}' ``` ## List strategies Get all strategies for the authenticated user. ```http theme={null} GET /api/strategies?limit=20&offset=0 ``` ### Query parameters | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------- | | `limit` | integer | No | Max results (default: 20) | | `offset` | integer | No | Results to skip (default: 0) | ### Response ```json theme={null} [ { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "name": "Product Scraper", "description": "Extract product names and prices", "url": "https://example.com/products", "preview_data": [...], "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ] ``` ### Example ```bash theme={null} curl https://api.meter.sh/api/strategies?limit=10 \ -H "Authorization: Bearer sk_live_..." ``` ## Get strategy Get details for a specific strategy. ```http theme={null} GET /api/strategies/{strategy_id} ``` ### Response Same format as list strategies items. ### Example ```bash theme={null} curl https://api.meter.sh/api/strategies/550e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." ``` ## Update strategy Update a strategy's filter configuration and/or output schema. ```http theme={null} PATCH /api/strategies/{strategy_id} ``` ### Request body ```json theme={null} { "filter_config": { "mode": "all", "conditions": [ {"field": "price", "operator": "gt", "value": "50"}, {"field": "in_stock", "operator": "equals", "value": "true"} ] }, "output_schema": {"title": "string", "price": "number"} } ``` | Field | Type | Required | Description | | --------------- | ------ | -------- | ----------------------------------------------------------------------------- | | `filter_config` | object | No | Post-extraction filter configuration. See [Filtering](/concepts/filtering) | | `output_schema` | object | No | Desired output JSON structure. See [Output Schemas](/concepts/output-schemas) | ### PATCH semantics Each field follows three-state PATCH semantics: | Body shape | Effect on stored value | | ------------------------------- | ---------------------- | | Field **omitted** from body | Left unchanged | | Field present as `null` or `{}` | Cleared | | Field present with a value | Replaced | This applies independently to both `filter_config` and `output_schema`. To clear only one, send `null` for that field and omit the other. **Behavior change.** Earlier versions of this endpoint cleared `filter_config` when it was omitted from the request body. The endpoint now treats omission as "leave unchanged" — consistent with standard PATCH semantics. If you were relying on the old behavior, send `"filter_config": null` explicitly to clear. ### Response Updated strategy details. ### Examples Replace the filter, leave the output schema unchanged: ```bash theme={null} curl -X PATCH https://api.meter.sh/api/strategies/550e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "filter_config": { "mode": "all", "conditions": [ {"field": "price", "operator": "gt", "value": "50"} ] } }' ``` Clear the output schema, leave the filter unchanged: ```bash theme={null} curl -X PATCH https://api.meter.sh/api/strategies/550e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{"output_schema": null}' ``` Update both at once: ```bash theme={null} curl -X PATCH https://api.meter.sh/api/strategies/550e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "output_schema": {"title": "string", "price": "number"}, "filter_config": null }' ``` ## Strategy quota headers Every response from a strategy-creation endpoint includes headers describing the caller's current standing against the rolling 30-day strategy quota. These ride on **successful responses and on the `403` thrown when the quota is exceeded**, so a client can render quota state from any creation call without an extra round-trip. | Header | Value | Notes | | ------------------------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | `X-Strategy-Quota-Used` | Integer | Strategies counted in the rolling 30-day window. Counts both currently-active strategies and strategies created-then-deleted in the window. | | `X-Strategy-Quota-Limit` | Integer | Plan limit. `-1` means unlimited (enterprise). | | `X-Strategy-Quota-Reset` | ISO 8601 UTC timestamp | When the next quota slot frees (oldest counted strategy + 30d). Omitted for unlimited plans and when no strategies have been counted yet. | | `X-Strategy-Quota-Tier` | `free` \| `hobby` \| `pro` \| `enterprise` | The plan tier used to compute the limit. | These headers are emitted by: * `POST /api/strategies/generate` * `POST /api/strategies/generate/stream` * `POST /api/watch` For a pure read of the same numbers without making a creation call, use [`GET /api/account/quota`](/api-reference/rest/account). ## Strategy audit log Paginated audit log of strategy create and delete events, merged into a single timeline (most recent first). ```http theme={null} GET /api/strategies/audit ``` The endpoint merges three event sources: * `created` events from currently-active strategies * `created` events from previously-deleted strategies (created within the window, then deleted) * `deleted` events All `created` events have `counted_against_quota: true` (every created strategy consumed one slot of the rolling 30-day quota, whether or not it was later deleted). `deleted` events have `counted_against_quota: false` — they are tracking-only and do not consume quota. ### Query parameters | Parameter | Type | Required | Description | | --------- | ------------------- | -------- | --------------------------------------------------------------------------- | | `from` | datetime (ISO 8601) | No | Inclusive lower bound. Default: `now - 30 days` (matches the quota window). | | `to` | datetime (ISO 8601) | No | Inclusive upper bound. Default: `now`. | | `limit` | integer | No | Page size. Default: 100. Max: 500. | | `offset` | integer | No | Default: 0. | ### Response ```json theme={null} { "events": [ { "event_type": "created", "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "name": "Product Scraper", "url": "https://example.com/products", "scraper_type": "html", "timestamp": "2026-05-10T14:32:11Z", "counted_against_quota": true }, { "event_type": "deleted", "strategy_id": "441e8400-e29b-41d4-a716-446655440000", "name": "Old Scraper", "url": "https://example.com/old", "scraper_type": "html", "timestamp": "2026-05-09T09:10:00Z", "counted_against_quota": false } ], "count": 12, "limit": 100, "offset": 0, "window_start": "2026-04-17T14:32:11Z", "window_end": "2026-05-17T14:32:11Z" } ``` | Field | Type | Description | | -------------------------------- | -------- | -------------------------------------------------------- | | `events[].event_type` | string | `created` or `deleted`. | | `events[].counted_against_quota` | boolean | `true` for every `created` event; `false` for `deleted`. | | `count` | integer | Total events in the window across all pages. | | `window_start` / `window_end` | datetime | Echo of the resolved `from` / `to` bounds. | ### Example ```bash theme={null} # Default last-30-days window, 50 events per page curl "https://api.meter.sh/api/strategies/audit?limit=50" \ -H "Authorization: Bearer sk_live_..." # Custom window curl "https://api.meter.sh/api/strategies/audit?from=2026-01-01T00:00:00Z&to=2026-02-01T00:00:00Z" \ -H "Authorization: Bearer sk_live_..." ``` ## Compare manifest Compare a manifest of known items against the latest scrape results for a strategy. This is a convenience endpoint that automatically uses the most recent completed job. ```http theme={null} POST /api/strategies/{strategy_id}/compare-manifest ``` ### Request body ```json theme={null} { "manifest": [ {"name": "Acme Corp"}, {"name": "Beta Industries"}, {"name": "Gamma Solutions"} ], "match_fields": ["name"], "threshold": 80 } ``` | Field | Type | Required | Description | | -------------- | ------ | -------- | ------------------------------------------------------------------- | | `manifest` | array | Yes | List of known items (objects with at least the `match_fields` keys) | | `match_fields` | array | Yes | Field name(s) to fuzzy-match on (e.g., `["name"]`) | | `threshold` | number | No | Minimum match score 0-100 (default: 80) | ### Response ```json theme={null} { "matched": [ { "manifest_item": {"name": "Acme Corp"}, "scraped_item": {"name": "Acme Corporation", "website": "acme.com"}, "score": 90.0, "matched_on": "name" } ], "added": [ {"name": "Delta Partners", "website": "delta.com"} ], "removed": [ {"name": "Beta Industries"} ], "summary": { "matched": 1, "added": 1, "removed": 1, "manifest_count": 2, "scraped_count": 2 }, "threshold_used": 80.0, "match_fields_used": ["name"], "job_id": "660e8400-e29b-41d4-a716-446655440000" } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/strategies/550e8400-e29b-41d4-a716-446655440000/compare-manifest \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "manifest": [ {"name": "Acme Corp"}, {"name": "Beta Industries"} ], "match_fields": ["name"], "threshold": 80 }' ``` Learn more about how fuzzy matching works and best practices for tuning the threshold in the [Manifest Comparison concept guide](/concepts/manifest-comparison). ## Delete strategy Delete a strategy and all associated jobs and schedules. ```http theme={null} DELETE /api/strategies/{strategy_id} ``` ### Response ```json theme={null} { "message": "Strategy deleted successfully" } ``` ### Example ```bash theme={null} curl -X DELETE https://api.meter.sh/api/strategies/550e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." ``` This action is irreversible and deletes all associated resources. ## Error responses | Status | Description | | ------ | ------------------------------------------------------------- | | `400` | Invalid request (missing required fields, invalid URL format) | | `401` | Invalid or missing API key | | `404` | Strategy not found | | `429` | Rate limit exceeded (strategy generation is rate-limited) | | `500` | Internal server error | | `503` | Service temporarily unavailable (AI service issues) | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Execute scrapes using strategies Use the Python SDK instead Learn about strategies ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Strategy Group Endpoints Source: https://docs.meter.sh/api-reference/rest/strategy-groups REST API endpoints for managing strategy groups # Strategy Group Endpoints Create and manage strategy groups for bulk operations on collections of strategies. ## Create strategy group ```http theme={null} POST /api/strategy-groups ``` ### Request body ```json theme={null} { "name": "E-commerce Monitors", "description": "Product price tracking across 50 stores" } ``` | Field | Type | Required | Description | | ------------- | ------ | -------- | ------------------------ | | `name` | string | Yes | Name for the group | | `description` | string | No | Description of the group | ### Response ```json theme={null} { "id": "aa0e8400-e29b-41d4-a716-446655440000", "name": "E-commerce Monitors", "description": "Product price tracking across 50 stores", "strategy_count": 0, "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/strategy-groups \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "name": "E-commerce Monitors", "description": "Product price tracking across 50 stores" }' ``` ## List strategy groups ```http theme={null} GET /api/strategy-groups?limit=50&offset=0 ``` ### Query parameters | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------- | | `limit` | integer | No | Max results (default: 50) | | `offset` | integer | No | Results to skip (default: 0) | ### Response ```json theme={null} [ { "id": "aa0e8400-e29b-41d4-a716-446655440000", "name": "E-commerce Monitors", "description": "Product price tracking across 50 stores", "strategy_count": 12, "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ] ``` ### Example ```bash theme={null} curl https://api.meter.sh/api/strategy-groups?limit=10 \ -H "Authorization: Bearer sk_live_..." ``` ## Get strategy group Get details for a specific group including member strategies. ```http theme={null} GET /api/strategy-groups/{group_id} ``` ### Response ```json theme={null} { "id": "aa0e8400-e29b-41d4-a716-446655440000", "name": "E-commerce Monitors", "description": "Product price tracking across 50 stores", "strategies": [ { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "name": "Amazon Products", "description": "Extract product listings", "url": "https://amazon.com/products", "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ], "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ``` ### Example ```bash theme={null} curl https://api.meter.sh/api/strategy-groups/{group_id} \ -H "Authorization: Bearer sk_live_..." ``` ## Update strategy group Update a group's name or description. ```http theme={null} PATCH /api/strategy-groups/{group_id} ``` ### Request body ```json theme={null} { "name": "Updated Name", "description": "Updated description" } ``` All fields are optional. Include only fields to update. ### Example ```bash theme={null} curl -X PATCH https://api.meter.sh/api/strategy-groups/{group_id} \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{"name": "New Group Name"}' ``` ## Delete strategy group Delete a group. Strategies in the group become ungrouped — they are not deleted. ```http theme={null} DELETE /api/strategy-groups/{group_id} ``` ### Response ```json theme={null} { "message": "Strategy group deleted successfully" } ``` ### Example ```bash theme={null} curl -X DELETE https://api.meter.sh/api/strategy-groups/{group_id} \ -H "Authorization: Bearer sk_live_..." ``` ## Add strategies to group Add existing strategies to a group. ```http theme={null} POST /api/strategy-groups/{group_id}/strategies ``` ### Request body ```json theme={null} { "strategy_ids": [ "550e8400-e29b-41d4-a716-446655440000", "660e8400-e29b-41d4-a716-446655440000" ] } ``` ### Response ```json theme={null} { "message": "Strategies added to group" } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/strategy-groups/{group_id}/strategies \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_ids": [ "550e8400-e29b-41d4-a716-446655440000", "660e8400-e29b-41d4-a716-446655440000" ] }' ``` ## Remove strategy from group Remove a strategy from a group (sets the strategy's group to null — the strategy is not deleted). ```http theme={null} DELETE /api/strategy-groups/{group_id}/strategies/{strategy_id} ``` ### Response ```json theme={null} { "message": "Strategy removed from group" } ``` ### Example ```bash theme={null} curl -X DELETE https://api.meter.sh/api/strategy-groups/{group_id}/strategies/{strategy_id} \ -H "Authorization: Bearer sk_live_..." ``` ## Apply group schedule Create or update a schedule for every strategy in the group. ```http theme={null} POST /api/strategy-groups/{group_id}/schedule ``` ### Request body ```json theme={null} { "interval_seconds": 3600, "webhook_url": "https://your-app.com/webhooks/meter", "webhook_type": "standard" } ``` | Field | Type | Required | Description | | ------------------ | ------- | ----------- | -------------------------------------------------------------------------------------------- | | `interval_seconds` | integer | Conditional | Run every N seconds (min: 60) | | `cron_expression` | string | Conditional | Cron expression | | `webhook_url` | string | No | Webhook URL for notifications | | `webhook_secret` | string | No | Webhook secret (auto-generated if not provided) | | `webhook_type` | string | No | `standard`, `slack`, `slack_workflow`, or `discord`. Auto-detected from URL if not specified | | `webhook_metadata` | object | No | Custom JSON metadata for webhook payloads | Provide either `interval_seconds` or `cron_expression`, not both. ### Response ```json theme={null} { "message": "Group schedule applied", "created": 8, "updated": 4 } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/strategy-groups/{group_id}/schedule \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "interval_seconds": 3600, "webhook_url": "https://your-app.com/webhooks/meter" }' ``` ## Delete group schedules Delete all schedules for strategies in the group. ```http theme={null} DELETE /api/strategy-groups/{group_id}/schedule ``` ### Response ```json theme={null} { "message": "Group schedules deleted" } ``` ### Example ```bash theme={null} curl -X DELETE https://api.meter.sh/api/strategy-groups/{group_id}/schedule \ -H "Authorization: Bearer sk_live_..." ``` ## Toggle group schedules Enable or disable all schedules for strategies in the group. ```http theme={null} PATCH /api/strategy-groups/{group_id}/schedule/toggle ``` ### Request body ```json theme={null} { "enabled": false } ``` ### Response ```json theme={null} { "message": "Group schedules updated" } ``` ### Example ```bash theme={null} curl -X PATCH https://api.meter.sh/api/strategy-groups/{group_id}/schedule/toggle \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{"enabled": false}' ``` ## Apply group schema Apply an output schema to all strategies in the group. This triggers asynchronous regeneration of each strategy. ```http theme={null} PATCH /api/strategy-groups/{group_id}/schema ``` ### Request body ```json theme={null} { "output_schema": { "title": "string", "price": "number", "in_stock": "boolean" } } ``` | Field | Type | Required | Description | | --------------- | ------ | -------- | ------------------------------------------------- | | `output_schema` | object | Yes | JSON schema defining the desired output structure | See [Output Schemas](/concepts/output-schemas) for supported types and nesting. ### Response ```json theme={null} { "message": "Schema applied and regeneration started", "strategy_count": 12 } ``` ### Example ```bash theme={null} curl -X PATCH https://api.meter.sh/api/strategy-groups/{group_id}/schema \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "output_schema": { "title": "string", "price": "number", "in_stock": "boolean" } }' ``` ## Get schema progress Poll the progress of a group schema regeneration task. ```http theme={null} GET /api/strategy-groups/{group_id}/schema/progress ``` ### Response Returns progress details from the background regeneration task. ### Example ```bash theme={null} curl https://api.meter.sh/api/strategy-groups/{group_id}/schema/progress \ -H "Authorization: Bearer sk_live_..." ``` ## Test group webhook Send a test webhook using a group's schedule webhook configuration. ```http theme={null} POST /api/strategy-groups/{group_id}/webhook/test ``` ### Response ```json theme={null} { "success": true, "status_code": 200, "message": "Webhook delivered successfully" } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/strategy-groups/{group_id}/webhook/test \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" ``` The test uses the webhook URL from any existing schedule in the group. At least one member strategy must have a schedule with a `webhook_url` configured. ## Error responses | Status | Description | | ------ | ------------------------------------------------------------------ | | `400` | Invalid request (missing required fields, invalid cron expression) | | `401` | Invalid or missing API key | | `404` | Strategy group not found | | `422` | Validation error (e.g., both interval and cron provided) | | `500` | Internal server error | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Understand strategy group architecture Define output structure for groups Strategy group methods in the SDK Handle webhook notifications ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Watch Endpoint Source: https://docs.meter.sh/api-reference/rest/watch One-step URL monitoring setup # Watch Endpoint Set up URL monitoring in a single API call. The watch endpoint combines strategy generation and schedule creation, eliminating the need for multi-step orchestration. ## Create watch Create a new watch to monitor a URL for changes. ```http theme={null} POST /api/watch ``` ### Request body ```json theme={null} { "url": "https://example.com/products", "description": "Extract product names and prices", "webhook_url": "https://your-app.com/webhooks/meter", "interval_seconds": 3600, "name": "Product Monitor" } ``` | Field | Type | Required | Description | | ------------------ | ------- | -------- | -------------------------------------------------------------- | | `url` | string | Yes | URL to monitor for changes | | `description` | string | Yes | What to extract from the page | | `webhook_url` | string | No | Webhook URL for change notifications | | `interval_seconds` | integer | No | How often to check (default: 3600, minimum: 60) | | `name` | string | No | Name for the strategy (auto-generated if not provided) | | `webhook_metadata` | object | No | Custom JSON metadata included in every webhook payload | | `webhook_type` | string | No | `standard` or `slack`. Auto-detected from URL if not specified | ### Response ```json theme={null} { "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "preview_data": [ {"name": "Product A", "price": "$19.99"}, {"name": "Product B", "price": "$29.99"} ], "next_run_at": "2025-01-15T10:30:00Z" } ``` | Field | Type | Description | | -------------- | -------- | ---------------------------------------- | | `strategy_id` | UUID | ID of the created extraction strategy | | `schedule_id` | UUID | ID of the created schedule | | `preview_data` | array | Sample data extracted from the URL | | `next_run_at` | datetime | When the first scheduled scrape will run | ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/watch \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://news.ycombinator.com", "description": "Extract post titles, scores, and links", "webhook_url": "https://my-app.com/webhook", "interval_seconds": 1800 }' ``` ## What happens internally The watch endpoint performs two operations in a single transaction: 1. **Generates a strategy** - Analyzes the page and creates extraction selectors using AI 2. **Creates a schedule** - Sets up recurring scrapes with your specified interval This is equivalent to calling: ```bash theme={null} # Step 1: Generate strategy POST /api/strategies/generate # Step 2: Create schedule with the strategy_id POST /api/schedules ``` ## Managing your watch After creation, use the standard endpoints to manage your watch: * **Update schedule**: `PATCH /api/schedules/{schedule_id}` * **Pause/resume**: `PATCH /api/schedules/{schedule_id}` with `{"enabled": false}` * **Get changes**: `GET /api/schedules/{schedule_id}/changes` * **Delete**: `DELETE /api/schedules/{schedule_id}` * **Refine strategy**: `POST /api/strategies/{strategy_id}/refine` ## Error responses | Status | Description | | ------ | ------------------------------------------------------------- | | `400` | Invalid request (missing required fields, interval too short) | | `401` | Invalid or missing API key | | `429` | Rate limit exceeded (watch endpoint is rate-limited) | | `500` | Internal server error | | `503` | Service temporarily unavailable (AI service issues) | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Receive change notifications Poll for changes instead of webhooks Manage your schedules Refine extraction strategies ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Webhooks Source: https://docs.meter.sh/api-reference/rest/webhooks Test and verify webhook delivery # Webhooks Test your webhook endpoint to verify it can receive notifications from meter. ## Test webhook delivery Send a sample webhook payload to verify your endpoint is configured correctly. ```http theme={null} POST /api/webhooks/test ``` ### Request body ```json theme={null} { "webhook_url": "https://your-app.com/webhooks/meter" } ``` | Field | Type | Required | Description | | ------------- | ------ | -------- | ----------------------- | | `webhook_url` | string | Yes | The webhook URL to test | ### Response ```json theme={null} { "success": true, "status_code": 200, "message": "Webhook delivered successfully" } ``` | Field | Type | Description | | ------------- | ------- | ---------------------------------------------------------------- | | `success` | boolean | Whether the webhook was delivered successfully (2xx response) | | `status_code` | integer | HTTP status code returned by your endpoint (0 if request failed) | | `message` | string | Human-readable result message | ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/webhooks/test \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "webhook_url": "https://my-app.com/webhook" }' ``` ## Webhook payload formats Meter sends webhooks for both successful and failed jobs. The payload structure differs based on the job status. ### Success payload Sent when a job completes successfully: ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "status": "completed", "url": "https://example.com/products", "results": [ {"title": "Sample Product A", "price": "$19.99"}, {"title": "Sample Product B", "price": "$29.99"} ], "item_count": 2, "has_changes": true, "content_hash": "7f3d9a2b4c1e5f8a9b0c1d2e3f4a5b6c", "completed_at": "2025-01-15T10:30:12Z", "delivery_reason": "first_run", "metadata": {"project": "my-project"} } ``` | Field | Type | Description | | ----------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `job_id` | UUID | ID of the scrape job | | `schedule_id` | UUID | ID of the schedule that triggered the job | | `status` | string | Always `completed` for success webhooks | | `url` | string | The URL that was scraped | | `results` | array | Extracted data from the page | | `item_count` | integer | Number of items extracted | | `has_changes` | boolean | Whether content changed since last run | | `content_hash` | string | Hash of the extracted content | | `completed_at` | datetime | When the job completed | | `delivery_reason` | string | Why this webhook fired: `first_run` for the first successful delivery for this `schedule + url`, `content_changed` for subsequent deliveries triggered by a `content_hash` change. | | `metadata` | object | Custom JSON metadata from `webhook_metadata` (if configured) | `delivery_reason` is only included in `standard` webhook payloads. The `slack`, `slack_workflow`, and `discord` formatters cherry-pick fields for human display and do not forward it. ### Failure payload Sent when a job fails: ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "status": "failed", "url": "https://example.com/products", "error": "Page not accessible: 404 Not Found", "completed_at": "2025-01-15T10:30:12Z", "metadata": {"project": "my-project"} } ``` | Field | Type | Description | | -------------- | -------- | ------------------------------------------------------------ | | `job_id` | UUID | ID of the scrape job | | `schedule_id` | UUID | ID of the schedule that triggered the job | | `status` | string | Always `failed` for failure webhooks | | `url` | string | The URL that failed to scrape | | `error` | string | Error message describing what went wrong | | `completed_at` | datetime | When the job failed | | `metadata` | object | Custom JSON metadata from `webhook_metadata` (if configured) | Failure webhooks do not include `results`, `item_count`, `has_changes`, or `content_hash` fields. ## Webhook types Meter supports four webhook formats: | Type | Description | Secret required | | ---------------- | ---------------------------------------- | :------------------: | | `standard` | Full JSON payload (default) | Yes (auto-generated) | | `slack` | Formatted Slack incoming webhook message | Yes (auto-generated) | | `slack_workflow` | Slack Workflow Builder trigger payload | No | | `discord` | Formatted Discord webhook embed | No | ### Auto-detection rules The webhook type is auto-detected from the URL when not explicitly set: | URL pattern | Detected type | | ---------------------------------------------------------------------- | ---------------- | | Contains `hooks.slack.com/services/` | `slack` | | Contains `hooks.slack.com/triggers/` | `slack_workflow` | | Contains `discord.com/api/webhooks/` or `discordapp.com/api/webhooks/` | `discord` | | Everything else | `standard` | You can override auto-detection by setting `webhook_type` explicitly when creating a schedule. Discord and Slack Workflow types do not require a `webhook_secret`. For `standard` and `slack` types, a secret is auto-generated if not provided. ## Webhook secrets When a schedule has a `webhook_url`, Meter auto-generates a secret with a `whsec_` prefix. The secret is sent in the `X-Webhook-Secret` header on every webhook delivery. Verify this header to ensure requests are from Meter. See the [Webhooks Guide](/guides/webhooks#webhook-secrets) for verification examples. ## Retry behavior Failed deliveries are retried up to 5 times with exponential backoff: 15 minutes, 30 minutes, 1 hour, 2 hours, 4 hours. * **2xx**: Success, no retry * **4xx**: Permanent failure, no retry * **5xx / timeout / connection error**: Retries with backoff ## Error responses | Status | Description | | ------ | -------------------------------------- | | `400` | Invalid request (missing webhook\_url) | | `401` | Invalid or missing API key | | `422` | Invalid URL format | | `500` | Internal server error | | `503` | Service temporarily unavailable | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Learn how to handle webhook notifications Set up scheduled scraping with webhooks # Workflow Endpoints Source: https://docs.meter.sh/api-reference/rest/workflows REST API endpoints for managing DAG-based scraping workflows # Workflow Endpoints Create and manage multi-step scraping pipelines via HTTP. ## Create workflow Create a new workflow with nodes and edges. ```http theme={null} POST /api/workflows ``` ### Request body ```json theme={null} { "name": "Job Scraper", "description": "Scrape job listings then detail pages", "nodes": [ { "node_key": "index", "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "input_type": "static_urls", "static_urls": ["https://jobs.com/listings"] }, { "node_key": "details", "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "input_type": "upstream_urls", "url_field": "job_url" } ], "edges": [ { "source_node_key": "index", "target_node_key": "details" } ] } ``` ### Node fields | Field | Type | Required | Description | | ------------------- | ------ | ----------- | ------------------------------------------------------------------ | | `node_key` | string | Yes | Unique identifier within the workflow | | `strategy_id` | UUID | Yes | Strategy to use for scraping | | `input_type` | string | Yes | `static_urls`, `upstream_urls`, `upstream_data`, or `trigger_only` | | `static_urls` | array | Conditional | URLs for `static_urls` input type | | `url_field` | string | Conditional | Field name for `upstream_urls` input type | | `static_parameters` | object | No | API parameter overrides | | `parameter_config` | object | No | Map upstream fields to strategy parameters | ### Edge fields | Field | Type | Required | Description | | ----------------- | ------ | -------- | -------------------------------- | | `source_node_key` | string | Yes | Source node identifier | | `target_node_key` | string | Yes | Target node identifier | | `filter_config` | object | No | Filter configuration (see below) | ### Filter config ```json theme={null} { "mode": "all", "conditions": [ {"field": "category", "operator": "contains", "value": "tech", "case_sensitive": false} ] } ``` | Operator | Description | | -------------- | -------------------------------- | | `contains` | Field contains substring | | `not_contains` | Field does not contain substring | | `equals` | Exact match | | `not_equals` | Not exact match | | `regex_match` | Regex pattern match | | `exists` | Field exists and is non-empty | | `not_exists` | Field is missing or empty | | `gt` | Greater than | | `lt` | Less than | Use `mode: "all"` for AND logic, `mode: "any"` for OR logic. ### Response ```json theme={null} { "id": "990e8400-e29b-41d4-a716-446655440000", "name": "Job Scraper", "description": "Scrape job listings then detail pages", "nodes": [...], "edges": [...], "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/workflows \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "name": "Job Scraper", "nodes": [ { "node_key": "index", "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "input_type": "static_urls", "static_urls": ["https://jobs.com/listings"] }, { "node_key": "details", "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "input_type": "upstream_urls", "url_field": "job_url" } ], "edges": [ {"source_node_key": "index", "target_node_key": "details"} ] }' ``` ## Get workflow ```http theme={null} GET /api/workflows/{workflow_id} ``` Returns workflow details including nodes and edges. ## List workflows ```http theme={null} GET /api/workflows?limit=50&offset=0 ``` ### Query parameters | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------- | | `limit` | integer | No | Max results (default: 50) | | `offset` | integer | No | Results to skip (default: 0) | ## Update workflow Update workflow metadata. ```http theme={null} PUT /api/workflows/{workflow_id} ``` ### Request body ```json theme={null} { "name": "Updated Name", "description": "Updated description" } ``` ## Delete workflow Delete a workflow and all associated runs and schedules. ```http theme={null} DELETE /api/workflows/{workflow_id} ``` ## Add node Add a node to an existing workflow. ```http theme={null} POST /api/workflows/{workflow_id}/nodes ``` ### Request body ```json theme={null} { "node_key": "new_node", "strategy_id": "550e8400-e29b-41d4-a716-446655440000", "input_type": "upstream_urls", "url_field": "link" } ``` ## Update node ```http theme={null} PUT /api/workflows/{workflow_id}/nodes/{node_id} ``` ## Delete node ```http theme={null} DELETE /api/workflows/{workflow_id}/nodes/{node_id} ``` ## Add edge Connect two nodes. ```http theme={null} POST /api/workflows/{workflow_id}/edges ``` ### Request body ```json theme={null} { "source_node_key": "index", "target_node_key": "details", "filter_config": { "mode": "all", "conditions": [ {"field": "category", "operator": "contains", "value": "tech"} ] } } ``` ## Delete edge ```http theme={null} DELETE /api/workflows/{workflow_id}/edges/{edge_id} ``` ## Run workflow Trigger a manual workflow run. ```http theme={null} POST /api/workflows/{workflow_id}/run ``` ### Request body ```json theme={null} { "force": false } ``` | Field | Type | Required | Description | | ------- | ------- | -------- | ----------------------------------------------------------- | | `force` | boolean | No | Skip change detection and re-run all nodes (default: false) | ### Response ```json theme={null} { "id": "aa0e8400-e29b-41d4-a716-446655440000", "workflow_id": "990e8400-e29b-41d4-a716-446655440000", "status": "pending", "trigger": "manual", "created_at": "2025-01-15T10:30:00Z" } ``` ### Example ```bash theme={null} curl -X POST https://api.meter.sh/api/workflows/990e8400-e29b-41d4-a716-446655440000/run \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{"force": false}' ``` ## Get workflow run Get details of a specific run including node execution results. ```http theme={null} GET /api/workflows/{workflow_id}/runs/{run_id} ``` ### Response ```json theme={null} { "id": "aa0e8400-e29b-41d4-a716-446655440000", "workflow_id": "990e8400-e29b-41d4-a716-446655440000", "status": "completed", "trigger": "manual", "node_executions": [ { "node_key": "index", "status": "completed", "job_id": "bb0e8400-e29b-41d4-a716-446655440000", "item_count": 25, "started_at": "2025-01-15T10:30:01Z", "completed_at": "2025-01-15T10:30:05Z" }, { "node_key": "details", "status": "completed", "job_id": "cc0e8400-e29b-41d4-a716-446655440000", "item_count": 25, "started_at": "2025-01-15T10:30:06Z", "completed_at": "2025-01-15T10:30:30Z" } ], "created_at": "2025-01-15T10:30:00Z", "completed_at": "2025-01-15T10:30:30Z" } ``` The run response includes `node_executions` with `item_count` per node but does not include inline results. Use the [output endpoint](#get-latest-workflow-output) to fetch full results. **Run status values:** `pending`, `running`, `completed`, `failed`, `partial`, `cancelled` ## List workflow runs ```http theme={null} GET /api/workflows/{workflow_id}/runs?limit=20&offset=0 ``` ## Get latest workflow output Get the most recent completed run's results. By default, results are grouped by URL and then by strategy label. ```http theme={null} GET /api/workflows/{workflow_id}/runs/latest/output ``` ### Query parameters | Parameter | Type | Required | Description | | ---------------------- | ------- | -------- | --------------------------------------------------------------------------- | | `flat` | boolean | No | Return flat per-URL results instead of grouped by strategy (default: false) | | `include_intermediate` | boolean | No | Include outputs from all nodes, not just leaf nodes (default: false) | ### Response (default — grouped by strategy) ```json theme={null} { "workflow_id": "990e8400-e29b-41d4-a716-446655440000", "run_id": "aa0e8400-e29b-41d4-a716-446655440000", "status": "completed", "final_results_by_url_grouped": { "https://jobs.com/listings/software-engineer": { "default": [{"title": "Software Engineer", "salary": "$120k"}], "benefits": [{"health": "Yes", "dental": "Yes", "401k": "Yes"}] }, "https://jobs.com/listings/product-manager": { "default": [{"title": "Product Manager", "salary": "$130k"}] } }, "changed_since_previous": false, "completed_at": "2025-01-15T10:30:30Z" } ``` ### Response with `?flat=true` ```json theme={null} { "workflow_id": "990e8400-e29b-41d4-a716-446655440000", "run_id": "aa0e8400-e29b-41d4-a716-446655440000", "status": "completed", "final_results_by_url": { "https://jobs.com/listings/software-engineer": [ {"title": "Software Engineer", "salary": "$120k"}, {"health": "Yes", "dental": "Yes", "401k": "Yes"} ], "https://jobs.com/listings/product-manager": [ {"title": "Product Manager", "salary": "$130k"} ] }, "changed_since_previous": false, "completed_at": "2025-01-15T10:30:30Z" } ``` ### Response with `?include_intermediate=true` When `include_intermediate` is set, the response includes a `node_outputs` field with results from all nodes (not just leaf nodes): ```json theme={null} { "node_outputs": { "index": [{"job_url": "https://jobs.com/listings/software-engineer", "title": "Software Engineer"}], "details": [{"title": "Software Engineer", "salary": "$120k"}] } } ``` ## Cancel workflow run Cancel a running workflow. ```http theme={null} POST /api/workflows/{workflow_id}/runs/{run_id}/cancel ``` ## Workflow schedules ### Create workflow schedule ```http theme={null} POST /api/workflows/{workflow_id}/schedules ``` #### Request body ```json theme={null} { "interval_seconds": 3600, "webhook_url": "https://your-app.com/webhook", "webhook_metadata": {"project": "my-project"}, "webhook_secret": "whsec_your_secret_here", "webhook_type": "standard" } ``` | Field | Type | Required | Description | | ------------------ | ------- | ----------- | ------------------------------------------------------------------------------------------------------------------ | | `interval_seconds` | integer | Conditional | Run every N seconds | | `cron_expression` | string | Conditional | Cron expression | | `webhook_url` | string | No | Webhook URL for results | | `webhook_metadata` | object | No | Custom JSON metadata for webhook payloads | | `webhook_secret` | string | No | Secret for `X-Webhook-Secret` header | | `webhook_type` | string | No | `standard`, `slack`, `slack_workflow`, or `discord` (default: `standard`). Auto-detected from URL if not specified | Provide either `interval_seconds` or `cron_expression`, not both. ### List workflow schedules ```http theme={null} GET /api/workflows/{workflow_id}/schedules ``` ### Update workflow schedule ```http theme={null} PATCH /api/workflows/{workflow_id}/schedules/{schedule_id} ``` All fields are optional. Include only fields to update: ```json theme={null} { "enabled": false, "interval_seconds": 7200 } ``` ### Delete workflow schedule ```http theme={null} DELETE /api/workflows/{workflow_id}/schedules/{schedule_id} ``` ## Polling for run completion Since workflow runs are asynchronous, poll the run endpoint until status is `completed` or `failed`: ```bash theme={null} # Start a run RUN_ID=$(curl -s -X POST https://api.meter.sh/api/workflows/{workflow_id}/run \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{}' | jq -r '.id') # Poll for completion while true; do STATUS=$(curl -s https://api.meter.sh/api/workflows/{workflow_id}/runs/$RUN_ID \ -H "Authorization: Bearer sk_live_..." | jq -r '.status') echo "Status: $STATUS" if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then break fi sleep 5 done ``` Use the Python SDK's `run_workflow(wait=True)` to handle polling automatically. ## Error responses | Status | Description | | ------ | ---------------------------------------------------------------- | | `400` | Invalid request (missing required fields, invalid DAG structure) | | `401` | Invalid or missing API key | | `404` | Workflow, run, or node not found | | `409` | Workflow is already running | | `500` | Internal server error | | `503` | Service temporarily unavailable | See [REST API Errors](/api-reference/rest/errors) for detailed error handling. ## Next steps Understand workflow architecture and patterns Use the Python SDK for workflows Compare with simple schedules Receive workflow results via webhook ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Authentication Source: https://docs.meter.sh/authentication Learn how to authenticate API requests with Meter # Authentication Meter uses API keys to authenticate requests. All API requests must include your API key to identify your account and track usage. ## Getting your API key 1. Sign up or log in at [meter.sh](https://meter.sh/login) 2. Navigate to your [dashboard](https://meter.sh/dashboard) 3. Click **Generate API Key** 4. Copy your API key immediately—it will only be shown once **Save your API key immediately**. For security, Meter only shows the full API key once during creation. After that, only the prefix (e.g., `sk_live_abc...`) is visible. If you lose your key, you'll need to generate a new one and delete the old one. ## Managing API keys ### Creating a new key 1. Go to your [dashboard](https://meter.sh/dashboard) 2. Click **Generate API Key** 3. Copy the full key (starts with `sk_live_`) 4. Store it securely ### Deleting a key To revoke an API key: 1. Go to your dashboard 2. Find the key by its prefix (e.g., `sk_live_abc...`) 3. Click **Delete** 4. Confirm deletion Deleting a key immediately revokes access. Any requests using that key will return `401 Unauthorized`. ### Key rotation Manual rotation is supported: 1. Generate a new API key 2. Update your applications with the new key 3. Verify the new key works 4. Delete the old key from the dashboard This enables zero-downtime key rotation for production systems. ## API key format API keys follow this format: * **Live keys**: `sk_live_` + random characters * Used for production and development ## Using API keys ### Python SDK Store your API key in an environment variable: ```bash theme={null} export METER_API_KEY="sk_live_your_key_here" ``` Initialize the client: ```python theme={null} from meter_sdk import MeterClient import os # Recommended: Load from environment client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Alternative: Direct initialization (not recommended for production) client = MeterClient(api_key="sk_live_your_key_here") ``` ### REST API Include your API key in the `Authorization` header using the Bearer scheme: ```bash theme={null} curl https://api.meter.sh/api/strategies \ -H "Authorization: Bearer sk_live_your_key_here" \ -H "Content-Type: application/json" ``` ## Best practices Store API keys in environment variables, never in code Generate new keys periodically and delete old ones Use tools like AWS Secrets Manager or HashiCorp Vault in production Check your dashboard for unusual API activity ## Storing API keys securely ### Development Use environment variables or a `.env` file (add to `.gitignore`): ```bash .env theme={null} METER_API_KEY=sk_live_your_key_here ``` Load with python-dotenv: ```python theme={null} from dotenv import load_dotenv import os load_dotenv() api_key = os.getenv("METER_API_KEY") ``` ### Production Use AWS Secrets Manager or Systems Manager Parameter Store: ```python theme={null} import boto3 def get_api_key(): client = boto3.client('secretsmanager', region_name='us-east-1') response = client.get_secret_value(SecretId='meter-api-key') return response['SecretString'] ``` Pass as an environment variable: ```bash theme={null} docker run -e METER_API_KEY=sk_live_xxx your-image ``` Or use Docker secrets (Swarm mode): ```yaml docker-compose.yml theme={null} services: app: image: your-image secrets: - meter_api_key secrets: meter_api_key: external: true ``` Create a secret: ```bash theme={null} kubectl create secret generic meter-api-key \ --from-literal=key=sk_live_xxx ``` Mount in your pod: ```yaml theme={null} env: - name: METER_API_KEY valueFrom: secretKeyRef: name: meter-api-key key: key ``` Add to environment variables in your project settings: * Vercel: Project Settings → Environment Variables * Netlify: Site Settings → Environment Variables ## Error responses ### 401 Unauthorized Your API key is missing or invalid: ```json theme={null} { "detail": "Invalid or missing API key" } ``` **Solutions:** * Verify your API key is correct * Check that you're including the `Authorization` header * Ensure the key hasn't been deleted from the dashboard ### 403 Forbidden Your API key doesn't have permission for the requested resource: ```json theme={null} { "detail": "You do not have permission to access this resource" } ``` **Solutions:** * Verify you're accessing your own resources * Check that the resource exists ## Rate limits Rate limits are based on: * Requests per minute * Strategies generated per day * Jobs executed per hour Rate limit information is included in response headers. ## Security model Meter follows security best practices for API key handling: * **One-time display**: Full keys are shown only once during creation * **Prefix storage**: Only the key prefix is stored and displayed after creation * **Hashed storage**: Full keys are hashed using bcrypt before storage * **Immediate revocation**: Deleted keys are invalidated instantly * **Per-user isolation**: API keys can only access resources belonging to the authenticated user ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Change Detection Source: https://docs.meter.sh/concepts/change-detection How Meter detects meaningful content changes and filters out noise # Change Detection Meter's change detection system identifies when scraped content has actually changed, filtering out layout updates, ads, and timestamps that don't represent meaningful updates. ## Why change detection matters Traditional scraping wastes resources by re-processing unchanged data. For RAG systems, this means: * **Wasted embeddings**: Re-embedding identical content * **Stale timestamps**: Triggers on irrelevant date changes * **Layout noise**: Reacting to CSS class or ad changes * **Higher costs**: Unnecessary API calls and storage Meter solves this by comparing content structurally, detecting only meaningful changes. ## How it works Meter generates multiple signatures for each scrape job: A hash of the extracted data itself. Changes only if the actual content changes. A fingerprint of the content structure and patterns. Detects additions, removals, and reordering. ### Content hash The content hash is a cryptographic hash of the extracted data: ```python theme={null} job = client.get_job(job_id) print(job['content_hash']) # e.g., "7f3d9a2b4c1e..." ``` **Changes trigger when**: * Text content is different * Prices, numbers, or values change * New items appear or old ones disappear * Item order changes significantly **Doesn't change for**: * CSS classes or styling * Ad content (if not part of extraction) * Timestamps (if not extracted) ### Structural signature The structural signature captures patterns in the data: ```python theme={null} job = client.get_job(job_id) print(job['structural_signature']) # Structural fingerprint dict ``` This detects: * Number of items changing * Field presence/absence * Data type changes * List length changes ## Comparing jobs ### Automatic comparison (schedules) Schedules automatically compare new jobs with previous ones: ```python theme={null} # Create schedule schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600 ) # Check for changes changes = client.get_schedule_changes(schedule['schedule_id']) if changes['count'] > 0: print(f"Detected {changes['count']} jobs with changes") ``` Only jobs where content actually changed are returned. ### Manual comparison Compare two specific jobs: ```python theme={null} comparison = client.compare_jobs(job_id_1, job_id_2) print(f"Content hash match: {comparison['content_hash_match']}") print(f"Structural match: {comparison['structural_match']}") if not comparison['content_hash_match']: print("Content has changed!") ``` Use manual comparison to build custom change detection logic or investigate specific changes. ## Change detection strategies ### Pull-based monitoring Poll for changes periodically: ```python theme={null} # Check every hour for changes import time while True: changes = client.get_schedule_changes( schedule_id, mark_seen=True ) if changes['count'] > 0: print(f"Processing {changes['count']} changes") for change in changes['changes']: process_change(change['results']) time.sleep(3600) # Wait 1 hour ``` ### Webhook-based monitoring Receive immediate notifications: ```python theme={null} # Create schedule with webhook schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter" ) # Webhook endpoint (FastAPI example) from fastapi import FastAPI app = FastAPI() @app.post("/webhooks/meter") async def handle_webhook(payload: dict): if payload['has_changes']: # Process only changed content results = payload['results'] await update_vector_db(results) return {"status": "ok"} ``` ## Use cases Only re-embed when content changes: ```python theme={null} changes = client.get_schedule_changes(schedule_id) for change in changes['changes']: # Get current embeddings for this URL existing_vectors = vector_db.get(url=change['url']) # Delete old vectors vector_db.delete(existing_vectors.ids) # Generate new embeddings only for changed content new_vectors = embed(change['results']) vector_db.upsert(new_vectors) ``` **Savings**: Up to 95% reduction in embedding costs Alert only on actual price changes: ```python theme={null} changes = client.get_schedule_changes(schedule_id) for change in changes['changes']: for product in change['results']: current_price = float(product['price'].replace('$', '')) if current_price < price_threshold: send_alert(f"{product['name']} dropped to ${current_price}!") ``` Track when content was last updated: ```python theme={null} changes = client.get_schedule_changes(schedule_id) for change in changes['changes']: # Update "last modified" timestamp db.update( url=change['url'], last_modified=change['completed_at'] ) ``` ## Filtering noise Meter's change detection automatically filters: * **Layout changes**: CSS classes, div structure changes * **Ad rotations**: If ads aren't part of your extraction strategy * **Timestamps**: If not included in extraction fields * **Order changes**: Minor reordering that doesn't affect content To further filter noise in your extraction: ### Focus extractions Be specific about what you extract: ```python theme={null} # Don't extract dynamic timestamps result = client.generate_strategy( url="https://example.com", description="Extract article title and content only, ignore publish date" ) # Extract only static product info result = client.generate_strategy( url="https://shop.com/product/123", description="Extract product name, price, and description. Ignore related products and ads." ) ``` ### Compare strategically Only compare the fields that matter: ```python theme={null} def meaningful_change(job1, job2): """Check if price or availability changed, ignore descriptions""" for item1, item2 in zip(job1['results'], job2['results']): if item1['price'] != item2['price']: return True if item1['in_stock'] != item2['in_stock']: return True return False ``` ## Roadmap: Semantic similarity **Coming soon**: Semantic similarity detection using embeddings to detect meaning-level changes even when wording differs. Future versions will include: * Semantic comparison of text content * Paraphrase detection * Meaning-level change scoring This will enable even smarter filtering: "Product is now on sale" vs. "Item currently discounted" would be detected as semantically identical. ## Best practices Avoid duplicate processing by marking changes as seen: ```python theme={null} # Always use mark_seen=True in production changes = client.get_schedule_changes( schedule_id, mark_seen=True ) # Only use mark_seen=False for previewing preview = client.get_schedule_changes( schedule_id, mark_seen=False ) ``` Not all scrapes will detect changes: ```python theme={null} changes = client.get_schedule_changes(schedule_id) if changes['count'] == 0: print("No changes detected - content is fresh") else: process_changes(changes['changes']) ``` Track when changes are detected: ```python theme={null} changes = client.get_schedule_changes(schedule_id) logger.info(f"Checked schedule {schedule_id}: {changes['count']} changes") for change in changes['changes']: logger.info( f"Job {change['job_id']}: " f"{change['item_count']} items, " f"content_hash={change['content_hash']}" ) ``` ## Troubleshooting **Problem**: Changes detected for minor updates **Solutions**: * Make extraction more specific (exclude dynamic elements) * Regenerate strategy with clearer description * Implement custom filtering logic on top of Meter's detection **Problem**: Actual changes aren't detected **Possible causes**: * Changes already marked as seen * Looking at wrong schedule * Strategy extraction failing **Solutions**: * Use `mark_seen=False` to check without affecting state * Verify schedule ID * Check recent jobs for failures: `client.list_jobs(status='failed')` **Problem**: Want to understand why change was detected **Solution**: Compare jobs manually: ```python theme={null} # Get last two jobs jobs = client.list_jobs(strategy_id=strategy_id, limit=2) if len(jobs) >= 2: comparison = client.compare_jobs(jobs[0]['job_id'], jobs[1]['job_id']) print(f"Content hash match: {comparison['content_hash_match']}") print(f"Structural match: {comparison['structural_match']}") if 'changes' in comparison: for change in comparison['changes']: print(f" - {change}") ``` ## Next steps Implement change polling in your application Set up real-time change notifications Connect change detection to your vector database Explore job comparison methods ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Post-Extraction Filtering Source: https://docs.meter.sh/concepts/filtering Filter extraction results to keep only the data that matches your conditions # Post-Extraction Filtering **Post-extraction filtering** lets you define conditions that extraction results must match. Only items that pass the filter are kept — everything else is discarded. This runs after extraction, so you're filtering structured data, not raw HTML. ## When to use filters * Keep only products above a price threshold * Exclude items in certain categories * Match URLs against a pattern * Filter for items that have a specific field present * Combine conditions with AND/OR logic ## Filter structure A filter configuration has two parts: 1. **Mode** — `all` (AND) or `any` (OR) 2. **Conditions** — a list of field/operator/value checks ```json theme={null} { "mode": "all", "conditions": [ {"field": "price", "operator": "gt", "value": "50"}, {"field": "category", "operator": "contains", "value": "electronics"} ] } ``` With `mode: "all"`, an item must match **every** condition. With `mode: "any"`, an item matches if **at least one** condition is true. ## Operator reference | Operator | Description | Requires `value` | | -------------- | --------------------------------- | :--------------: | | `contains` | Field contains substring | Yes | | `not_contains` | Field does not contain substring | Yes | | `equals` | Exact match | Yes | | `not_equals` | Not an exact match | Yes | | `regex_match` | Regex pattern match | Yes | | `exists` | Field exists and is non-empty | No | | `not_exists` | Field is missing or empty | No | | `gt` | Greater than (numeric comparison) | Yes | | `lt` | Less than (numeric comparison) | Yes | All string operators support an optional `case_sensitive` flag (default: `false`). ## Where filters apply ### Strategy generation Pass a `filter_config` when generating a strategy to filter results at extraction time: ```python theme={null} result = client.generate_strategy( url="https://shop.com/products", description="Extract product listings", name="Premium Products", filter_config={ "mode": "all", "conditions": [ {"field": "price", "operator": "gt", "value": "100"}, {"field": "in_stock", "operator": "equals", "value": "true"} ] } ) ``` ### Strategy updates Update an existing strategy's filter: ```bash theme={null} curl -X PATCH https://api.meter.sh/api/strategies/{strategy_id} \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "filter_config": { "mode": "any", "conditions": [ {"field": "category", "operator": "equals", "value": "electronics"}, {"field": "category", "operator": "equals", "value": "computers"} ] } }' ``` ### Workflow edges Filters on workflow edges control which results pass between nodes. See [Workflows](/concepts/workflows#filters) for details. ```python theme={null} from meter_sdk.workflow import Workflow, Filter workflow = Workflow("Filtered Pipeline") index = workflow.start("index", index_strategy_id, urls=["https://news.com"]) # Only pass articles in the technology section tech = index.then( "tech", article_strategy_id, url_field="link", filter=Filter.contains("category", "technology") ) ``` ### Watch creation Pass `filter_config` when creating a watch (combined strategy + schedule): ```bash theme={null} curl -X POST https://api.meter.sh/api/watch \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://jobs.com/listings", "description": "Extract job listings", "name": "Remote Jobs", "interval_seconds": 3600, "filter_config": { "mode": "all", "conditions": [ {"field": "location", "operator": "contains", "value": "remote"} ] } }' ``` ## Examples ### Price threshold filtering Keep only items above a minimum price: ```json theme={null} { "mode": "all", "conditions": [ {"field": "price", "operator": "gt", "value": "50"} ] } ``` ### Category filtering with OR logic Keep items in any of several categories: ```json theme={null} { "mode": "any", "conditions": [ {"field": "category", "operator": "equals", "value": "electronics"}, {"field": "category", "operator": "equals", "value": "computers"}, {"field": "category", "operator": "equals", "value": "phones"} ] } ``` ### Regex URL pattern matching Keep items whose URL matches a pattern: ```json theme={null} { "mode": "all", "conditions": [ {"field": "url", "operator": "regex_match", "value": "/products/\\d+$"} ] } ``` ### Combined AND conditions Keep only in-stock premium products: ```json theme={null} { "mode": "all", "conditions": [ {"field": "price", "operator": "gt", "value": "100"}, {"field": "in_stock", "operator": "equals", "value": "true"}, {"field": "image_url", "operator": "exists"} ] } ``` ## Next steps Apply filters during strategy generation Use filters on workflow edges between nodes Define extraction result structure Update strategy filters via REST API ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Jobs Source: https://docs.meter.sh/concepts/jobs Execute scrapes and retrieve results using strategies # Jobs A **job** is a single execution of a scrape using a strategy. Jobs run asynchronously in the background, extract data according to the strategy's rules, and store the results for retrieval. ## What is a job? When you create a job, Meter: 1. Fetches the target URL 2. Applies the strategy's extraction logic (CSS selectors) 3. Extracts structured data 4. Generates content signatures for change detection 5. Stores results for retrieval Jobs are the execution layer—strategies define **what** to extract, jobs **execute** the extraction. ## Job lifecycle ```mermaid theme={null} graph LR A[Created] --> B[Pending] B --> C[Running] C --> D{Success?} D -->|Yes| E[Completed] D -->|No| F[Failed] ``` **Job statuses:** * `pending`: Job is queued, waiting to start * `running`: Job is currently executing * `completed`: Job finished successfully, results available * `failed`: Job encountered an error ## Creating jobs ### Basic job creation ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") # Create a job job = client.create_job( strategy_id="your-strategy-uuid", url="https://example.com/page" ) print(f"Job ID: {job['job_id']}") print(f"Status: {job['status']}") # Usually 'pending' ``` ### Waiting for completion Jobs run asynchronously. Use `wait_for_job()` to poll until completion: ```python theme={null} # Wait indefinitely (polls every 1 second by default) completed_job = client.wait_for_job(job['job_id']) print(f"Status: {completed_job['status']}") # 'completed' print(f"Items extracted: {completed_job['item_count']}") # Access results for item in completed_job['results']: print(item) ``` ### With timeout Set a timeout to avoid waiting forever: ```python theme={null} from meter_sdk import MeterError try: completed_job = client.wait_for_job( job['job_id'], poll_interval=2.0, # Check every 2 seconds timeout=300.0 # 5 minute timeout ) except MeterError as e: print(f"Job timed out or failed: {e}") ``` ## Checking job status Poll job status manually: ```python theme={null} job = client.get_job(job_id) print(f"Status: {job['status']}") print(f"Items: {job['item_count']}") if job['status'] == 'completed': results = job['results'] print(f"Extracted {len(results)} items") elif job['status'] == 'failed': print(f"Error: {job['error']}") ``` ## Job results Completed jobs contain extracted data in the `results` field: ```python theme={null} job = client.get_job(job_id) if job['status'] == 'completed': for item in job['results']: print(item) # Output example: # {'title': 'Product A', 'price': '$19.99', 'image': 'https://...'} # {'title': 'Product B', 'price': '$29.99', 'image': 'https://...'} ``` ### Job metadata Jobs also include metadata for change detection: ```python theme={null} job = client.get_job(job_id) print(f"Content hash: {job['content_hash']}") print(f"Structural signature: {job['structural_signature']}") print(f"Item count: {job['item_count']}") print(f"Started: {job['started_at']}") print(f"Completed: {job['completed_at']}") ``` * **`content_hash`**: Hash of the extracted content for quick comparison * **`structural_signature`**: Structural fingerprint for detecting layout changes * **`item_count`**: Number of items extracted ## Listing jobs ### All jobs ```python theme={null} # Get recent jobs (newest first) jobs = client.list_jobs(limit=20, offset=0) for job in jobs: print(f"Job {job['job_id']}: {job['status']}") ``` ### Filter by strategy ```python theme={null} # Get jobs for a specific strategy jobs = client.list_jobs( strategy_id="your-strategy-uuid", limit=50 ) ``` ### Filter by status ```python theme={null} # Get only completed jobs completed = client.list_jobs(status="completed", limit=10) # Get failed jobs to investigate errors failed = client.list_jobs(status="failed", limit=10) ``` ## Comparing jobs Compare two jobs to detect changes: ```python theme={null} comparison = client.compare_jobs(job_id_1, job_id_2) print(f"Content hash match: {comparison['content_hash_match']}") print(f"Structural match: {comparison['structural_match']}") print(f"Semantic similarity: {comparison['semantic_similarity']}") if not comparison['content_hash_match']: print("Content has changed!") if comparison['changes']: print(f"Detected changes:") for change in comparison['changes']: print(f" - {change}") ``` Use job comparison to build custom change detection logic beyond what's provided by schedules. ## Strategy history Get a timeline of all jobs for a strategy: ```python theme={null} history = client.get_strategy_history(strategy_id) for entry in history: print(f"Job {entry['job_id']} ({entry['created_at']}):") print(f" Status: {entry['status']}") print(f" Items: {entry['item_count']}") print(f" Has changes: {entry['has_changes']}") ``` The `has_changes` field indicates if content changed compared to the previous job. ## Advanced features The following features require feature gating. Contact [mckinnon@meter.sh](mailto:mckinnon@meter.sh) to request access. ### Antibot bypass Meter can handle antibot protection on pages that use common bot detection systems. When enabled for your account, jobs automatically attempt to bypass antibot measures when fetching pages. This is useful for scraping sites that use: * Cloudflare Bot Management * PerimeterX * DataDome * Other common antibot solutions No code changes are required—antibot handling is applied automatically when enabled for your account. ### LLM summary Jobs can include an LLM-generated summary of the page content. This is useful for: * Quick content overviews without parsing full results * Change detection at a semantic level * Building RAG pipelines with scraped content When enabled, completed jobs include a `summary` field with the AI-generated summary of the extracted content. ```python theme={null} job = client.get_job(job_id) if job['status'] == 'completed': print(f"Summary: {job.get('summary')}") print(f"Results: {job['results']}") ``` ## Best practices Jobs can fail if websites are down, block requests, or change structure: ```python theme={null} job = client.get_job(job_id) if job['status'] == 'failed': error = job['error'] print(f"Job failed: {error}") # Implement retry logic if "timeout" in error.lower(): # Retry with same strategy retry_job = client.create_job(strategy_id, url) elif "selector" in error.lower(): # Website structure changed, regenerate strategy new_strategy = client.generate_strategy(url, description, name) ``` Different sites have different response times: ```python theme={null} # Fast sites client.wait_for_job(job_id, timeout=60) # Slow sites or large pages client.wait_for_job(job_id, timeout=300) ``` Jobs are stored indefinitely. For large-scale monitoring: ```python theme={null} # Keep only recent jobs, delete old ones old_jobs = client.list_jobs( strategy_id=strategy_id, limit=100, offset=50 # Skip 50 most recent ) for job in old_jobs: # Delete if older than 30 days if should_delete(job['created_at']): client.delete_job(job['job_id']) ``` Job deletion is not yet implemented but is planned. For one-off scrapes, `wait_for_job()` is convenient: ```python theme={null} job = client.create_job(strategy_id, url) results = client.wait_for_job(job['job_id'])['results'] ``` For monitoring, use schedules instead of manually creating jobs. ## Troubleshooting **Possible causes:** * High server load * Job queue backlog **Solutions:** * Wait longer (jobs typically complete in 10-60 seconds) * Check status manually: `client.get_job(job_id)` * Contact support if stuck for >5 minutes **Cause:** Website HTML structure changed **Solution:** Generate a new strategy: ```python theme={null} new_strategy = client.generate_strategy( url=url, description=description, name=f"{old_name} (Updated)" ) ``` **Problem:** Job completes but `results` is empty **Possible causes:** * Strategy selectors don't match the page * Page content is dynamically loaded (JavaScript) **Solutions:** * Regenerate strategy for the current page structure * For JS-heavy sites, contact support (browser automation coming soon) ## Next steps Automate job execution with recurring schedules Learn how Meter detects content changes Explore all job methods View job endpoints in the REST API ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Manifest Comparison Source: https://docs.meter.sh/concepts/manifest-comparison Compare a known list of items against scrape results to detect additions and removals # Manifest Comparison **Manifest comparison** lets you submit a list of known items (a "manifest") and compare it against your scrape results using fuzzy matching. Meter identifies which items were added, removed, or still present — even when names don't match exactly. ## When to use manifest comparison * Track portfolio companies on a firm's website — detect when new companies are added or removed * Monitor team pages for personnel changes * Compare a known product catalog against a competitor's current listings * Verify that a list of partners or clients on a website matches your records ## How it works 1. You scrape a page using a strategy (e.g., extract company names from a portfolio page) 2. You submit your manifest — a JSON list of items you already know about 3. Meter fuzzy-matches each manifest item against the scraped results 4. You get back three lists: **matched**, **added**, and **removed** ``` Manifest (12 items) Website (13 items) ├── Acme Corp ←→ Acme Corporation ✓ matched (90%) ├── Beta Industries ✗ (not found) ✗ removed ├── Gamma Solutions ←→ Gamma Solutions Inc ✓ matched (95%) │ ... ... └── Delta Partners + added ``` ### Fuzzy matching Meter uses fuzzy string matching to handle common variations: | Manifest | Website | Score | | --------------- | ------------------- | ----- | | Acme Corp | Acme Corporation | 90 | | Beta Inc | Beta Industries | 86 | | JP Morgan | JPMorgan Chase | 85 | | Gamma Solutions | Gamma Solutions Inc | 95 | The default threshold is **80** (out of 100). Items scoring below the threshold are treated as non-matches. You can adjust this per request. Fuzzy matching handles abbreviations ("Corp" → "Corporation"), word order differences, and minor spelling variations. It does **not** handle semantic equivalence like "Facebook" → "Meta Platforms" or "IBM" → "International Business Machines". For those cases, consider lowering the threshold or using exact field matches on other identifiers (like URLs). ### Match fields You choose which field(s) to match on via `match_fields`. For example, if your scrape results have `name` and `website` fields, you can match on `["name"]` or `["name", "website"]`. When multiple match fields are provided, Meter takes the **best score** across fields. This means an exact URL match will count even if the name is slightly different. ## Quick example ```bash theme={null} # Compare your manifest against the latest results for a strategy curl -X POST https://api.meter.sh/api/strategies/{strategy_id}/compare-manifest \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "manifest": [ {"name": "Acme Corp"}, {"name": "Beta Industries"}, {"name": "Gamma Solutions"} ], "match_fields": ["name"], "threshold": 80 }' ``` ### Response ```json theme={null} { "matched": [ { "manifest_item": {"name": "Acme Corp"}, "scraped_item": {"name": "Acme Corporation", "website": "acme.com"}, "score": 90.0, "matched_on": "name" }, { "manifest_item": {"name": "Gamma Solutions"}, "scraped_item": {"name": "Gamma Solutions Inc", "website": "gamma.com"}, "score": 95.0, "matched_on": "name" } ], "added": [ {"name": "Delta Partners", "website": "delta.com"} ], "removed": [ {"name": "Beta Industries"} ], "summary": { "matched": 2, "added": 1, "removed": 1, "manifest_count": 3, "scraped_count": 3 }, "threshold_used": 80.0, "match_fields_used": ["name"] } ``` ## Typical workflow ### 1. Create a strategy with an output schema Define the exact fields you want extracted: ```bash theme={null} curl -X POST https://api.meter.sh/api/strategies/generate \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://example-vc.com/portfolio", "description": "Extract portfolio company names and websites", "name": "Portfolio Tracker", "output_schema": { "name": "string", "website": "string" } }' ``` ### 2. Schedule regular scrapes ```bash theme={null} curl -X POST https://api.meter.sh/api/schedules \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "STRATEGY_ID", "url": "https://example-vc.com/portfolio", "interval_seconds": 86400 }' ``` ### 3. Compare your manifest whenever you need ```bash theme={null} curl -X POST https://api.meter.sh/api/strategies/STRATEGY_ID/compare-manifest \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "manifest": [ {"name": "Company A", "website": "companya.com"}, {"name": "Company B", "website": "companyb.com"} ], "match_fields": ["name"], "threshold": 80 }' ``` ## Tuning the threshold | Threshold | Use case | | ---------------- | ----------------------------------------------------------------- | | **90-100** | Strict matching — names must be nearly identical | | **80** (default) | Balanced — handles "Corp" vs "Corporation", "Inc" vs "Industries" | | **60-70** | Loose — catches more variations but may produce false positives | Start with the default threshold of 80. If you see items incorrectly showing as "removed" that are actually on the site with a slightly different name, lower the threshold. If you see false matches, raise it. ## Endpoints There are two ways to compare a manifest: | Endpoint | Description | | -------------------------------------------- | ----------------------------------------------------------- | | `POST /api/strategies/{id}/compare-manifest` | Compare against the **latest completed job** for a strategy | | `POST /api/jobs/{id}/compare-manifest` | Compare against a **specific job's** results | The strategy endpoint is the most common choice — it automatically uses the most recent results. Use the job endpoint when you need to compare against a specific point in time. See the full API reference: [Jobs REST API](/api-reference/rest/jobs#compare-manifest) and [Strategies REST API](/api-reference/rest/strategies#compare-manifest). ## Best practices Define an `output_schema` when creating your strategy so that field names are predictable and consistent across scrapes. This makes `match_fields` reliable. Company names are usually the best match field. URLs can be a good secondary field. Avoid matching on generic fields like "description" where content varies significantly. If you have both `name` and `website` fields, use `match_fields: ["name", "website"]`. Meter takes the best score across fields, so an exact URL match will work even if the name format differs. Your manifest items only need to contain the fields listed in `match_fields`. Extra fields are preserved in the response but ignored during matching. ## Next steps Define consistent extraction shapes Automatic change tracking between scrapes Automate regular scrapes Full endpoint documentation ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Output Schemas Source: https://docs.meter.sh/concepts/output-schemas Define the exact JSON structure for extraction results # Output Schemas An **output schema** defines the exact JSON structure you want Meter to return from extractions. Instead of getting whatever field names the AI chooses, you specify the fields, types, and nesting — and Meter maps the extracted data to match. ## When to use output schemas * You need consistent field names across different strategies (e.g., all product scrapers return `price`, not `cost` or `product_price`) * You want type coercion (prices as numbers, not strings like `"$19.99"`) * Your downstream system expects a specific JSON shape * You're using [strategy groups](/concepts/strategy-groups) and want uniform output across all members ## Defining a schema A schema is a JSON object where keys are field names and values are type strings: ```json theme={null} { "title": "string", "price": "number", "in_stock": "boolean", "tags": "array", "image_url": "string" } ``` ### Supported types | Type | Description | Coercion behavior | | --------- | ----------------- | ----------------------------------------------------------------------------- | | `string` | Text value | Converts to string, `None` if empty | | `number` | Decimal number | Strips `$`, `€`, `£`, commas; parses float (e.g., `"$19.99"` → `19.99`) | | `float` | Same as `number` | Alias for `number` | | `integer` | Whole number | Same stripping as `number`, then rounds to int (e.g., `"225 points"` → `225`) | | `int` | Same as `integer` | Alias for `integer` | | `boolean` | True/false | `"true"`, `"yes"`, `"1"` → `True`; `"false"`, `"no"`, `"0"` → `False` | | `array` | List of values | If string, splits on commas; if single value, wraps in list | ### Nested schemas You can define nested structures: ```json theme={null} { "product": { "name": "string", "pricing": { "amount": "number", "currency": "string" } }, "seller": { "name": "string", "rating": "float" } } ``` Meter flattens extracted data internally using underscore-separated keys (e.g., `product_name`, `product_pricing_amount`) and then re-nests it to match your schema structure. ### Wrapper array pattern For schemas with a single top-level key containing an array, Meter automatically wraps results: ```json theme={null} { "listings": [ { "title": "string", "price": "number" } ] } ``` This produces output like: ```json theme={null} { "listings": [ {"title": "MacBook Pro", "price": 1999}, {"title": "iPad Air", "price": 599} ] } ``` ## Where schemas apply ### Strategy generation Pass an `output_schema` when generating a strategy to guide extraction and get consistent output from the start: ```python theme={null} result = client.generate_strategy( url="https://shop.com/products", description="Extract product listings", name="Product Scraper", output_schema={ "product_name": "string", "price": "number", "rating": "float", "in_stock": "boolean" } ) # Preview data matches the schema structure for item in result["preview_data"]: print(f"{item['product_name']}: ${item['price']}") ``` ### Strategy groups Apply a schema to all strategies in a group at once: ```python theme={null} client.apply_group_schema( group_id="group-uuid", output_schema={ "title": "string", "price": "number", "url": "string" } ) ``` See [Strategy Groups](/concepts/strategy-groups#group-output-schemas) for details. ### Bulk upload When using [Bulk Upload](/guides/bulk-upload), you can define an output schema that applies to all generated strategies in the batch. ## URL field handling Meter automatically detects URL fields in your schema (keys containing `url`, `link`, or `href`) and optimizes extraction to capture the `href` attribute rather than link text. You don't need to configure this — it happens automatically. ## Examples ### E-commerce product schema ```json theme={null} { "product_name": "string", "price": "number", "original_price": "number", "discount_percentage": "integer", "rating": "float", "review_count": "integer", "in_stock": "boolean", "image_url": "string", "product_url": "string" } ``` ### Job listing schema ```json theme={null} { "title": "string", "company": "string", "location": "string", "salary_min": "integer", "salary_max": "integer", "job_type": "string", "posted_date": "string", "apply_url": "string", "tags": "array" } ``` ## Next steps Apply schemas to many strategies at once Filter extraction results post-processing Learn how strategies use output schemas Create many strategies with a shared schema ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Core Concepts Source: https://docs.meter.sh/concepts/overview Understand the fundamental building blocks of Meter # Core Concepts Meter is built around a few key abstractions that make web scraping and monitoring simple and cost-effective. Understanding these concepts will help you get the most out of the platform. ## The big picture A reusable extraction plan generated by AI that defines **how** to scrape a website. Created once, used many times. A single execution of a scrape using a strategy. Jobs run asynchronously and return extracted data. Automated recurring jobs that run at specified intervals or cron times. Perfect for monitoring websites. Chain strategies into DAG-based pipelines where the output of one scraper feeds into the next. For multi-step scraping like index → detail pages. Intelligent diffing that compares jobs to detect meaningful content changes, filtering out noise. ## Key concepts Learn how AI-generated extraction strategies work and when to use them Organize strategies for bulk scheduling and shared output schemas Define the exact JSON structure for extraction results Filter extraction results with conditions and operators Understand job execution, status checking, and result retrieval Set up automated monitoring with intervals or cron expressions Chain strategies into multi-step scraping pipelines Discover how Meter detects meaningful content changes ## How it all fits together ```mermaid theme={null} graph TD A[Generate Strategy] --> B{Simple or Pipeline?} B -->|Simple| C[Create Schedule] B -->|Pipeline| W[Create Workflow] W --> C C --> D[Jobs Run Automatically] D --> E[Compare with Previous Job] E --> F{Changes Detected?} F -->|Yes| G[Webhook or Pull API] F -->|No| H[Skip - No Action] G --> I[Update Vector DB] ``` ### Example workflow 1. **Generate a strategy** for extracting product data from an e-commerce site 2. **Create a schedule** to scrape the site every hour 3. **Jobs run automatically**, extracting current product data 4. **Changes are detected** by comparing content hashes and structural signatures 5. **You get notified** via webhook or pull from the changes API 6. **Update your database** only with changed content ## Cost model Understanding Meter's cost structure helps you optimize usage: | Action | Cost | Frequency | | ----------------------- | ------------- | --------------------- | | **Strategy generation** | \~\$0.02-0.06 | Once per site/pattern | | **Job execution** | Included | Unlimited | | **Change detection** | Included | Automatic | | **API calls** | Included | Unlimited | ### Why strategy-based is cheaper Traditional LLM scraping costs scale with usage: * **Traditional**: Pay per scrape (\$0.02-0.10 each) * **Meter**: Pay once for strategy (\$0.02-0.06), then scrape unlimited times at no extra cost For a site scraped 100 times: * Traditional LLM scraping: \$2-10 * Meter: \$0.02-0.06 (97-99% savings) ## Data model Understanding the data model helps you work with the API: ``` User ├── Strategy Groups │ └── Strategies (grouped for bulk management) ├── Strategies │ ├── Preview Data (sample extraction) │ ├── Output Schema (optional) │ ├── Filter Config (optional) │ ├── Jobs │ │ ├── Results (extracted data) │ │ ├── Content Hash │ │ └── Structural Signature │ └── Schedules │ ├── Interval or Cron │ ├── Webhook URL (optional) │ └── Associated Jobs └── Workflows ├── Nodes (strategy + input config) ├── Edges (data flow + filters) ├── Runs (execution history) └── Schedules (interval or cron) ``` ## Best practices If multiple pages have the same structure (e.g., product pages, blog posts), you can reuse the same strategy with different URLs: ```python theme={null} # Generate once for pattern strategy = client.generate_strategy( url="https://shop.com/product/123", description="Extract name, price, description" ) # Reuse for different products job1 = client.create_job(strategy_id, "https://shop.com/product/123") job2 = client.create_job(strategy_id, "https://shop.com/product/456") ``` Instead of webhooks for every change, poll the changes API periodically: ```python theme={null} # Check once per hour for all changes changes = client.get_schedule_changes(schedule_id) if changes['count'] > 0: batch_process(changes['changes']) ``` This reduces webhook traffic and allows batching updates. **Faster intervals** (15-30 min): * Stock prices, sports scores, breaking news * High-priority monitoring **Moderate intervals** (1-6 hours): * E-commerce products, job listings * Most monitoring use cases **Slow intervals** (daily): * Documentation, blog posts, policies * Low-frequency content Jobs can fail if sites are down or block requests: ```python theme={null} job = client.get_job(job_id) if job['status'] == 'failed': print(f"Job failed: {job['error']}") # Retry logic, alerts, etc. ``` ## Next steps Learn how AI generates extraction strategies Master job execution and result handling Set up automated monitoring Build multi-step scraping pipelines ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Schedules Source: https://docs.meter.sh/concepts/schedules Automate scraping with recurring jobs and change notifications # Schedules A **schedule** automatically runs scrape jobs at specified intervals or cron times. Schedules are perfect for monitoring websites for changes without manual intervention. ## What is a schedule? A schedule combines: * **A strategy**: What to extract * **A URL**: Where to scrape * **A timing rule**: When to scrape (interval or cron) * **Optional webhook**: Where to send change notifications Once created, schedules run automatically, creating jobs at the specified times. ## Creating schedules ### Interval-based schedules Run jobs at regular intervals: ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") # Run every hour (3600 seconds) schedule = client.create_schedule( strategy_id="your-strategy-uuid", url="https://example.com/page", interval_seconds=3600 ) print(f"Schedule ID: {schedule['schedule_id']}") print(f"Next run: {schedule['next_run_at']}") ``` Common intervals: * **15 minutes**: `900` seconds * **1 hour**: `3600` seconds * **6 hours**: `21600` seconds * **Daily**: `86400` seconds ### Cron-based schedules Use cron expressions for precise scheduling: ```python theme={null} # Run daily at 9 AM schedule = client.create_schedule( strategy_id="your-strategy-uuid", url="https://example.com/page", cron_expression="0 9 * * *" ) # Run every weekday at 8 AM schedule = client.create_schedule( strategy_id="your-strategy-uuid", url="https://example.com/page", cron_expression="0 8 * * 1-5" ) # Run every 6 hours schedule = client.create_schedule( strategy_id="your-strategy-uuid", url="https://example.com/page", cron_expression="0 */6 * * *" ) ``` Use [crontab.guru](https://crontab.guru/) to build and test cron expressions. ## Webhooks Receive real-time notifications when jobs complete: ```python theme={null} schedule = client.create_schedule( strategy_id="your-strategy-uuid", url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter" ) ``` Meter will POST job results to your webhook URL. See the [webhooks guide](/guides/webhooks) for payload details and implementation. ### Webhook types Meter supports two webhook formats: `standard` (full JSON payload) and `slack` (formatted for Slack incoming webhooks). The type is auto-detected from the URL — Slack URLs are automatically detected. ```python theme={null} # Slack webhook (auto-detected from URL) schedule = client.create_schedule( strategy_id="your-strategy-uuid", url="https://example.com/products", interval_seconds=3600, webhook_url="https://hooks.slack.com/services/T.../B.../xxx" ) ``` ### Webhook metadata Attach custom JSON metadata to every webhook payload. This is useful for routing, tagging, or identifying schedules: ```python theme={null} schedule = client.create_schedule( strategy_id="your-strategy-uuid", url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter", webhook_metadata={"project": "price-monitor", "env": "prod"} ) ``` ### Webhook secrets Every schedule with a webhook URL gets an auto-generated secret (prefixed `whsec_`). The secret is sent in the `X-Webhook-Secret` header on every delivery, allowing you to verify requests are from Meter. The secret is returned **once** when the schedule is created — store it securely. If compromised, regenerate it: ```python theme={null} result = client.regenerate_webhook_secret(schedule_id) new_secret = result["webhook_secret"] ``` See the [webhooks guide](/guides/webhooks#webhook-secrets) for verification examples. ## Pull-based change detection Instead of webhooks, poll for changes: ```python theme={null} # Create schedule without webhook schedule = client.create_schedule( strategy_id="your-strategy-uuid", url="https://example.com/products", interval_seconds=3600 ) # Later, check for changes changes = client.get_schedule_changes( schedule_id=schedule['schedule_id'], mark_seen=True # Mark changes as seen after reading ) if changes['count'] > 0: print(f"Found {changes['count']} jobs with changes") for change in changes['changes']: print(f"Job {change['job_id']}: {change['item_count']} items") # Process change['results'] ``` Set `mark_seen=False` to preview changes without marking them as read. This is useful for: * **Batch processing**: Check for changes once per hour, process in bulk * **Webhook alternatives**: When webhooks aren't feasible * **Manual review**: Preview changes before processing ## Managing schedules ### Listing schedules ```python theme={null} schedules = client.list_schedules() for schedule in schedules: print(f"Schedule {schedule['schedule_id']}:") print(f" Enabled: {schedule['enabled']}") print(f" Type: {schedule['schedule_type']}") # 'interval' or 'cron' print(f" Next run: {schedule['next_run_at']}") ``` ### Updating schedules ```python theme={null} # Disable a schedule temporarily client.update_schedule( schedule_id, enabled=False ) # Change the interval client.update_schedule( schedule_id, interval_seconds=7200 # Every 2 hours instead ) # Switch to cron client.update_schedule( schedule_id, cron_expression="0 10 * * *" # Daily at 10 AM ) # Update webhook URL client.update_schedule( schedule_id, webhook_url="https://your-new-domain.com/webhooks/meter" ) # Remove webhook (use pull-based instead) client.update_schedule( schedule_id, webhook_url=None ) ``` ### Deleting schedules ```python theme={null} # Delete a schedule (stops future jobs) client.delete_schedule(schedule_id) ``` Deleting a schedule doesn't delete past jobs. Use `list_jobs()` to access historical data. ## Monitoring schedules ### Check recent runs ```python theme={null} # Get jobs created by this schedule jobs = client.list_jobs( strategy_id=schedule['strategy_id'], limit=20 ) for job in jobs: print(f"Job {job['job_id']} ({job['created_at']}):") print(f" Status: {job['status']}") print(f" Items: {job['item_count']}") ``` ### Detect failures ```python theme={null} # Check for recent failures failed_jobs = client.list_jobs( strategy_id=schedule['strategy_id'], status='failed', limit=5 ) if len(failed_jobs) > 0: print(f"Warning: {len(failed_jobs)} recent failures") print(f"Error: {failed_jobs[0]['error']}") ``` ## Best practices Balance freshness with cost and load: **Every 15-30 minutes**: * Stock prices, sports scores * Time-sensitive monitoring * High-value data **Every 1-6 hours**: * E-commerce products * News articles * Job listings * Most monitoring use cases **Daily**: * Documentation, policies * Blog posts * Low-frequency content Webhooks are ideal when: * Changes need immediate action * Building real-time systems * Triggering downstream workflows Pull-based is better when: * Batch processing changes * Webhooks aren't feasible (firewall, no public endpoint) * Manual review before processing Set up alerts for schedule failures: ```python theme={null} import time def check_schedule_health(schedule_id, threshold=3): """Alert if >threshold failures in recent jobs""" failed = client.list_jobs( strategy_id=schedule['strategy_id'], status='failed', limit=10 ) if len(failed) >= threshold: send_alert(f"Schedule {schedule_id} has {len(failed)} failures") ``` Temporarily disable schedules when doing maintenance: ```python theme={null} # Disable before maintenance client.update_schedule(schedule_id, enabled=False) # Do maintenance work update_strategy_or_database() # Re-enable after client.update_schedule(schedule_id, enabled=True) ``` ## Change detection workflow Schedules automatically compare jobs to detect changes: ```mermaid theme={null} graph TD A[Schedule Runs Job] --> B[Extract Data] B --> C[Generate Content Hash] C --> D{Compare with Previous} D -->|Different Hash| E[Mark as Changed] D -->|Same Hash| F[Mark as Unchanged] E --> G[Available in Changes API] E --> H[Send Webhook if configured] F --> I[Not returned by Changes API] ``` When you call `get_schedule_changes()`, Meter returns only jobs where content actually changed. ## Troubleshooting **Possible causes:** * Schedule is disabled * Cron expression is incorrect * Server issues **Solutions:** * Check `enabled` field: `client.get_schedule(schedule_id)` * Verify cron expression at [crontab.guru](https://crontab.guru/) * Check `next_run_at` to see when it's scheduled **Problem:** `get_schedule_changes()` returns 0 results but you expect changes **Possible causes:** * Content genuinely hasn't changed * Changes already marked as seen * Looking at wrong schedule **Solutions:** * Use `mark_seen=False` to check without marking * Compare jobs manually: `client.compare_jobs(job1_id, job2_id)` * Verify schedule ID is correct **Problem:** Webhooks aren't being received **Solutions:** * Verify webhook URL is publicly accessible * Check endpoint responds with 200 OK within 30 seconds * Test webhook with tools like [webhook.site](https://webhook.site) * Switch to pull-based if webhooks aren't working ## Next steps Learn how Meter detects content changes Implement webhook endpoints for real-time updates Use the changes API for batch processing Explore all schedule methods ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Site Crawling Source: https://docs.meter.sh/concepts/site-crawling Automatically discover URLs on a website for batch scraping # Site Crawling **Site crawling** lets you automatically discover URLs on a website so you can scrape them in bulk. Instead of manually collecting URLs, you configure a discovery method and Meter finds all matching pages. ## Why use site crawling? Traditional scraping requires you to know every URL upfront. Site crawling solves this by: * **Discovering URLs automatically** from sitemaps, pagination, or link patterns * **Filtering URLs** to target only the pages you need * **Batch execution** to scrape hundreds or thousands of pages at once * **Creating schedules** to re-crawl and scrape on a recurring basis ## Discovery methods Meter supports three methods for discovering URLs: Parse sitemap.xml files to extract all indexed URLs Generate URLs by incrementing a page number in a template Crawl a site and collect URLs matching a pattern ### Sitemap discovery Best for sites with a `sitemap.xml` file. Meter parses the sitemap (including nested sitemaps) and extracts all URLs. **Configuration:** * **Sitemap URL**: The URL to the sitemap.xml file * **URL Pattern** (optional): Glob pattern to filter URLs (e.g., `products/*/`) * **Max URLs**: Maximum number of URLs to discover (1-10,000) **Example use case**: Scraping all product pages from an e-commerce site that maintains a sitemap. ### Pagination discovery Best for sites with predictable paginated URLs. You provide a URL template with a page number placeholder. **Configuration:** * **URL Template**: URL with `{n}` placeholder for page number (e.g., `https://shop.com/products?page={n}`) * **Start Index**: First page number (default: 1) * **Step**: Increment between pages (default: 1) * **Max Pages**: Maximum pages to generate **Example use case**: Scraping all pages of search results or product listings where URLs follow a pattern like `?page=1`, `?page=2`, etc. ### Link pattern discovery Best for sites without sitemaps or predictable pagination. Meter crawls from a seed URL and collects links matching your pattern. **Configuration:** * **Seed URL**: Starting URL for the crawl * **Link Pattern**: Glob pattern for URLs to collect (e.g., `/product/*/`) * **Navigation Pattern** (optional): Pattern for pages to visit during crawl (e.g., `/category/`) * **Max Depth**: How many links deep to crawl (1-10) * **Max URLs**: Maximum URLs to discover **Example use case**: Discovering all article pages on a news site by crawling category pages and collecting article links. ## How site crawling works ```mermaid theme={null} graph LR A[Configure Discovery] --> B[Start Crawl] B --> C[URLs Discovered] C --> D{Review Results} D --> E[Execute Now] D --> F[Create Schedule] E --> G[Jobs Created] F --> H[Recurring Scrapes] ``` 1. **Configure**: Choose a discovery method and set parameters 2. **Discover**: Meter crawls and finds matching URLs 3. **Review**: Check the discovered URLs and adjust if needed 4. **Execute**: Run a one-time batch scrape or create a recurring schedule ## Execution options After discovering URLs, you can: ### One-time execution Create scrape jobs for all discovered URLs immediately. Each URL becomes a separate job that runs through your chosen strategy. * Jobs are created with a shared `batch_id` for tracking * URL filtering with regex is supported * Set a maximum number of URLs to process ### Scheduled execution Create a recurring schedule that re-runs the scrape on a regular basis. * **Interval-based**: Run every N hours/days (e.g., every 24 hours) * **Cron-based**: Run on a cron schedule (e.g., `0 9 * * *` for 9 AM daily) * **Webhook notifications**: Get notified when scrapes complete Schedules store a copy of the discovered URLs. If you need to update the URL list, create a new schedule from a fresh discovery. ## Best practices Sitemaps are the fastest and most reliable discovery method. Check if your target site has one at `/sitemap.xml` or in `robots.txt` before trying other methods. Most sitemaps include URLs you don't need (about pages, terms of service, etc.). Use URL patterns to filter down to just the pages you want to scrape. For example, `products/*/` matches product pages while excluding other site content. Start with a low `max_urls` limit (e.g., 10-50) to verify your configuration before running a full crawl. This saves time and resources. Make sure your extraction strategy works with the pages you're discovering. If you're crawling product pages, use a strategy created from a product page. ## Limits | Parameter | Limit | | ------------------------------ | ------ | | Max URLs per discovery | 10,000 | | Max crawl depth (link pattern) | 10 | | Max pages (pagination) | 1,000 | ## Next steps Step-by-step guide to crawling your first site View all discovery endpoints Learn how to create extraction strategies Set up recurring scrapes # Strategies Source: https://docs.meter.sh/concepts/strategies AI-generated extraction plans that define how to scrape a website # Strategies A **strategy** is a reusable extraction plan that tells Meter how to extract data from a webpage. Think of it like a recipe: you create it once by describing what you want, and Meter's AI figures out the exact selectors and extraction logic. ## What is a strategy? A strategy contains: * **Extraction method**: Either CSS Path (for traditional HTML) or API Path (for JavaScript-heavy sites) * **Field definitions** mapping selectors or API responses to your data fields * **Extraction metadata** like item containers, scopes, or API endpoints * **Output schema** (optional): Defines the exact JSON structure for results. See [Output Schemas](/concepts/output-schemas) * **Filter config** (optional): Post-extraction filtering conditions. See [Filtering](/concepts/filtering) * **Strategy group** (optional): Group membership for bulk management. See [Strategy Groups](/concepts/strategy-groups) Meter automatically detects which extraction method works best for each site—you don't need to choose. Once created, a strategy can be reused unlimited times across similar pages—**no LLM costs after initial generation**. ## How strategies are generated Meter uses AI to analyze your target webpage and generate precise extraction strategies: 1. **You provide**: A URL and plain-English description of what to extract 2. **Meter analyzes**: The page structure, HTML patterns, and content layout 3. **AI generates**: CSS selectors and extraction rules optimized for that page 4. **You get**: A reusable strategy plus preview data showing what was extracted This approach combines the intelligence of AI setup with the speed and reliability of traditional scraping. ### Example ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") # Generate a strategy result = client.generate_strategy( url="https://news.ycombinator.com", description="Extract post titles and scores", name="HN Front Page" ) # Check the preview print(f"Extracted {len(result['preview_data'])} items") for item in result['preview_data'][:3]: print(item) # Output: # {'title': 'Launch HN: ...', 'score': 42} # {'title': 'Ask HN: ...', 'score': 15} # ... ``` ## Extraction methods Meter automatically selects the best extraction method for each site. You describe what you want, and Meter figures out how to get it. ### CSS Path extraction For traditional HTML pages, Meter generates CSS selectors that target the content you need. **Best for:** * Static HTML sites * Server-rendered pages * Sites with stable DOM structure * Blogs, news sites, and content pages CSS Path extraction is fast and reliable for sites where content is present in the initial HTML response. ### API Path extraction For JavaScript-heavy sites, Meter automatically discovers the underlying APIs that power the page and extracts data directly from them. Meter identifies when a page relies on JavaScript to load its content. The data source APIs are automatically identified—no reverse engineering required. Any required tokens or session data are handled automatically. Data is extracted directly from API responses—cleaner and more reliable than parsing the DOM. **Best for:** * Single-page applications (React, Vue, Angular) * Financial data sites * Dynamic dashboards * Sites with client-side rendering API Path extraction often returns cleaner, more structured data than DOM scraping—and it's more resilient to UI changes. ## Automatic token handling JavaScript-heavy sites often require authentication tokens to access their APIs. Meter handles this automatically—you don't need to worry about the details. Many sites protect their APIs with CSRF tokens. Meter detects and includes these tokens automatically, so your extractions work without manual configuration. Session state is maintained across the extraction process. If a site requires cookies to access its APIs, Meter handles this for you. API keys, authorization headers, and other required headers are automatically included in requests. Some sites require multiple API calls in sequence. Meter handles these dependencies and chains requests in the correct order. ## Real-world example: Financial data Consider extracting stock quotes from a financial data site. When you visit the page, you see prices updating in real-time—but the HTML source shows almost nothing. The data is loaded via JavaScript from a hidden API. With traditional scraping, you would need to: 1. Reverse-engineer the API endpoints 2. Figure out the authentication requirements 3. Handle CSRF tokens and session management 4. Parse the JSON response format **With Meter**, you simply describe what you want: "Extract stock symbol, current price, and daily change." Meter automatically: * Discovers the quote API endpoint * Captures any required authentication tokens * Extracts the structured data from API responses * Returns clean, normalized data The result is faster extraction, cleaner data, and a strategy that's resilient to UI redesigns—because you're hitting the same API the site uses internally. ## Strategy lifecycle ```mermaid theme={null} graph LR A[Generate] --> B{Preview OK?} B -->|Yes| C[Use in Jobs] B -->|No| D[Refine] D --> B C --> E[Monitor Results] E --> F{Still Working?} F -->|Yes| C F -->|No| D ``` 1. **Generate**: Create strategy with AI 2. **Preview**: Check the `preview_data` to verify extraction 3. **Refine** (optional): Provide feedback if something's missing 4. **Use**: Run jobs with the strategy 5. **Monitor**: Check if results are still accurate over time ## Refining strategies If the initial extraction isn't perfect, refine it with feedback: ```python theme={null} # Initial generation result = client.generate_strategy( url="https://shop.com/products", description="Extract product info", name="Product Scraper" ) # Check preview - oops, missing images print(result['preview_data']) # No 'image' field # Refine with feedback refined = client.refine_strategy( strategy_id=result['strategy_id'], feedback="Also extract product images" ) # Check again print(refined['preview_data']) # Now has 'image' field ``` Refinement uses cached HTML from the initial generation, so it's fast and doesn't re-fetch the page. ## When to create new strategies Create a new strategy when: Each website layout needs its own strategy Different extraction requirements need different strategies If a site changes its HTML structure significantly Product pages vs. category pages need separate strategies ## Reusing strategies You can reuse the same strategy across: * **Multiple URLs** on the same site (e.g., different products) * **Pagination** (if the structure is consistent) * **Similar pages** (if they share HTML structure) ```python theme={null} # Generate once strategy = client.generate_strategy( url="https://shop.com/product/123", description="Extract name, price, description" ) strategy_id = strategy['strategy_id'] # Reuse for different products job1 = client.create_job(strategy_id, "https://shop.com/product/123") job2 = client.create_job(strategy_id, "https://shop.com/product/456") job3 = client.create_job(strategy_id, "https://shop.com/product/789") ``` ## Strategy management ### Listing strategies ```python theme={null} # Get all strategies strategies = client.list_strategies(limit=20) for strategy in strategies: print(f"{strategy['name']}: {strategy['strategy_id']}") ``` ### Getting strategy details ```python theme={null} strategy = client.get_strategy(strategy_id) print(f"Name: {strategy['name']}") print(f"Description: {strategy['description']}") print(f"Created: {strategy['created_at']}") print(f"Preview: {strategy['preview_data']}") ``` ### Deleting strategies ```python theme={null} # Delete a strategy (also deletes associated jobs and schedules) client.delete_strategy(strategy_id) ``` Deleting a strategy also deletes all associated jobs and schedules. This action cannot be undone. ## Best practices Give strategies clear names that describe their purpose: **Good**: `"HN Front Page - Titles and Scores"` **Bad**: `"Strategy 1"` This helps when managing multiple strategies. Always check `preview_data` before creating jobs: ```python theme={null} result = client.generate_strategy(...) # Verify all required fields are present required_fields = {'title', 'price', 'image'} actual_fields = set(result['preview_data'][0].keys()) if not required_fields.issubset(actual_fields): missing = required_fields - actual_fields client.refine_strategy( strategy_id=result['strategy_id'], feedback=f"Also extract: {', '.join(missing)}" ) ``` Provide clear, specific extraction instructions: **Good**: "Extract product name, price with currency, main image URL, and stock availability from the product grid" **Bad**: "Get products" Specific descriptions lead to better strategies on the first try. Strategies can break if sites change their HTML: ```python theme={null} # Check recent jobs for failures jobs = client.list_jobs( strategy_id=strategy_id, status='failed', limit=5 ) if len(jobs) > 0: print(f"Strategy {strategy_id} may need updating") ``` ## Troubleshooting **Possible causes**: * URL is not accessible * Page requires authentication * Description is too vague **Solutions**: * Verify the URL loads in a browser * For auth-required pages, contact support * Make your description more specific **Problem**: Some expected fields aren't in `preview_data` **Solution**: Use refinement: ```python theme={null} client.refine_strategy( strategy_id=strategy_id, feedback="Also extract the product SKU and brand name" ) ``` **Problem**: Jobs that worked before now fail or return incorrect data **Cause**: Website HTML structure changed **Solutions**: 1. Generate a new strategy for the updated site 2. Update your jobs to use the new strategy 3. Delete the old strategy **Problem**: Meter detected an API but returns no data **Possible causes**: * The API requires authentication that expired * The site changed its API endpoints * Rate limiting is blocking requests **Solutions**: * Generate a fresh strategy to capture new authentication tokens * If the site has changed significantly, the strategy may need regeneration * For rate-limited sites, reduce scrape frequency ## Next steps Learn how to run scrapes using your strategies Automate scraping with recurring schedules Explore all strategy methods in the SDK View strategy endpoints in the REST API ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Strategy Groups Source: https://docs.meter.sh/concepts/strategy-groups Organize strategies into groups for bulk management, shared schedules, and unified output schemas # Strategy Groups A **strategy group** is an organizational layer above individual strategies. It lets you manage many strategies as a single unit — applying schedules, output schemas, and webhooks to all members at once. ## When to use strategy groups **Use strategy groups when:** * You monitor many URLs with similar extraction needs (e.g., 50 product pages) * You want a single schedule and webhook for all strategies in a batch * You need to apply or change an output schema across many strategies at once * You create strategies via [Bulk Upload](/guides/bulk-upload) and want to keep them organized **Use individual strategies when:** * You have a handful of unrelated strategies * Each strategy needs its own schedule or webhook configuration ## How they work ```mermaid theme={null} graph TD G[Strategy Group] --> S1[Strategy A] G --> S2[Strategy B] G --> S3[Strategy C] G --> |"Group Schedule"| SCH[Shared Schedule Config] G --> |"Group Schema"| OS[Shared Output Schema] SCH --> S1 SCH --> S2 SCH --> S3 OS --> S1 OS --> S2 OS --> S3 ``` A strategy group provides: * **Group-level scheduling** — Apply a single schedule configuration (interval or cron, webhook URL, webhook type) to every strategy in the group. Each strategy gets its own schedule instance, but they share the same configuration. * **Group-level output schemas** — Apply an [output schema](/concepts/output-schemas) to all strategies at once. Meter regenerates each strategy asynchronously to match the new schema. * **Bulk management** — Enable/disable all schedules, delete all schedules, or test webhooks for the whole group in one call. ## Creating and managing groups ### Create a group ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") group = client.create_strategy_group( name="E-commerce Monitors", description="Product price tracking across 50 stores" ) print(f"Group ID: {group['id']}") ``` ```bash theme={null} curl -X POST https://api.meter.sh/api/strategy-groups \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "name": "E-commerce Monitors", "description": "Product price tracking across 50 stores" }' ``` ### Add strategies to a group ```python theme={null} client.add_strategies_to_group( group_id=group["id"], strategy_ids=[ "550e8400-e29b-41d4-a716-446655440000", "660e8400-e29b-41d4-a716-446655440000" ] ) ``` ```bash theme={null} curl -X POST https://api.meter.sh/api/strategy-groups/{group_id}/strategies \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_ids": [ "550e8400-e29b-41d4-a716-446655440000", "660e8400-e29b-41d4-a716-446655440000" ] }' ``` ### Remove a strategy from a group Removing a strategy from a group does not delete the strategy — it just becomes ungrouped. ```python theme={null} client.remove_strategy_from_group( group_id=group["id"], strategy_id="550e8400-e29b-41d4-a716-446655440000" ) ``` ```bash theme={null} curl -X DELETE https://api.meter.sh/api/strategy-groups/{group_id}/strategies/{strategy_id} \ -H "Authorization: Bearer sk_live_..." ``` ### List and inspect groups ```python theme={null} # List all groups groups = client.list_strategy_groups() for g in groups: print(f"{g['name']}: {g['strategy_count']} strategies") # Get group details with member strategies detail = client.get_strategy_group(group["id"]) for strategy in detail["strategies"]: print(f" - {strategy['name']}") ``` ```bash theme={null} # List groups curl https://api.meter.sh/api/strategy-groups \ -H "Authorization: Bearer sk_live_..." # Get group detail curl https://api.meter.sh/api/strategy-groups/{group_id} \ -H "Authorization: Bearer sk_live_..." ``` ## Group scheduling Apply a schedule to every strategy in the group with a single call. Each strategy gets its own schedule instance with the same configuration. ```python theme={null} # Schedule all strategies to run every hour with a webhook client.apply_group_schedule( group_id=group["id"], interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter" ) # Or use a cron expression client.apply_group_schedule( group_id=group["id"], cron_expression="0 */6 * * *", # Every 6 hours webhook_url="https://hooks.slack.com/services/T.../B.../xxx" # webhook_type auto-detected as "slack" ) ``` ```bash theme={null} curl -X POST https://api.meter.sh/api/strategy-groups/{group_id}/schedule \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "interval_seconds": 3600, "webhook_url": "https://your-app.com/webhooks/meter" }' ``` ### Manage group schedules ```python theme={null} # Pause all schedules in the group client.toggle_group_schedules(group["id"], enabled=False) # Resume all schedules client.toggle_group_schedules(group["id"], enabled=True) # Delete all schedules in the group client.delete_group_schedules(group["id"]) ``` Provide either `interval_seconds` or `cron_expression`, not both. The minimum interval is 60 seconds. ## Group output schemas Apply an [output schema](/concepts/output-schemas) to every strategy in a group. Meter regenerates each strategy asynchronously to match the schema structure. ```python theme={null} # Apply a product schema to all strategies in the group client.apply_group_schema( group_id=group["id"], output_schema={ "product_name": "string", "price": "number", "in_stock": "boolean", "image_url": "string" } ) # Poll for regeneration progress progress = client.get_schema_progress(group["id"]) print(progress) ``` See [Output Schemas](/concepts/output-schemas) for details on schema definition and type support. ## Deleting a group Deleting a group does **not** delete the strategies inside it. They become ungrouped. ```python theme={null} client.delete_strategy_group(group["id"]) ``` ```bash theme={null} curl -X DELETE https://api.meter.sh/api/strategy-groups/{group_id} \ -H "Authorization: Bearer sk_live_..." ``` ## Next steps Define the exact JSON structure for extraction results Create many strategies at once with bulk upload Full REST API reference for strategy groups Strategy group methods in the Python SDK ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Workflows Source: https://docs.meter.sh/concepts/workflows Chain scraping strategies into DAG-based pipelines # Workflows A **workflow** chains multiple scraping strategies into a directed acyclic graph (DAG), where the output of one scraper feeds into the next. This lets you build multi-step pipelines like "scrape an index page for links, then scrape each detail page." ## When to use workflows **Use workflows when:** * You need to scrape pages discovered by a previous scrape * Data extraction requires multiple stages (index -> detail -> sub-detail) * You want to filter results between stages * Different pages need different extraction strategies **Use simple schedules instead when:** * You're scraping a single URL or static list of URLs * All pages use the same extraction strategy * There's no dependency between scrapes ## Key concepts ### Nodes Each node in a workflow represents a scraping operation using a specific strategy. Nodes have one of three **input types**: | Input Type | Description | Use Case | | --------------- | ------------------------------------------------ | --------------------------------------------------------- | | `static_urls` | Fixed list of URLs | Starting nodes (index pages, sitemaps) | | `upstream_urls` | Extract URLs from a field in upstream results | Following links to detail pages | | `upstream_data` | Pass full upstream results as context | Parameter mapping between stages | | `trigger_only` | No input data — triggered by upstream completion | Nodes that run independently after a dependency completes | ### Edges Edges connect nodes and define data flow. Each edge goes from a source node to a target node. Edges can optionally include **filters** that control which upstream results are passed downstream. ### Filters Filters let you selectively pass data between nodes. For example, only follow links that contain "article" or skip items where the price is below a threshold. See [Post-Extraction Filtering](/concepts/filtering) for the full operator reference. ```python theme={null} from meter_sdk.workflow import Filter # Single condition Filter.contains("url", "/article/") # Combine with AND Filter.all( Filter.contains("category", "electronics"), Filter.gt("price", "100") ) # Combine with OR Filter.any( Filter.contains("url", "/sale/"), Filter.contains("url", "/clearance/") ) ``` Available filter operators: | Method | Description | | ------------------------------------ | -------------------------------------- | | `Filter.contains(field, value)` | Field contains substring | | `Filter.not_contains(field, value)` | Field does not contain substring | | `Filter.equals(field, value)` | Exact match | | `Filter.not_equals(field, value)` | Not exact match | | `Filter.regex_match(field, pattern)` | Regex match | | `Filter.exists(field)` | Field exists and is non-empty | | `Filter.not_exists(field)` | Field is missing or empty | | `Filter.gt(field, value)` | Greater than | | `Filter.lt(field, value)` | Less than | | `Filter.all(*conditions)` | AND — all conditions must match | | `Filter.any(*conditions)` | OR — at least one condition must match | All string operators accept an optional `case_sensitive` parameter (default: `False`). ## How workflows execute ```mermaid theme={null} graph TD A[Create Workflow] --> B[Run Workflow] B --> C[Execute Root Nodes] C --> D[Filter Results] D --> E[Pass URLs/Data to Downstream Nodes] E --> F[Execute Downstream Nodes] F --> G{More Downstream?} G -->|Yes| D G -->|No| H[Collect Final Results] ``` 1. **Root nodes** execute first using their static URLs 2. Results flow through edges, optionally filtered 3. **Downstream nodes** receive URLs or data from upstream 4. This continues until all leaf nodes complete 5. Final results are collected from **leaf nodes**, grouped by URL ## Building workflows ### Basic chain (A -> B) Scrape an index page, then follow each link to a detail page: ```python theme={null} from meter_sdk import MeterClient from meter_sdk.workflow import Workflow client = MeterClient(api_key="sk_live_...") # Build the workflow workflow = Workflow("Job Scraper") # Start node: scrape the index page index = workflow.start("index", index_strategy_id, urls=["https://jobs.com/listings"]) # Chain: scrape each job's detail page details = index.then("details", detail_strategy_id, url_field="job_url") # Run it run = client.run_workflow(workflow) # Get results (grouped by URL, then by strategy) output = client.get_workflow_output(run["workflow_id"]) for url, strategies in output["final_results_by_url_grouped"].items(): for strategy, items in strategies.items(): print(f"{strategy}: {len(items)} items") ``` ### Fan-out (A -> B, C, D) One source feeding multiple downstream scrapers: ```python theme={null} workflow = Workflow("Multi-Extractor") index = workflow.start("index", index_strategy_id, urls=["https://shop.com"]) # Fan out to different detail strategies prices = index.then("prices", price_strategy_id, url_field="product_url") reviews = index.then("reviews", review_strategy_id, url_field="product_url") images = index.then("images", image_strategy_id, url_field="product_url") run = client.run_workflow(workflow) output = client.get_workflow_output(run["workflow_id"]) ``` ### Filtered pipeline Only follow links that match a condition: ```python theme={null} from meter_sdk.workflow import Workflow, Filter workflow = Workflow("News Pipeline") index = workflow.start("index", index_strategy_id, urls=["https://news.com"]) # Only scrape articles in the technology section tech_articles = index.then( "tech_articles", article_strategy_id, url_field="link", filter=Filter.contains("category", "technology") ) run = client.run_workflow(workflow) output = client.get_workflow_output(run["workflow_id"]) ``` ### Multi-stage chain (A -> B -> C) ```python theme={null} workflow = Workflow("Deep Scraper") sitemap = workflow.start("sitemap", sitemap_strategy_id, urls=["https://shop.com/sitemap"]) categories = sitemap.then("categories", category_strategy_id, url_field="category_url") products = categories.then("products", product_strategy_id, url_field="product_url") run = client.run_workflow(workflow) output = client.get_workflow_output(run["workflow_id"]) ``` ## Change detection Workflows support change detection through two mechanisms: * **`trigger_on_change_only`**: When set on an edge, downstream nodes only execute if the upstream results have changed since the last run * **`force`**: When running a workflow with `force=True`, change detection is skipped and all nodes re-execute ```python theme={null} # Normal run — uses change detection result = client.run_workflow(workflow_id) # Force re-run — skip change detection result = client.run_workflow(workflow_id, force=True) ``` ## Scheduling workflows Workflows can be scheduled to run automatically, just like single-strategy schedules: ```python theme={null} # Run every hour client.schedule_workflow( workflow_id, interval_seconds=3600 ) # Run daily at 9 AM with webhook client.schedule_workflow( workflow_id, cron_expression="0 9 * * *", webhook_url="https://your-app.com/webhook" ) ``` See [Workflow SDK Reference](/api-reference/python/workflows) for all scheduling methods. ## Parameters Nodes can pass parameters to their strategies: ```python theme={null} # Static parameters on a start node index = workflow.start( "index", api_strategy_id, urls=["https://api.example.com/items"], parameters={"page": 1, "limit": 100} ) # Parameter config for downstream nodes (map upstream fields to parameters) details = index.then( "details", detail_strategy_id, url_field="detail_url", parameter_config={"item_id": "$.id"} ) ``` ## Next steps Complete workflow class and method documentation Workflow endpoints in the REST API Learn about the extraction strategies workflows use Compare with simple scheduled scrapes ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Bulk Upload Source: https://docs.meter.sh/guides/bulk-upload Create multiple strategies at once by uploading a list of URLs # Bulk Upload Bulk Upload lets you generate extraction strategies for many URLs in a single batch. Upload a CSV file, a TXT file, or paste URLs directly — Meter processes them in parallel and groups the resulting strategies together. ## Overview Instead of creating strategies one at a time, Bulk Upload lets you: 1. Provide a list of URLs (tens or hundreds) 2. Define what to extract (description, output schema, filters) 3. Optionally enable auto-scheduling with webhooks 4. Let Meter generate strategies for each URL in parallel All generated strategies are automatically placed in a [strategy group](/concepts/strategy-groups) for easy management. ## Input methods ### CSV file Upload a `.csv` file with URLs. Meter looks for a column named `url`, `link`, or `website` (case-insensitive). If none is found, it uses the first column. ```csv theme={null} url,name https://shop-a.com/products,Shop A https://shop-b.com/products,Shop B https://shop-c.com/products,Shop C ``` ### TXT file Upload a `.txt` file with one URL per line: ```text theme={null} https://shop-a.com/products https://shop-b.com/products https://shop-c.com/products ``` ### Paste URLs Paste URLs directly into the text area, one per line. URLs are automatically deduplicated, and `https://` is added if no protocol is present. ## Configuration options ### Strategy template Choose how strategies are generated: * **New template** — Define a strategy name, description, and optionally an output schema and filter. Each generated strategy is named `"{Name} - domain.com"`. * **Existing template** — Select an existing strategy as a template. The description and output schema from the selected strategy are reused. ### Output schema Define the exact JSON structure for extraction results. See [Output Schemas](/concepts/output-schemas) for details. ```json theme={null} { "product_name": "string", "price": "number", "in_stock": "boolean", "image_url": "string" } ``` When an output schema is provided, all strategies in the batch produce results with the same field names and types. ### Result filter Apply a [post-extraction filter](/concepts/filtering) to all generated strategies. Only items matching the filter conditions are kept in results. ### Sub-URL discovery When enabled, Meter discovers the best matching sub-page for each URL before generating a strategy. This is useful when you provide homepages but need a specific page type (e.g., a product listing page within each site). ### Auto-scheduling Enable automatic scheduling to create recurring scrape schedules for every generated strategy: * **Interval** — Every 1, 2, 6, 12, or 24 hours, or weekly * **Webhook URL** — Optional. Receive results via webhook for every scheduled run * **Webhook type** — `standard`, `slack`, or auto-detected from URL * **Webhook secret** — Optional. Leave empty to auto-generate a `whsec_` secret shared across all schedules in the batch * **Webhook metadata** — Optional JSON object included in every webhook payload When a webhook secret is auto-generated, it is displayed **once** in the results step. Store it securely — it won't be shown again. ### Strategy groups All strategies generated in a bulk upload are automatically placed in a [strategy group](/concepts/strategy-groups). You can provide a custom group name, or one is generated from the strategy name. ## Processing * Up to 4 URLs are processed in parallel * If rate-limited (429 response), all workers pause for 60 seconds before retrying * You can cancel processing at any time — pending URLs are marked as cancelled * Free-tier accounts are limited to 10 total strategies ## Results After processing, you can see: * Success and failure counts for each URL * Links to individual strategies and the strategy group * Error messages for failed URLs * A **Retry Failed** button to re-process only the URLs that failed ## Step-by-step walkthrough Navigate to the dashboard and click **Bulk Upload** (or the upload icon). Select **New template** and enter a strategy name and description, or select **Existing template** to reuse an existing strategy's configuration. Optionally enable output schema, result filter, sub-URL discovery, and auto-scheduling. Upload a CSV/TXT file or paste URLs directly. Review the parsed URL count and preview. Click **Start** and watch progress as strategies are generated for each URL. Check successes and failures. Retry any failed URLs. Navigate to the strategy group to manage all strategies together. ## Next steps Manage bulk-uploaded strategies as a group Define consistent output structure Filter extraction results Handle webhook notifications from scheduled strategies ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Change Detection Workflow Source: https://docs.meter.sh/guides/change-detection Build custom change detection logic on top of Meter # Change Detection Workflow Learn how to build custom change detection workflows using Meter's job comparison and history features. ## Overview While Meter's schedules include automatic change detection, you can build custom workflows for specific needs using job comparison APIs. **Use custom change detection when:** * Need custom change thresholds * Comparing non-adjacent jobs * Building complex diff logic * Generating change reports ## Basic comparison ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") # Get last two jobs for a strategy jobs = client.list_jobs(strategy_id="your-strategy-id", limit=2) if len(jobs) >= 2: # Compare latest with previous comparison = client.compare_jobs(jobs[0]['job_id'], jobs[1]['job_id']) print(f"Content hash match: {comparison['content_hash_match']}") print(f"Structural match: {comparison['structural_match']}") if not comparison['content_hash_match']: print("Content has changed!") for change in comparison.get('changes', []): print(f" - {change}") ``` ## Custom workflows ### Track specific field changes **TODO: Implement field-level change detection** ```python theme={null} def detect_price_changes(strategy_id): """Detect price changes specifically""" jobs = client.list_jobs(strategy_id=strategy_id, limit=2, status='completed') if len(jobs) < 2: return [] old_results = jobs[1]['results'] new_results = jobs[0]['results'] changes = [] # TODO: Implement comparison logic # for old, new in zip(old_results, new_results): # if old.get('price') != new.get('price'): # changes.append({ # 'product': new.get('name'), # 'old_price': old.get('price'), # 'new_price': new.get('price') # }) return changes ``` ### Generate change reports **TODO: Create change reporting** ```python theme={null} def generate_change_report(strategy_id, days=7): """Generate report of changes over time""" history = client.get_strategy_history(strategy_id) report = { "total_jobs": len(history), "jobs_with_changes": sum(1 for j in history if j['has_changes']), "change_rate": 0, "recent_changes": [] } # TODO: Implement reporting logic # Calculate change rate, find patterns, etc. return report ``` ### Alert on specific changes **TODO: Implement alerting logic** ```python theme={null} def monitor_with_alerts(strategy_id, alert_threshold=10): """Monitor and alert on significant changes""" comparison = client.compare_jobs(job_id_1, job_id_2) if not comparison['content_hash_match']: # TODO: Implement alert logic based on change magnitude # if change_magnitude > alert_threshold: # send_alert(...) pass ``` ## See also * [Change Detection Concept](/concepts/change-detection) - How Meter detects changes * [Job Comparison API](/api-reference/python/jobs#compare_jobs) - API reference * [Pull-Based Monitoring](/guides/pull-based-monitoring) - Polling for changes ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Content Change Monitoring Source: https://docs.meter.sh/guides/examples/content-monitoring Monitor pages for content changes and get notified via webhooks # Content Change Monitoring Monitor pages for content changes and get notified when something changes — no CSS selectors or descriptions needed. `extract_content()` reads a page's content and breaks it into sections. Schedule it on any number of URLs and Meter will notify you via webhook whenever content changes, telling you exactly which sections were added, modified, or removed. ## Step 1: Create a content strategy ```python theme={null} from meter_sdk import MeterClient import os client = MeterClient(api_key=os.getenv("METER_API_KEY")) strategy = client.extract_content( url="https://docs.example.com/api/overview", name="Example API Docs" ) strategy_id = strategy["strategy_id"] print(f"Strategy: {strategy_id}") print(f"Sections found: {len(strategy['preview_data'])}") for section in strategy["preview_data"]: print(f" [{section['level']}] {section['heading']}") ``` `extract_content()` is instant — no LLM call, no waiting. It returns in \~2 seconds. ## Step 2: Schedule monitoring with a webhook One strategy can monitor many URLs. Create a schedule with all the URLs you want to track: ```python theme={null} schedule = client.create_schedule( strategy_id=strategy_id, urls=[ "https://docs.example.com/api/overview", "https://docs.example.com/api/authentication", "https://docs.example.com/api/endpoints", "https://docs.example.com/api/errors", ], interval_seconds=3600, # check hourly webhook_url="https://your-app.com/webhooks/content-changes", ) print(f"Schedule: {schedule['schedule_id']}") print(f"Next run: {schedule['next_run_at']}") # Store the webhook secret (shown only once) if schedule.get("webhook_secret"): print(f"Webhook secret: {schedule['webhook_secret']}") ``` ## Step 3: Monitor many URLs at scale For monitoring hundreds of URLs, create the strategy once and attach all URLs to the schedule: ```python theme={null} # Your list of URLs to monitor urls = [ "https://docs.example.com/api/overview", "https://docs.example.com/api/authentication", "https://docs.example.com/guides/quickstart", # ... hundreds more ] # One strategy, one schedule, many URLs strategy = client.extract_content( url=urls[0], # any representative URL name="Example Docs Monitor" ) schedule = client.create_schedule( strategy_id=strategy["strategy_id"], urls=urls, cron_expression="0 */6 * * *", # every 6 hours webhook_url="https://your-app.com/webhooks/content-changes", webhook_metadata={"source": "example-docs", "team": "content"} ) ``` Use `webhook_metadata` to tag schedules. The metadata is included in every webhook payload, making it easy to route changes in your handler. ## Step 4: Handle webhook notifications When content changes, Meter sends a webhook with the changed sections. Here's a FastAPI handler: ```python theme={null} from fastapi import FastAPI, Request, HTTPException import hmac import hashlib app = FastAPI() WEBHOOK_SECRET = "your-webhook-secret" @app.post("/webhooks/content-changes") async def handle_content_change(request: Request): # Verify signature body = await request.body() signature = request.headers.get("X-Webhook-Secret") expected = hmac.new( WEBHOOK_SECRET.encode(), body, hashlib.sha256 ).hexdigest() if not hmac.compare_digest(signature or "", expected): raise HTTPException(status_code=401) payload = await request.json() url = payload["url"] metadata = payload.get("metadata", {}) new_items = payload.get("new_items", []) removed_items = payload.get("removed_items", []) print(f"Content changed on {url}") print(f" {len(new_items)} sections added/modified") print(f" {len(removed_items)} sections removed") for section in new_items: print(f" Changed: {section['heading']}") # Trigger your downstream action: # - Re-scrape with Firecrawl # - Update RAG embeddings # - Notify your team return {"status": "ok"} ``` ## Step 5: Pull-based alternative If you prefer polling over webhooks, use `get_schedule_changes()`: ```python theme={null} changes = client.get_schedule_changes( schedule_id=schedule["schedule_id"], mark_seen=True ) if changes["count"] > 0: for change in changes["changes"]: url = change["url"] sections = change["results"] print(f"{url}: {len(sections)} sections extracted") ``` Or view the full change history: ```python theme={null} changelog = client.get_schedule_changelog( schedule_id=schedule["schedule_id"], limit=10 ) for entry in changelog["entries"]: print(f"{entry['completed_at']} — {entry['url']}") print(f" +{entry['new_items_count']} / -{entry['removed_items_count']}") ``` ## Complete example End-to-end script that sets up monitoring for a list of URLs: ```python theme={null} """content_monitor.py — Monitor pages for content changes.""" from meter_sdk import MeterClient import os import json client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Configuration URLS = [ "https://docs.example.com/api/overview", "https://docs.example.com/api/authentication", "https://docs.example.com/api/endpoints", "https://docs.example.com/guides/quickstart", "https://docs.example.com/guides/deployment", ] WEBHOOK_URL = "https://your-app.com/webhooks/content-changes" STATE_FILE = "monitor_state.json" def load_state(): try: with open(STATE_FILE) as f: return json.load(f) except FileNotFoundError: return {} def save_state(state): with open(STATE_FILE, "w") as f: json.dump(state, f, indent=2) def setup(): """One-time setup: create strategy + schedule.""" state = load_state() if "strategy_id" not in state: strategy = client.extract_content( url=URLS[0], name="Docs Content Monitor" ) state["strategy_id"] = strategy["strategy_id"] print(f"Created strategy: {state['strategy_id']}") print(f"Preview sections: {len(strategy['preview_data'])}") if "schedule_id" not in state: schedule = client.create_schedule( strategy_id=state["strategy_id"], urls=URLS, interval_seconds=3600, webhook_url=WEBHOOK_URL, webhook_metadata={"monitor": "docs-content"}, ) state["schedule_id"] = schedule["schedule_id"] state["webhook_secret"] = schedule.get("webhook_secret") print(f"Created schedule: {state['schedule_id']}") print(f"Webhook secret: {state['webhook_secret']}") save_state(state) print(f"\nMonitoring {len(URLS)} URLs. Webhook → {WEBHOOK_URL}") return state if __name__ == "__main__": setup() ``` Run once to set up: ```bash theme={null} export METER_API_KEY="sk_live_..." python content_monitor.py ``` That's it — Meter handles the recurring checks, diffing, and webhook delivery from here. ## See also How Meter detects and reports changes Webhook payload formats and verification Full schedule API reference Organize monitors with groups ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # E-commerce Price Monitoring Source: https://docs.meter.sh/guides/examples/ecommerce-monitoring Monitor product prices and availability with Meter # E-commerce Price Monitoring Complete example showing how to monitor e-commerce products for price changes and stock availability. ## Use case Track competitor prices, product availability, and detect when products go on sale. ## What you'll build * Strategy for extracting product data * Hourly monitoring schedule * Price drop alerts * Inventory tracking ## Prerequisites * Meter API key * **TODO**: Your alerting system (email, Slack, etc.) * **TODO**: Your target e-commerce site ## Step 1: Generate strategy **TODO**: Replace with your actual e-commerce site and data ```python theme={null} from meter_sdk import MeterClient import os client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Generate strategy strategy = client.generate_strategy( url="https://example-shop.com/products", # TODO: Your URL description="Extract product name, price, image URL, and stock availability", name="E-commerce Product Monitor" ) strategy_id = strategy["strategy_id"] print(f"Strategy ID: {strategy_id}") # Check preview print("\nPreview data:") for item in strategy['preview_data'][:3]: print(f" {item}") ``` ## Step 2: Set up monitoring ```python theme={null} # Monitor every hour schedule = client.create_schedule( strategy_id=strategy_id, url="https://example-shop.com/products", # TODO: Your URL interval_seconds=3600 # Every hour ) print(f"Schedule ID: {schedule['schedule_id']}") ``` ## Step 3: Process price changes **TODO**: Implement your price tracking logic ```python theme={null} def process_price_changes(schedule_id, price_threshold=50.0): """Check for price changes and alert on drops""" changes = client.get_schedule_changes(schedule_id, mark_seen=True) if changes['count'] == 0: print("No changes detected") return print(f"Processing {changes['count']} changes") for change in changes['changes']: for product in change['results']: # TODO: Parse price (format depends on your site) # price = float(product['price'].replace('$', '')) # TODO: Implement price tracking # old_price = get_last_known_price(product['name']) # if price < old_price * 0.9: # 10% drop # send_price_drop_alert(product['name'], old_price, price) # TODO: Store current price # store_price(product['name'], price) pass # Run periodically import time while True: process_price_changes(schedule['schedule_id']) time.sleep(3600) ``` ## Step 4: Track inventory **TODO**: Implement stock tracking ```python theme={null} def track_inventory(schedule_id): """Alert when out-of-stock items come back""" changes = client.get_schedule_changes(schedule_id, mark_seen=True) for change in changes['changes']: for product in change['results']: # TODO: Check stock status # in_stock = product.get('stock_availability') == 'In Stock' # was_out_of_stock = check_previous_stock_status(product['name']) # if in_stock and was_out_of_stock: # send_back_in_stock_alert(product['name']) # TODO: Update stock status # update_stock_status(product['name'], in_stock) pass ``` ## Complete example **TODO**: Full working implementation ```python theme={null} # full_example.py from meter_sdk import MeterClient import os import time import json from datetime import datetime client = MeterClient(api_key=os.getenv("METER_API_KEY")) # TODO: Add your configuration CONFIG = { "url": "https://example-shop.com/products", "price_threshold": 50.0, # Alert on prices below this "check_interval": 3600, # Check every hour "price_drop_percent": 10 # Alert on 10%+ price drops } # TODO: Implement price tracking storage PRICE_HISTORY_FILE = "price_history.json" def load_price_history(): """Load price history from file""" try: with open(PRICE_HISTORY_FILE, 'r') as f: return json.load(f) except FileNotFoundError: return {} def save_price_history(history): """Save price history to file""" with open(PRICE_HISTORY_FILE, 'w') as f: json.dump(history, f, indent=2) def send_alert(message): """Send alert (TODO: implement your alerting)""" print(f"ALERT: {message}") # TODO: Send email, Slack notification, etc. def main(): """Main monitoring loop""" # Generate or load strategy # TODO: Store strategy_id in config file strategy_id = "your-strategy-id" # Replace with actual # Create or load schedule # TODO: Store schedule_id in config file schedule_id = "your-schedule-id" # Replace with actual price_history = load_price_history() while True: print(f"\n[{datetime.now()}] Checking for changes...") changes = client.get_schedule_changes(schedule_id, mark_seen=True) if changes['count'] > 0: print(f"Processing {changes['count']} changes") for change in changes['changes']: for product in change['results']: # TODO: Implement your processing logic product_name = product.get('name') # Example: Track prices # current_price = parse_price(product.get('price')) # old_price = price_history.get(product_name) # if old_price and current_price < old_price * (1 - CONFIG['price_drop_percent'] / 100): # send_alert(f"{product_name} price dropped: ${old_price} -> ${current_price}") # price_history[product_name] = current_price save_price_history(price_history) else: print("No changes detected") print(f"Waiting {CONFIG['check_interval']} seconds...") time.sleep(CONFIG['check_interval']) if __name__ == "__main__": main() ``` ## Deployment ### Option 1: Run as service **TODO**: Set up as systemd service or similar ```bash theme={null} # Run continuously python ecommerce_monitor.py ``` ### Option 2: Cron job **TODO**: Configure cron ```bash theme={null} # Add to crontab (check every hour) 0 * * * * cd /path/to/project && python ecommerce_monitor.py ``` ### Option 3: Cloud function **TODO**: Deploy to AWS Lambda, Google Cloud Functions, etc. ## Next steps * Add database for price history tracking * Implement email/Slack alerts * Build price trends visualization * Add multi-page monitoring ## See also * [Schedules Concept](/concepts/schedules) - Understanding monitoring * [Pull-Based Monitoring](/guides/pull-based-monitoring) - Polling patterns * [Webhooks Guide](/guides/webhooks) - Real-time alerts ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Job Board Scraping Source: https://docs.meter.sh/guides/examples/job-board-scraping Monitor job boards and track new postings automatically # Job Board Scraping Build a job board monitoring system that tracks new postings and sends alerts. ## Use case Monitor job boards for new positions matching your criteria, alert candidates or recruiters, and track hiring trends. ## What you'll build * Strategy for job listing extraction * Regular monitoring (every 30 minutes) * New job detection * Alert system for matching positions ## Prerequisites * Meter API key * **TODO**: Alerting system (email, Slack, Discord, etc.) * **TODO**: Target job board URLs ## Step 1: Generate strategy **TODO**: Replace with your job board ```python theme={null} from meter_sdk import MeterClient import os client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Generate strategy for job board strategy = client.generate_strategy( url="https://jobs.example.com/search?q=software+engineer", # TODO: Your URL description="Extract job title, company name, location, and job URL. Ignore sponsored ads.", name="Software Engineer Jobs Monitor" ) strategy_id = strategy["strategy_id"] print(f"Strategy ID: {strategy_id}") # Preview jobs print("\nPreview of current postings:") for job in strategy['preview_data'][:5]: print(f" {job}") ``` ## Step 2: Set up frequent monitoring ```python theme={null} # Check every 30 minutes for new postings schedule = client.create_schedule( strategy_id=strategy_id, url="https://jobs.example.com/search?q=software+engineer", # TODO: Your URL interval_seconds=1800 # 30 minutes ) print(f"Schedule ID: {schedule['schedule_id']}") ``` ## Step 3: Detect and alert on new jobs **TODO**: Implement job tracking and alerts ```python theme={null} def track_new_jobs(schedule_id, job_database_file="jobs.json"): """Detect new jobs and send alerts""" import json # Load known jobs try: with open(job_database_file, 'r') as f: known_jobs = set(json.load(f)) except FileNotFoundError: known_jobs = set() # Check for changes changes = client.get_schedule_changes(schedule_id, mark_seen=True) new_jobs = [] if changes['count'] > 0: for change in changes['changes']: for job in change['results']: # TODO: Create unique job ID (depends on your data) # job_id = f"{job['company']}_{job['title']}" job_id = str(job) # Placeholder if job_id not in known_jobs: new_jobs.append(job) known_jobs.add(job_id) if len(new_jobs) > 0: print(f"Found {len(new_jobs)} new jobs!") for job in new_jobs: # TODO: Send alert # send_job_alert(job) print(f" NEW: {job}") # Save updated job database with open(job_database_file, 'w') as f: json.dump(list(known_jobs), f) return new_jobs # Monitor continuously import time while True: new_jobs = track_new_jobs(schedule['schedule_id']) if len(new_jobs) == 0: print("No new jobs") time.sleep(1800) # Wait 30 minutes ``` ## Step 4: Filter and match criteria **TODO**: Implement job filtering ```python theme={null} def filter_jobs(jobs, criteria): """Filter jobs by criteria""" matched = [] for job in jobs: # TODO: Implement matching logic # location_match = criteria['location'].lower() in job.get('location', '').lower() # keyword_match = any(kw.lower() in job.get('title', '').lower() for kw in criteria['keywords']) # if location_match and keyword_match: # matched.append(job) pass return matched # Define criteria CRITERIA = { "keywords": ["python", "backend", "api"], "location": "remote", "exclude": ["senior", "lead"] } # Filter new jobs new_jobs = track_new_jobs(schedule_id) matched_jobs = filter_jobs(new_jobs, CRITERIA) if len(matched_jobs) > 0: print(f"Found {len(matched_jobs)} matching jobs!") # send_alert(matched_jobs) ``` ## Complete example **TODO**: Full implementation ```python theme={null} # job_monitor.py from meter_sdk import MeterClient import os import time import json from datetime import datetime client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Configuration CONFIG = { "schedule_id": "your-schedule-id", # TODO: Replace "check_interval": 1800, # 30 minutes "criteria": { "keywords": ["python", "backend"], "location": "remote", "exclude": ["senior", "lead", "staff"] }, "alert_email": "your-email@example.com" # TODO: Replace } JOB_DB_FILE = "known_jobs.json" def load_known_jobs(): """Load known job IDs""" try: with open(JOB_DB_FILE, 'r') as f: return set(json.load(f)) except FileNotFoundError: return set() def save_known_jobs(known_jobs): """Save known job IDs""" with open(JOB_DB_FILE, 'w') as f: json.dump(list(known_jobs), f, indent=2) def create_job_id(job): """Create unique job identifier""" # TODO: Create stable ID based on your data # return f"{job['company']}_{job['title']}_{job.get('location', '')}" return str(hash(str(job))) # Placeholder def matches_criteria(job, criteria): """Check if job matches criteria""" # TODO: Implement matching logic # title = job.get('title', '').lower() # location = job.get('location', '').lower() # Keywords match # keyword_match = any(kw.lower() in title for kw in criteria['keywords']) # Location match # location_match = criteria['location'].lower() in location # Exclude terms # excluded = any(term.lower() in title for term in criteria['exclude']) # return keyword_match and location_match and not excluded return True # Placeholder def send_alert(jobs): """Send alert for new jobs""" # TODO: Implement alerting (email, Slack, etc.) print(f"\n🚨 ALERT: {len(jobs)} new matching jobs!") for job in jobs: print(f" - {job}") def monitor(): """Main monitoring loop""" known_jobs = load_known_jobs() while True: print(f"\n[{datetime.now()}] Checking for new jobs...") changes = client.get_schedule_changes( CONFIG['schedule_id'], mark_seen=True ) if changes['count'] == 0: print("No new job postings") else: print(f"Processing {changes['count']} updates") new_matching_jobs = [] for change in changes['changes']: for job in change['results']: job_id = create_job_id(job) if job_id not in known_jobs: known_jobs.add(job_id) if matches_criteria(job, CONFIG['criteria']): new_matching_jobs.append(job) if len(new_matching_jobs) > 0: send_alert(new_matching_jobs) save_known_jobs(known_jobs) else: print("No new jobs matching criteria") print(f"Waiting {CONFIG['check_interval']} seconds...") time.sleep(CONFIG['check_interval']) if __name__ == "__main__": monitor() ``` ## Multi-board monitoring **TODO**: Monitor multiple job boards ```python theme={null} JOB_BOARDS = [ { "name": "Indeed", "schedule_id": "schedule-1", "criteria": {...} }, { "name": "LinkedIn", "schedule_id": "schedule-2", "criteria": {...} }, # TODO: Add more boards ] def monitor_all_boards(): """Monitor multiple job boards""" for board in JOB_BOARDS: print(f"\nChecking {board['name']}...") # Check for new jobs... ``` ## Deployment **TODO**: Deploy your monitor * Run as systemd service * Use PM2 for Node.js * Deploy to cloud (AWS, Heroku, etc.) * Set up cron job ## Advanced features **TODO**: Extend functionality * Salary range tracking * Company research integration * Application tracking * Trend analysis (hiring surges, salary trends) ## See also * [Pull-Based Monitoring](/guides/pull-based-monitoring) - Polling patterns * [Webhooks Guide](/guides/webhooks) - Real-time alerts * [Schedules Concept](/concepts/schedules) - Monitoring schedules ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # News Aggregation Source: https://docs.meter.sh/guides/examples/news-aggregation Aggregate and monitor news articles across multiple sources # News Aggregation Build a news aggregation system that monitors multiple sources and detects new articles. ## Use case Aggregate news from multiple sources, detect new articles, and feed them into your knowledge base or RAG system. ## What you'll build * Strategies for multiple news sources * Hourly monitoring of all sources * New article detection * RAG/vector database integration ## Prerequisites * Meter API key * **TODO**: Vector database (Pinecone, Weaviate, etc.) * **TODO**: Embedding service (OpenAI, Cohere, etc.) ## Step 1: Generate strategies for news sources **TODO**: Add your news sources ```python theme={null} from meter_sdk import MeterClient import os client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Define news sources NEWS_SOURCES = [ { "name": "Tech News Site", "url": "https://technews.example.com", # TODO: Replace "description": "Extract article title, author, publish date, and excerpt" }, { "name": "Industry Blog", "url": "https://blog.example.com", # TODO: Replace "description": "Extract post title, author, date, and summary" }, # TODO: Add more sources ] # Generate strategies strategies = [] for source in NEWS_SOURCES: strategy = client.generate_strategy( url=source['url'], description=source['description'], name=source['name'] ) strategies.append({ "source": source['name'], "strategy_id": strategy['strategy_id'], "url": source['url'] }) print(f"Created strategy for {source['name']}: {strategy['strategy_id']}") # TODO: Save strategy IDs to config file ``` ## Step 2: Set up monitoring for all sources ```python theme={null} # Create schedules for each source schedules = [] for strategy in strategies: schedule = client.create_schedule( strategy_id=strategy['strategy_id'], url=strategy['url'], interval_seconds=3600 # Check hourly ) schedules.append({ "source": strategy['source'], "schedule_id": schedule['schedule_id'] }) print(f"Monitoring {strategy['source']}: {schedule['schedule_id']}") # TODO: Save schedule IDs to config file ``` ## Step 3: Aggregate new articles ```python theme={null} def aggregate_news(schedules): """Collect new articles from all sources""" all_articles = [] for schedule_info in schedules: changes = client.get_schedule_changes( schedule_info['schedule_id'], mark_seen=True ) if changes['count'] > 0: for change in changes['changes']: for article in change['results']: # Add source metadata article['source'] = schedule_info['source'] article['scraped_at'] = change['completed_at'] all_articles.append(article) return all_articles # Check all sources new_articles = aggregate_news(schedules) print(f"Found {len(new_articles)} new articles") ``` ## Step 4: Keyword filtering for specific topics Use keyword filters to only retrieve articles matching specific topics. This is useful when monitoring news for specific keywords relevant to your business. ### Filter syntax | Syntax | Meaning | Example | | ---------- | -------------- | ------------------------------- | | `+keyword` | Required (AND) | `+jfk +tariff` - must have both | | `keyword` | Optional (OR) | `jfk elon` - either matches | | `-keyword` | Excluded (NOT) | `-bitcoin` - exclude these | | `"phrase"` | Exact phrase | `"elon musk"` - exact match | ### Example: Monitor news for specific keywords ```python theme={null} def get_filtered_news(schedule_id, keywords): """Get only articles matching specific keywords""" changes = client.get_schedule_changes( schedule_id=schedule_id, filter=keywords, mark_seen=True ) articles = [] for change in changes['changes']: for article in change['results']: articles.append(article) return articles # Get articles about JFK AND tariffs tariff_news = get_filtered_news(schedule_id, "+jfk +tariff") print(f"Found {len(tariff_news)} articles about jfk and tariffs") # Get articles mentioning either Tesla OR SpaceX tech_news = get_filtered_news(schedule_id, "tesla spacex") print(f"Found {len(tech_news)} articles about Tesla or SpaceX") # Get crypto news but exclude Bitcoin altcoin_news = get_filtered_news(schedule_id, "+crypto -bitcoin") print(f"Found {len(altcoin_news)} altcoin articles") ``` ### Complete example: Multi-topic news monitor ```python theme={null} from meter_sdk import MeterClient import os client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Define topics to monitor with their keyword filters TOPICS = { "tariffs": "+jfk +tariff", "tech_earnings": "+earnings tesla apple nvidia", "crypto_regulation": "+crypto +regulation -bitcoin", "ai_news": '+ai +"artificial intelligence" +openai anthropic', } def monitor_topics(schedule_id): """Monitor multiple topics from a single news source""" results = {} for topic_name, filter_query in TOPICS.items(): # Get articles matching this topic # Note: Use mark_seen=False to allow the same articles # to match multiple topics changes = client.get_schedule_changes( schedule_id=schedule_id, filter=filter_query, mark_seen=False # Don't mark as seen yet ) articles = [] for change in changes['changes']: articles.extend(change['results']) results[topic_name] = articles print(f"{topic_name}: {len(articles)} articles") # Now mark all as seen client.get_schedule_changes(schedule_id=schedule_id, mark_seen=True) return results # Monitor all topics topic_results = monitor_topics(schedule_id) # Process results by topic for topic, articles in topic_results.items(): if articles: print(f"\n=== {topic.upper()} ===") for article in articles[:3]: # Show top 3 print(f" - {article.get('title', 'Untitled')}") ``` The filter applies to individual result items. If a job returns 50 articles but only 5 match your filter, you'll only receive those 5 matching articles. Jobs with zero matching items are excluded entirely. ## Step 5: Feed into RAG system **TODO**: Integrate with your vector database ```python theme={null} def update_vector_db(articles): """Add new articles to vector database""" for article in articles: # TODO: Generate embedding # embedding = openai.embeddings.create( # model="text-embedding-3-small", # input=f"{article['title']} {article.get('excerpt', '')}" # ).data[0].embedding # TODO: Upsert to vector database # pinecone_index.upsert([ # ( # article['url'], # ID # embedding, # { # "title": article['title'], # "source": article['source'], # "date": article.get('publish_date'), # "url": article['url'] # } # ) # ]) pass # Process new articles update_vector_db(new_articles) ``` ## Complete example **TODO**: Full implementation ```python theme={null} # news_aggregator.py from meter_sdk import MeterClient import os import time import json from datetime import datetime client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Load configuration CONFIG_FILE = "news_config.json" def load_config(): """Load news sources and schedule IDs""" try: with open(CONFIG_FILE, 'r') as f: return json.load(f) except FileNotFoundError: return {"sources": []} def save_config(config): """Save configuration""" with open(CONFIG_FILE, 'w') as f: json.dump(config, f, indent=2) def setup_monitoring(): """Setup strategies and schedules for all sources""" config = load_config() # TODO: Define your news sources NEWS_SOURCES = [ # Add your sources here ] for source in NEWS_SOURCES: # Generate strategy strategy = client.generate_strategy( url=source['url'], description=source['description'], name=source['name'] ) # Create schedule schedule = client.create_schedule( strategy_id=strategy['strategy_id'], url=source['url'], interval_seconds=3600 ) config['sources'].append({ "name": source['name'], "url": source['url'], "strategy_id": strategy['strategy_id'], "schedule_id": schedule['schedule_id'] }) save_config(config) return config def monitor_all_sources(): """Monitor all news sources and aggregate articles""" config = load_config() all_new_articles = [] for source in config['sources']: print(f"Checking {source['name']}...") changes = client.get_schedule_changes( source['schedule_id'], mark_seen=True ) if changes['count'] > 0: for change in changes['changes']: for article in change['results']: article['source'] = source['name'] article['source_url'] = source['url'] all_new_articles.append(article) print(f" Found {len(changes['changes'])} new articles") else: print(f" No new articles") return all_new_articles def main(): """Main aggregation loop""" # Setup (run once) # config = setup_monitoring() # Load existing config config = load_config() if len(config['sources']) == 0: print("No sources configured. Run setup_monitoring() first.") return print(f"Monitoring {len(config['sources'])} sources") while True: print(f"\n[{datetime.now()}] Checking all sources...") new_articles = monitor_all_sources() if len(new_articles) > 0: print(f"\nProcessing {len(new_articles)} new articles:") for article in new_articles: print(f" - [{article['source']}] {article.get('title', 'Untitled')}") # TODO: Update vector database # update_vector_db(new_articles) else: print("No new articles across all sources") print("\nWaiting 1 hour...") time.sleep(3600) if __name__ == "__main__": main() ``` ## Deployment **TODO**: Deploy your aggregator * Run as background service * Use cron for periodic checks * Deploy to cloud (AWS, GCP, etc.) ## Advanced features **TODO**: Extend functionality * Article deduplication across sources * Trend detection * Topic clustering * Sentiment analysis integration ## See also * [RAG Integration Guide](/guides/rag-integration) - Vector database integration * [Pull-Based Monitoring](/guides/pull-based-monitoring) - Polling patterns * [Schedules Concept](/concepts/schedules) - Understanding schedules ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Integration Guides Source: https://docs.meter.sh/guides/overview Learn how to integrate Meter into your application # Integration Guides These guides show you how to integrate Meter into real-world applications and workflows. ## Available guides Create multiple strategies at once by uploading a list of URLs Connect Meter to your vector database for automatic RAG updates Set up webhook endpoints to receive real-time change notifications Poll for changes on your own schedule using the changes API Build custom change detection logic on top of Meter ## Example projects See complete, working examples for common use cases: Monitor product prices and availability Aggregate and track news articles Track job listings and new postings ## What you'll learn Each guide covers: * Complete working code examples * Best practices and optimization tips * Error handling and edge cases * Production deployment considerations ## Prerequisites Most guides assume you have: * A Meter account and API key * Python 3.8+ or Node.js 16+ * Basic understanding of Meter [core concepts](/concepts/overview) ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Pull-Based Monitoring Guide Source: https://docs.meter.sh/guides/pull-based-monitoring Poll for content changes on your own schedule # Pull-Based Monitoring Guide Learn how to use Meter's changes API to poll for updates on your own schedule, giving you control over when and how changes are processed. ## Overview Pull-based monitoring uses the `get_schedule_changes()` API to check for changes on demand, rather than receiving webhooks. **Use pull-based when:** * Batch processing changes (e.g., once per hour) * Webhooks aren't feasible (firewall restrictions, no public endpoint) * You need manual control over processing timing * Building admin dashboards or reporting tools **Use webhooks instead when:** * Changes need immediate action * Building real-time systems ## How it works ```mermaid theme={null} graph LR A[Your App] -->|Check for changes| B[Meter API] B -->|Return changed jobs| A A -->|Process changes| C[Update Database] A -->|Mark as seen| B ``` ## Basic implementation ```python theme={null} from meter_sdk import MeterClient import time client = MeterClient(api_key="sk_live_...") # TODO: Replace with your schedule ID schedule_id = "your-schedule-id" while True: # Check for changes changes = client.get_schedule_changes( schedule_id=schedule_id, mark_seen=True # Mark as seen after reading ) if changes['count'] > 0: print(f"Processing {changes['count']} changes") for change in changes['changes']: # TODO: Add your processing logic print(f"Job {change['job_id']}: {change['item_count']} items") # Process change['results']... else: print("No changes detected") # Wait before next check time.sleep(3600) # Check every hour ``` ## REST API implementation Use the REST API directly if you're not using Python or prefer HTTP calls. ### Endpoint ```http theme={null} GET /api/schedules/{schedule_id}/changes?mark_seen=true ``` ### Basic example (curl) ```bash theme={null} #!/bin/bash SCHEDULE_ID="your-schedule-id" API_KEY="sk_live_..." while true; do echo "Checking for changes..." # Get changes response=$(curl -s https://api.meter.sh/api/schedules/$SCHEDULE_ID/changes \ -H "Authorization: Bearer $API_KEY") # Parse count (requires jq) count=$(echo $response | jq -r '.count') if [ "$count" -gt 0 ]; then echo "Processing $count changes" echo $response | jq '.changes' # TODO: Process changes else echo "No changes detected" fi # Wait before next check sleep 3600 # 1 hour done ``` ### JavaScript/Node.js example ```javascript theme={null} const SCHEDULE_ID = 'your-schedule-id'; const API_KEY = 'sk_live_...'; async function checkForChanges() { const response = await fetch( `https://api.meter.sh/api/schedules/${SCHEDULE_ID}/changes`, { headers: { 'Authorization': `Bearer ${API_KEY}` } } ); const data = await response.json(); if (data.count > 0) { console.log(`Processing ${data.count} changes`); for (const change of data.changes) { console.log(`Job ${change.job_id}: ${change.item_count} items`); // TODO: Process change.results await processChange(change); } } else { console.log('No changes detected'); } } // Poll every hour setInterval(checkForChanges, 3600000); // Run immediately on start checkForChanges(); ``` ### Python (requests) example ```python theme={null} import requests import time SCHEDULE_ID = "your-schedule-id" API_KEY = "sk_live_..." def check_for_changes(): response = requests.get( f"https://api.meter.sh/api/schedules/{SCHEDULE_ID}/changes", headers={"Authorization": f"Bearer {API_KEY}"} ) data = response.json() if data['count'] > 0: print(f"Processing {data['count']} changes") for change in data['changes']: print(f"Job {change['job_id']}: {change['item_count']} items") # TODO: Process change['results'] process_change(change) else: print("No changes detected") # Poll every hour while True: check_for_changes() time.sleep(3600) ``` ### Go example ```go theme={null} package main import ( "encoding/json" "fmt" "io" "net/http" "time" ) const ( scheduleID = "your-schedule-id" apiKey = "sk_live_..." ) type ChangesResponse struct { ScheduleID string `json:"schedule_id"` Changes []Change `json:"changes"` Count int `json:"count"` MarkedSeen bool `json:"marked_seen"` } type Change struct { JobID string `json:"job_id"` Status string `json:"status"` Results []interface{} `json:"results"` ItemCount int `json:"item_count"` ContentHash string `json:"content_hash"` CompletedAt string `json:"completed_at"` Seen bool `json:"seen"` } func checkForChanges() error { url := fmt.Sprintf("https://api.meter.sh/api/schedules/%s/changes", scheduleID) req, err := http.NewRequest("GET", url, nil) if err != nil { return err } req.Header.Set("Authorization", "Bearer "+apiKey) client := &http.Client{} resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { return err } var changesResp ChangesResponse if err := json.Unmarshal(body, &changesResp); err != nil { return err } if changesResp.Count > 0 { fmt.Printf("Processing %d changes\n", changesResp.Count) for _, change := range changesResp.Changes { fmt.Printf("Job %s: %d items\n", change.JobID, change.ItemCount) // TODO: Process change.Results } } else { fmt.Println("No changes detected") } return nil } func main() { ticker := time.NewTicker(1 * time.Hour) defer ticker.Stop() // Run immediately checkForChanges() // Then poll every hour for range ticker.C { if err := checkForChanges(); err != nil { fmt.Printf("Error checking for changes: %v\n", err) } } } ``` ### mark\_seen parameter Control whether changes are marked as seen: ```bash theme={null} # Mark as seen (default) curl https://api.meter.sh/api/schedules/$SCHEDULE_ID/changes \ -H "Authorization: Bearer $API_KEY" # Preview without marking as seen curl "https://api.meter.sh/api/schedules/$SCHEDULE_ID/changes?mark_seen=false" \ -H "Authorization: Bearer $API_KEY" ``` ```javascript theme={null} // Preview without marking as seen const response = await fetch( `https://api.meter.sh/api/schedules/${SCHEDULE_ID}/changes?mark_seen=false`, { headers: { 'Authorization': `Bearer ${API_KEY}` } } ); ``` ### filter parameter Filter results by keywords using Lucene-style syntax: ```bash theme={null} # Filter for items containing both "jfk" AND "tariff" curl "https://api.meter.sh/api/schedules/$SCHEDULE_ID/changes?filter=%2Bjfk+%2Btariff" \ -H "Authorization: Bearer $API_KEY" # Filter for items containing "jfk" OR "elon" curl "https://api.meter.sh/api/schedules/$SCHEDULE_ID/changes?filter=jfk+elon" \ -H "Authorization: Bearer $API_KEY" ``` ```python theme={null} # Filter for articles about specific topics changes = client.get_schedule_changes( schedule_id=schedule_id, filter="+jfk +tariff", # Both required mark_seen=True ) ``` | Syntax | Meaning | Example | | ---------- | -------------- | -------------- | | `+keyword` | Required (AND) | `+jfk +tariff` | | `keyword` | Optional (OR) | `jfk elon` | | `-keyword` | Excluded (NOT) | `-bitcoin` | | `"phrase"` | Exact phrase | `"elon musk"` | Filters apply to individual result items, not entire jobs. Only matching items are returned. Jobs with zero matches are excluded. ### Response format ```json theme={null} { "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "changes": [ { "job_id": "660e8400-e29b-41d4-a716-446655440000", "status": "completed", "results": [ { "title": "Product Name", "price": "$29.99", "url": "https://example.com/product/123" } ], "item_count": 12, "content_hash": "7f3d9a2b4c1e...", "completed_at": "2025-01-15T10:30:12Z", "seen": true } ], "count": 1, "marked_seen": true, "filter_applied": "+jfk +tariff" } ``` ## Key concepts ### mark\_seen parameter ```python theme={null} # Mark changes as seen (default) changes = client.get_schedule_changes(schedule_id, mark_seen=True) # These changes won't be returned in future calls # Preview without marking as seen preview = client.get_schedule_changes(schedule_id, mark_seen=False) # These changes will be returned again next time ``` Use `mark_seen=False` to: * Preview changes before processing * Debug change detection * Implement custom "seen" tracking ### Changes API returns only new changes The API only returns jobs with content changes that haven't been marked as seen: ```python theme={null} # First call changes = client.get_schedule_changes(schedule_id, mark_seen=True) print(changes['count']) # e.g., 3 changes # Second call (immediately after) changes = client.get_schedule_changes(schedule_id, mark_seen=True) print(changes['count']) # 0 (all marked as seen) # After new scrape runs and detects changes changes = client.get_schedule_changes(schedule_id, mark_seen=True) print(changes['count']) # New changes only ``` ## Polling patterns ### Fixed interval polling ```python theme={null} import time def poll_fixed_interval(schedule_id, interval_seconds): """Poll at fixed intervals""" while True: changes = client.get_schedule_changes(schedule_id, mark_seen=True) if changes['count'] > 0: # TODO: Process changes process_changes(changes['changes']) time.sleep(interval_seconds) # Poll every hour poll_fixed_interval(schedule_id, 3600) ``` ### Cron-based polling ```python theme={null} from apscheduler.schedulers.blocking import BlockingScheduler scheduler = BlockingScheduler() @scheduler.scheduled_job('cron', hour=9) # Daily at 9 AM def check_changes(): """Check for changes on a schedule""" changes = client.get_schedule_changes(schedule_id, mark_seen=True) if changes['count'] > 0: # TODO: Process changes process_changes(changes['changes']) scheduler.start() ``` ### Batch processing ```python theme={null} def batch_process_changes(schedule_ids, batch_size=10): """Process changes for multiple schedules""" all_changes = [] # Collect changes from all schedules for schedule_id in schedule_ids: changes = client.get_schedule_changes(schedule_id, mark_seen=True) all_changes.extend(changes['changes']) if len(all_changes) == 0: print("No changes across all schedules") return # Process in batches for i in range(0, len(all_changes), batch_size): batch = all_changes[i:i + batch_size] # TODO: Process batch print(f"Processing batch {i//batch_size + 1}: {len(batch)} changes") process_batch(batch) ``` ## Advanced patterns ### Exponential backoff on errors ```python theme={null} import time def poll_with_backoff(schedule_id, max_retries=5): """Poll with exponential backoff on errors""" retry_count = 0 while True: try: changes = client.get_schedule_changes(schedule_id, mark_seen=True) if changes['count'] > 0: process_changes(changes['changes']) retry_count = 0 # Reset on success time.sleep(3600) # Normal interval except Exception as e: retry_count += 1 if retry_count > max_retries: print("Max retries exceeded") raise wait_time = 2 ** retry_count print(f"Error: {e}. Retrying in {wait_time}s") time.sleep(wait_time) ``` ### Track processing status ```python theme={null} import json from datetime import datetime def poll_with_tracking(schedule_id, state_file="processing_state.json"): """Poll and track processing status""" # Load state try: with open(state_file, 'r') as f: state = json.load(f) except FileNotFoundError: state = {"last_check": None, "processed_jobs": []} # Check for changes changes = client.get_schedule_changes(schedule_id, mark_seen=True) if changes['count'] > 0: for change in changes['changes']: # TODO: Process change process_change(change) # Track processed job state["processed_jobs"].append({ "job_id": change['job_id'], "processed_at": datetime.now().isoformat() }) # Update state state["last_check"] = datetime.now().isoformat() with open(state_file, 'w') as f: json.dump(state, f, indent=2) ``` ### Preview before processing ```python theme={null} def preview_and_confirm(schedule_id): """Preview changes and ask for confirmation""" # Preview without marking as seen preview = client.get_schedule_changes(schedule_id, mark_seen=False) if preview['count'] == 0: print("No changes to process") return # Show preview print(f"\nFound {preview['count']} changes:") for i, change in enumerate(preview['changes'], 1): print(f"{i}. Job {change['job_id']}: {change['item_count']} items") # Confirm response = input("\nProcess these changes? (y/n): ") if response.lower() == 'y': # Now mark as seen and process changes = client.get_schedule_changes(schedule_id, mark_seen=True) process_changes(changes['changes']) else: print("Skipped processing") ``` ## Best practices ### 1. Match polling interval to schedule frequency ```python theme={null} # If schedule runs every hour, poll every hour or less frequently schedule = client.create_schedule( strategy_id=strategy_id, url=url, interval_seconds=3600 # Every hour ) # Poll slightly less frequently to batch changes while True: changes = client.get_schedule_changes(schedule['schedule_id'], mark_seen=True) # Process... time.sleep(3900) # 65 minutes ``` ### 2. Handle empty results gracefully ```python theme={null} def process_changes_safe(schedule_id): """Handle empty results without errors""" changes = client.get_schedule_changes(schedule_id, mark_seen=True) if changes['count'] == 0: # No changes is normal - don't treat as error return # Process changes... ``` ### 3. Log polling activity ```python theme={null} import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) def poll_with_logging(schedule_id): """Poll with detailed logging""" changes = client.get_schedule_changes(schedule_id, mark_seen=True) logger.info( f"Checked schedule {schedule_id}: {changes['count']} changes", extra={"schedule_id": schedule_id, "change_count": changes['count']} ) for change in changes['changes']: logger.info( f"Processing job {change['job_id']}", extra={"job_id": change['job_id'], "item_count": change['item_count']} ) ``` ## Complete example **TODO: Add complete working example** See [Example: News Aggregation](/guides/examples/news-aggregation) for a full implementation with pull-based monitoring. ## Comparison: Pull vs. Webhooks | Feature | Pull-Based | Webhooks | | ------------ | ----------------------------- | ----------------------------- | | **Latency** | Polling interval | Real-time | | **Control** | You control timing | Meter triggers | | **Setup** | No public endpoint needed | Requires public endpoint | | **Batching** | Easy to batch | Requires queuing | | **Firewall** | Works behind firewall | Requires inbound access | | **Best for** | Batch processing, admin tools | Real-time systems, automation | ## Next steps * See [Webhooks Guide](/guides/webhooks) for real-time alternative * Check [RAG Integration](/guides/rag-integration) for vector database updates * Read [Schedules API Reference](/api-reference/python/schedules) for `get_schedule_changes()` details ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # RAG Integration Guide Source: https://docs.meter.sh/guides/rag-integration Connect Meter to your vector database for automatic knowledge base updates # RAG Integration Guide Learn how to integrate Meter with your RAG (Retrieval-Augmented Generation) system to keep your vector database fresh without wasting embedding costs. ## Overview This guide shows you how to: * Set up Meter to monitor content sources * Detect meaningful changes automatically * Update only changed content in your vector database * Reduce embedding costs by up to 95% ## Architecture ```mermaid theme={null} graph LR A[Meter Schedule] --> B[Content Changes] B --> C{Changes Detected?} C -->|Yes| D[Delete Old Vectors] D --> E[Generate Embeddings] E --> F[Upsert to Vector DB] C -->|No| G[Skip - No Action] ``` ## Prerequisites * Meter API key * Vector database (Pinecone, Weaviate, Qdrant, etc.) * Embedding service (OpenAI, Cohere, etc.) ## Implementation ### Step 1: Generate strategy for content source ```python theme={null} from meter_sdk import MeterClient import os client = MeterClient(api_key=os.getenv("METER_API_KEY")) # TODO: Replace with your content source strategy = client.generate_strategy( url="https://your-docs-site.com/page", description="Extract article title, content, and metadata", name="Documentation Monitor" ) strategy_id = strategy["strategy_id"] print(f"Strategy created: {strategy_id}") ``` ### Step 2: Set up monitoring schedule ```python theme={null} # TODO: Adjust interval based on your needs schedule = client.create_schedule( strategy_id=strategy_id, url="https://your-docs-site.com/page", interval_seconds=3600 # Check every hour ) print(f"Schedule created: {schedule['schedule_id']}") ``` ### Step 3: Process changes and update vector DB **TODO: Add your vector DB integration code here** Example structure (adapt for your vector database): ```python theme={null} # Example with Pinecone import pinecone from openai import OpenAI # TODO: Initialize your vector DB client # pinecone.init(api_key=os.getenv("PINECONE_API_KEY")) # index = pinecone.Index("your-index") # TODO: Initialize your embedding service # openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY")) def process_changes(): """Check for changes and update vector database""" changes = client.get_schedule_changes( schedule_id=schedule['schedule_id'], mark_seen=True ) if changes['count'] == 0: print("No changes detected") return print(f"Processing {changes['count']} changed jobs") for change in changes['changes']: # TODO: Implement your update logic # 1. Delete old vectors for this URL # 2. Generate new embeddings for changed content # 3. Upsert new vectors url = change['url'] results = change['results'] print(f"Processing change for {url}: {len(results)} items") # Example: Delete old vectors # index.delete(filter={"url": url}) # Example: Generate embeddings and upsert # for item in results: # embedding = generate_embedding(item['content']) # index.upsert([(item['id'], embedding, {"url": url, ...})]) # Run periodically import time while True: process_changes() time.sleep(3600) # Check every hour ``` ## Best practices ### 1. Batch vector operations **TODO: Add batching logic for your vector DB** ```python theme={null} # Example: Batch upserts for better performance def batch_upsert(vectors, batch_size=100): """Upsert vectors in batches""" for i in range(0, len(vectors), batch_size): batch = vectors[i:i + batch_size] # TODO: Implement batch upsert for your vector DB # index.upsert(batch) pass ``` ### 2. Handle embedding failures gracefully **TODO: Add error handling for your embedding service** ```python theme={null} def generate_embedding_with_retry(text, max_retries=3): """Generate embedding with retry logic""" for attempt in range(max_retries): try: # TODO: Call your embedding service # response = openai_client.embeddings.create( # model="text-embedding-3-small", # input=text # ) # return response.data[0].embedding pass except Exception as e: if attempt < max_retries - 1: time.sleep(2 ** attempt) else: raise ``` ### 3. Track which URLs are indexed **TODO: Implement URL tracking for your use case** ```python theme={null} # Example: Store metadata to track indexed content def track_indexed_content(url, content_hash, vector_ids): """Track which content has been indexed""" # TODO: Implement tracking (database, file, etc.) pass ``` ## Example integrations ### Pinecone **TODO: Add Pinecone-specific code** ```python theme={null} # import pinecone # pinecone.init(api_key=os.getenv("PINECONE_API_KEY")) # index = pinecone.Index("docs") # Delete and upsert # index.delete(filter={"source": url}) # index.upsert(vectors) ``` ### Weaviate **TODO: Add Weaviate-specific code** ```python theme={null} # import weaviate # client = weaviate.Client(url=os.getenv("WEAVIATE_URL")) # Delete and create # client.batch.delete_objects(...) # client.batch.add_data_object(...) ``` ### Qdrant **TODO: Add Qdrant-specific code** ```python theme={null} # from qdrant_client import QdrantClient # qdrant = QdrantClient(url=os.getenv("QDRANT_URL")) # Delete and upsert # qdrant.delete(collection_name="docs", points_selector=...) # qdrant.upsert(collection_name="docs", points=...) ``` ## Monitoring and logging **TODO: Add monitoring for your setup** ```python theme={null} import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) def process_changes_with_logging(): """Process changes with detailed logging""" changes = client.get_schedule_changes(schedule_id, mark_seen=True) logger.info(f"Checked schedule: {changes['count']} changes") for change in changes['changes']: logger.info(f"Processing {change['url']}: {change['item_count']} items") # Process... logger.info(f"Completed {change['url']}") ``` ## Cost optimization Meter helps you reduce costs by: 1. **Avoiding re-embeddings**: Only embed changed content 2. **Efficient change detection**: Content hashing catches changes instantly 3. **Batching updates**: Process multiple changes together **Before Meter:** * Scrape daily: 30 scrapes/month * No change detection: Embed all content every time * Cost: 30 × \$X = High embedding costs **After Meter:** * Scrape hourly: 720 scrapes/month * Change detection: Embed only changes * Cost: \~5% of old cost (only when content changes) ## Next steps * See the [Pull-Based Monitoring Guide](/guides/pull-based-monitoring) for polling patterns * Check the [Webhooks Guide](/guides/webhooks) for real-time updates * Explore [Example: News Aggregation](/guides/examples/news-aggregation) for a complete implementation ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Site Crawling Guide Source: https://docs.meter.sh/guides/site-crawling Discover and scrape URLs across an entire website # Site Crawling Guide Learn how to discover URLs on a website and scrape them in bulk using Meter's site crawling feature. ## Overview Site crawling automates the process of finding URLs to scrape. Instead of manually collecting URLs, you configure how to discover them and Meter does the rest. **Use site crawling when:** * You need to scrape many pages from the same site * URLs follow a pattern (sitemaps, pagination, link structure) * You want to create recurring scrapes of discovered URLs ## Prerequisites Before you start: * Create a Meter account at [meter.sh](https://meter.sh) * Have a strategy ready for the pages you want to scrape * Know how URLs are organized on your target site ## Choosing a discovery method | Method | Best for | Example | | ------------ | --------------------------- | --------------------------- | | Sitemap | Sites with `sitemap.xml` | E-commerce product catalogs | | Pagination | Predictable page URLs | Search results, listings | | Link Pattern | Crawling by following links | News articles, blog posts | ## Method 1: Sitemap discovery Sitemaps are the fastest and most reliable discovery method. ### Step 1: Find the sitemap Most sites have a sitemap at `/sitemap.xml` or listed in `robots.txt`: ```bash theme={null} # Check common locations curl https://example.com/sitemap.xml curl https://example.com/robots.txt | grep -i sitemap ``` ### Step 2: Start discovery 1. Go to the **Dashboard** and click **Discover URLs** 2. Select **Sitemap** as the discovery method 3. Enter the sitemap URL (e.g., `https://shop.com/sitemap.xml`) 4. Optionally add a URL pattern to filter results 5. Click **Start Discovery** ```bash theme={null} curl -X POST https://api.meter.sh/discover \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "discovery": { "method": "sitemap", "sitemap_url": "https://shop.com/sitemap.xml", "url_pattern": "products/*/", "max_urls": 1000 } }' ``` Response: ```json theme={null} { "discovery_id": "550e8400-e29b-41d4-a716-446655440000", "status": "pending", "discovery_method": "sitemap", "root_url": "https://shop.com/sitemap.xml" } ``` ### Step 3: Poll for results Discovery runs asynchronously. Poll until status is `completed`: ```bash theme={null} curl https://api.meter.sh/discover/550e8400-e29b-41d4-a716-446655440000 \ -H "Authorization: Bearer sk_live_..." ``` Response when complete: ```json theme={null} { "discovery_id": "550e8400-e29b-41d4-a716-446655440000", "status": "completed", "total_urls": 847, "sample_urls": [ "https://shop.com/products/widget-a", "https://shop.com/products/widget-b", "..." ] } ``` ## Method 2: Pagination discovery Use this when URLs follow a numbered pattern. ### Configuration | Parameter | Description | Example | | -------------- | -------------------------- | ------------------------------------ | | `url_template` | URL with `{n}` placeholder | `https://shop.com/products?page={n}` | | `start_index` | First page number | `1` | | `step` | Increment between pages | `1` | | `max_pages` | Maximum pages to generate | `100` | ### Example 1. Select **Pagination** as the discovery method 2. Enter URL template: `https://shop.com/search?page={n}` 3. Set start index, step, and max pages 4. Click **Start Discovery** ```bash theme={null} curl -X POST https://api.meter.sh/discover \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "discovery": { "method": "pagination", "url_template": "https://shop.com/search?page={n}", "start_index": 1, "step": 1, "max_pages": 50 } }' ``` ## Method 3: Link pattern discovery Use this to crawl a site and collect URLs matching a pattern. ### Configuration | Parameter | Description | Example | | -------------------- | --------------------------- | ------------------ | | `seed_url` | Starting URL | `https://news.com` | | `link_pattern` | Pattern to match (glob) | `/article/*/` | | `navigation_pattern` | Pages to visit during crawl | `/category/` | | `max_depth` | How deep to crawl | `2` | | `max_urls` | Maximum URLs to collect | `500` | ### Example 1. Select **Link Pattern** as the discovery method 2. Enter seed URL: `https://news.com` 3. Enter link pattern: `/article/*/` 4. Set max depth and max URLs 5. Click **Start Discovery** ```bash theme={null} curl -X POST https://api.meter.sh/discover \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "discovery": { "method": "link_pattern", "seed_url": "https://news.com", "link_pattern": "/article/*/", "navigation_pattern": "/category/", "max_depth": 2, "max_urls": 500 } }' ``` The **navigation pattern** defines which pages to visit during the crawl. The **link pattern** defines which URLs to collect. They work together: Meter visits navigation pages to find links matching your collection pattern. ## Executing discovered URLs Once discovery completes, you can execute immediately or create a schedule. ### One-time execution Scrape all discovered URLs immediately: 1. Review the discovered URLs 2. Select your extraction strategy 3. Set maximum URLs to process 4. Click **Execute Now** ```bash theme={null} curl -X POST https://api.meter.sh/discover/550e8400.../execute \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "max_urls": 100, "url_filter": ".*widget.*" }' ``` Response: ```json theme={null} { "batch_id": "770e8400-e29b-41d4-a716-446655440000", "jobs_queued": 100 } ``` ### Create a schedule Set up recurring scrapes: 1. Click **Create Schedule** 2. Choose interval (e.g., every 24 hours) or cron expression 3. Optionally add a webhook URL 4. Click **Create Scheduled Job** ```bash theme={null} curl -X POST https://api.meter.sh/discover/550e8400.../schedule \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "strategy_id": "660e8400-e29b-41d4-a716-446655440000", "interval_seconds": 86400, "webhook_url": "https://your-app.com/webhooks/meter", "max_urls": 500 }' ``` ## Filtering URLs Use regex patterns to filter discovered URLs: ```bash theme={null} # Only URLs containing "widget" "url_filter": ".*widget.*" # Only product pages with numeric IDs "url_filter": "/products/\\d+" # Exclude certain paths "url_filter": "^(?!.*/archive/).*$" ``` ## Complete example Here's a full workflow for scraping a product catalog: ```bash theme={null} # 1. Create a strategy for product pages STRATEGY=$(curl -X POST https://api.meter.sh/strategies \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://shop.com/products/sample", "description": "Extract product name, price, and description", "name": "Shop Products" }' | jq -r '.strategy_id') # 2. Start sitemap discovery DISCOVERY=$(curl -X POST https://api.meter.sh/discover \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "discovery": { "method": "sitemap", "sitemap_url": "https://shop.com/sitemap.xml", "url_pattern": "products/*/" } }' | jq -r '.discovery_id') # 3. Wait for discovery to complete sleep 30 # 4. Check status curl https://api.meter.sh/discover/$DISCOVERY \ -H "Authorization: Bearer sk_live_..." # 5. Execute with the strategy curl -X POST https://api.meter.sh/discover/$DISCOVERY/execute \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d "{ \"strategy_id\": \"$STRATEGY\", \"max_urls\": 100 }" ``` ## Troubleshooting **Solutions:** * Check `robots.txt` for the sitemap location * Try common paths: `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap/sitemap.xml` * Some sites use dynamic sitemaps - check the page source for sitemap links **Causes:** * URL pattern too restrictive * Sitemap is empty or blocked * Link pattern doesn't match any URLs **Solutions:** * Remove or broaden the URL pattern * Verify the sitemap loads in a browser * Test your link pattern against sample URLs **Cause:** Large sitemaps or deep crawls take time **Solutions:** * Reduce `max_urls` or `max_depth` * Use URL patterns to target specific sections * For very large sites, run multiple smaller discoveries **Solutions:** * Add a URL pattern to filter during discovery * Use `url_filter` regex when executing * For link pattern crawls, be more specific with patterns ## Next steps Understand how site crawling works View all discovery endpoints Get notified when scrapes complete Track changes between scrapes ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Webhooks Guide Source: https://docs.meter.sh/guides/webhooks Receive real-time notifications when content changes # Webhooks Guide Set up webhook endpoints to receive immediate notifications when Meter detects content changes. ## Overview Webhooks allow Meter to push change notifications to your application in real-time, eliminating the need for polling. **Use webhooks when:** * Changes need immediate action * Building event-driven systems * Triggering downstream workflows **Use pull-based instead when:** * Batch processing changes * Webhooks aren't feasible (firewall, no public endpoint) * Prefer manual control over timing ## How it works ```mermaid theme={null} sequenceDiagram participant M as Meter participant Y as Your Webhook participant D as Your Database M->>M: Run scheduled job M->>M: Detect changes M->>Y: POST /webhooks/meter Y->>D: Update data Y->>M: 200 OK ``` ## Webhook types Meter supports four webhook formats: | Type | Description | When to use | | ---------------- | ---------------------------------------- | ------------------------------------ | | `standard` | Full JSON payload with all results | Most integrations | | `slack` | Formatted Slack incoming webhook message | Slack channels via incoming webhooks | | `slack_workflow` | Slack Workflow Builder trigger payload | Slack Workflow Builder automations | | `discord` | Formatted Discord webhook embed | Discord channels | The webhook type is **auto-detected from the URL**: | URL pattern | Detected type | | ------------------------------------ | ---------------- | | Contains `hooks.slack.com/services/` | `slack` | | Contains `hooks.slack.com/triggers/` | `slack_workflow` | | Contains `discord.com/api/webhooks/` | `discord` | | Everything else | `standard` | You can also set it explicitly: ```python theme={null} schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600, webhook_url="https://hooks.slack.com/services/T.../B.../xxx", webhook_type="slack" # auto-detected, but can be explicit ) ``` ## Webhook payload Meter sends a POST request for both successful and failed jobs. **Success payload:** ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "status": "completed", "url": "https://example.com/products", "results": [ {"title": "Product A", "price": "$19.99"}, {"title": "Product B", "price": "$29.99"} ], "item_count": 2, "has_changes": true, "content_hash": "7f3d9a2b4c1e...", "completed_at": "2025-01-15T10:30:12Z", "delivery_reason": "first_run", "metadata": {"project": "my-project", "env": "prod"} } ``` `delivery_reason` distinguishes the initial backfill (`first_run`) from change-driven deliveries (`content_changed`). Use it to skip notifying end-users on the very first run for a new schedule. **Failure payload:** ```json theme={null} { "job_id": "660e8400-e29b-41d4-a716-446655440000", "schedule_id": "880e8400-e29b-41d4-a716-446655440000", "status": "failed", "url": "https://example.com/products", "error": "Page not accessible: 404 Not Found", "completed_at": "2025-01-15T10:30:12Z", "metadata": {"project": "my-project", "env": "prod"} } ``` The `metadata` field contains your custom JSON from `webhook_metadata` — it's included in every payload so you can identify which project or environment the webhook belongs to. See [Webhook payload formats](/api-reference/rest/webhooks#webhook-payload-formats) for full field documentation. ## Webhook metadata Attach custom JSON data to every webhook payload. This is useful for routing, tagging, or identifying which schedule triggered the webhook: ```python theme={null} schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter", webhook_metadata={ "project": "price-monitor", "env": "production", "team": "data-eng" } ) ``` Your webhook handler can then use this metadata for routing: ```python theme={null} @app.post("/webhooks/meter") async def handle_webhook(request: Request): payload = await request.json() metadata = payload.get("metadata", {}) if metadata.get("project") == "price-monitor": await update_price_database(payload) elif metadata.get("project") == "news-tracker": await update_news_feed(payload) ``` ## Webhook secrets Webhook secrets let you verify that incoming requests are from Meter, not a third party. ### How it works 1. When you create a schedule with a `webhook_url`, Meter auto-generates a secret with a `whsec_` prefix 2. Every webhook request includes the secret in the `X-Webhook-Secret` header 3. Your endpoint verifies the header matches your stored secret ### Storing the secret The secret is returned **once** when the schedule is created. Store it securely: ```python theme={null} schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter" ) # Save this — it won't be shown again in other API responses webhook_secret = schedule.get("webhook_secret") print(f"Store this secret: {webhook_secret}") ``` You can also provide your own secret: ```python theme={null} schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter", webhook_secret="whsec_my_custom_secret_here" ) ``` ### Verifying requests Check the `X-Webhook-Secret` header in your webhook handler: ```python theme={null} from fastapi import FastAPI, Request, HTTPException app = FastAPI() WEBHOOK_SECRET = "whsec_..." # From schedule creation @app.post("/webhooks/meter") async def handle_webhook(request: Request): # Verify the secret secret = request.headers.get("X-Webhook-Secret") if secret != WEBHOOK_SECRET: raise HTTPException(status_code=401, detail="Invalid webhook secret") payload = await request.json() # Process payload... return {"status": "ok"} ``` ```javascript theme={null} const express = require('express'); const app = express(); app.use(express.json()); const WEBHOOK_SECRET = 'whsec_...'; // From schedule creation app.post('/webhooks/meter', (req, res) => { // Verify the secret const secret = req.headers['x-webhook-secret']; if (secret !== WEBHOOK_SECRET) { return res.status(401).json({ error: 'Invalid webhook secret' }); } const payload = req.body; // Process payload... res.status(200).json({ status: 'ok' }); }); ``` ```python theme={null} from flask import Flask, request, jsonify app = Flask(__name__) WEBHOOK_SECRET = "whsec_..." # From schedule creation @app.route('/webhooks/meter', methods=['POST']) def handle_webhook(): # Verify the secret secret = request.headers.get('X-Webhook-Secret') if secret != WEBHOOK_SECRET: return jsonify({"error": "Invalid secret"}), 401 payload = request.json # Process payload... return jsonify({"status": "ok"}), 200 ``` ### Regenerating secrets If a secret is compromised, regenerate it: ```python theme={null} result = client.regenerate_webhook_secret(schedule_id) new_secret = result["webhook_secret"] # Update your webhook handler with the new secret ``` The old secret is immediately invalidated. Update your webhook handler before the next delivery. ## Retry behavior Meter automatically retries failed webhook deliveries with exponential backoff: | Retry | Delay | Total elapsed | | ----------- | ---------- | ------------------ | | 1st | 15 minutes | 15 minutes | | 2nd | 30 minutes | 45 minutes | | 3rd | 1 hour | 1 hour 45 minutes | | 4th | 2 hours | 3 hours 45 minutes | | 5th (final) | 4 hours | 7 hours 45 minutes | **Retry rules:** * **2xx response**: Delivery successful, no retry * **4xx response** (client error): Delivery stops immediately — no retries. Fix your endpoint and the next scheduled job will deliver normally * **5xx response** (server error): Retries with backoff * **Timeout** (>30 seconds): Retries with backoff * **Connection failure**: Retries with backoff Return `200 OK` as quickly as possible. Process the payload asynchronously in a background task to avoid timeouts. ## Implementation ### Step 1: Create a webhook endpoint ```python theme={null} from fastapi import FastAPI, Request, HTTPException, BackgroundTasks app = FastAPI() WEBHOOK_SECRET = "whsec_..." @app.post("/webhooks/meter") async def handle_meter_webhook( request: Request, background_tasks: BackgroundTasks ): # Verify secret secret = request.headers.get("X-Webhook-Secret") if secret != WEBHOOK_SECRET: raise HTTPException(status_code=401, detail="Invalid secret") payload = await request.json() if payload["status"] == "failed": background_tasks.add_task(handle_failure, payload) return {"status": "ok"} if payload.get("has_changes"): background_tasks.add_task(process_changes, payload) return {"status": "ok"} async def process_changes(payload): results = payload["results"] metadata = payload.get("metadata", {}) # Update your database, trigger pipelines, etc. async def handle_failure(payload): # Log, alert, or retry logic print(f"Job failed: {payload['error']}") ``` ```javascript theme={null} const express = require('express'); const app = express(); app.use(express.json()); const WEBHOOK_SECRET = 'whsec_...'; app.post('/webhooks/meter', async (req, res) => { // Verify secret const secret = req.headers['x-webhook-secret']; if (secret !== WEBHOOK_SECRET) { return res.status(401).json({ error: 'Invalid secret' }); } const payload = req.body; if (payload.status === 'failed') { console.error(`Job failed: ${payload.error}`); return res.status(200).json({ status: 'ok' }); } if (payload.has_changes) { // Process asynchronously setImmediate(() => processChanges(payload)); } res.status(200).json({ status: 'ok' }); }); async function processChanges(payload) { const results = payload.results; const metadata = payload.metadata || {}; // Update database, etc. } app.listen(3000); ``` ```python theme={null} from flask import Flask, request, jsonify import threading app = Flask(__name__) WEBHOOK_SECRET = "whsec_..." @app.route('/webhooks/meter', methods=['POST']) def handle_meter_webhook(): # Verify secret secret = request.headers.get('X-Webhook-Secret') if secret != WEBHOOK_SECRET: return jsonify({"error": "Invalid secret"}), 401 payload = request.json if payload["status"] == "failed": print(f"Job failed: {payload['error']}") return jsonify({"status": "ok"}), 200 if payload.get("has_changes"): # Process in background thread threading.Thread( target=process_changes, args=(payload,) ).start() return jsonify({"status": "ok"}), 200 def process_changes(payload): results = payload["results"] metadata = payload.get("metadata", {}) # Update database, etc. ``` ### Step 2: Make endpoint publicly accessible Options: * Deploy to cloud (AWS Lambda, Google Cloud Functions, etc.) * Use ngrok for local development: `ngrok http 3000` * Use a VPS with public IP ### Step 3: Create schedule with webhook URL ```python theme={null} from meter_sdk import MeterClient client = MeterClient(api_key="sk_live_...") schedule = client.create_schedule( strategy_id="your-strategy-id", url="https://example.com/products", interval_seconds=3600, webhook_url="https://your-app.com/webhooks/meter", webhook_metadata={"project": "price-monitor"} ) # Store the auto-generated webhook secret print(f"Webhook secret: {schedule.get('webhook_secret')}") ``` ## Slack webhooks Send notifications directly to Slack channels using incoming webhooks: ```python theme={null} schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600, webhook_url="https://hooks.slack.com/services/T.../B.../xxx" # webhook_type auto-detected as "slack" ) ``` Slack payloads are automatically formatted as readable messages with item counts and a preview of results. ## Slack Workflow webhooks Trigger Slack Workflow Builder automations with scrape results. Use this when you've built a workflow in Slack's Workflow Builder and want Meter to trigger it. ```python theme={null} schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600, webhook_url="https://hooks.slack.com/triggers/T.../B.../xxx" # webhook_type auto-detected as "slack_workflow" ) ``` Slack Workflow webhooks do not require a webhook secret. The URL itself serves as the authentication mechanism. ## Discord webhooks Send formatted notifications to Discord channels: ```python theme={null} schedule = client.create_schedule( strategy_id=strategy_id, url="https://example.com/products", interval_seconds=3600, webhook_url="https://discord.com/api/webhooks/123/abc..." # webhook_type auto-detected as "discord" ) ``` Discord payloads are formatted as embeds with item counts and a preview of results. Discord webhooks do not require a webhook secret. Discord authenticates via the webhook URL. ## Best practices ### 1. Respond quickly Return `200 OK` within 30 seconds. Process payloads asynchronously: ```python theme={null} @app.post("/webhooks/meter") async def handle_webhook(request: Request, background_tasks: BackgroundTasks): payload = await request.json() background_tasks.add_task(process_changes, payload) return {"status": "ok"} # Respond immediately ``` ### 2. Handle duplicates Make processing idempotent using the `job_id`: ```python theme={null} processed_jobs = set() async def process_changes(payload): job_id = payload['job_id'] if job_id in processed_jobs: return # Skip duplicate processed_jobs.add(job_id) # Process... ``` ### 3. Handle failures gracefully ```python theme={null} @app.post("/webhooks/meter") async def handle_webhook(request: Request): try: payload = await request.json() await process_changes(payload) return {"status": "ok"} except Exception as e: logger.error(f"Webhook processing failed: {e}") # Return 200 to prevent retries (payload is stored for manual review) await save_failed_webhook(payload, str(e)) return {"status": "error", "message": str(e)} ``` ## Testing webhooks ### Local testing with ngrok ```bash theme={null} # Start your webhook server python app.py # In another terminal, expose with ngrok ngrok http 3000 # Use the ngrok URL in Meter # https://abc123.ngrok.io/webhooks/meter ``` ### Testing with webhook.site 1. Go to [https://webhook.site](https://webhook.site) 2. Copy the unique URL 3. Use it in your Meter schedule 4. Trigger a job and view the payload on webhook.site ### Test endpoint Use the Meter test endpoint to verify delivery: ```bash theme={null} curl -X POST https://api.meter.sh/api/webhooks/test \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{"webhook_url": "https://your-app.com/webhooks/meter"}' ``` ### Manual testing ```bash theme={null} curl -X POST http://localhost:3000/webhooks/meter \ -H "Content-Type: application/json" \ -H "X-Webhook-Secret: whsec_test" \ -d '{ "job_id": "test-123", "status": "completed", "url": "https://example.com", "has_changes": true, "results": [{"test": "data"}], "item_count": 1, "metadata": {"project": "test"} }' ``` ## Troubleshooting **Solutions:** * Verify URL is publicly accessible * Check endpoint returns 200 OK * Test with webhook.site * Check server logs for errors * Verify the schedule has a `webhook_url` set **Cause:** Endpoint takes >30 seconds to respond **Solution:** Return 200 OK immediately, process asynchronously: ```python theme={null} @app.post("/webhooks/meter") async def handle_webhook(request: Request, background_tasks: BackgroundTasks): payload = await request.json() background_tasks.add_task(process_changes, payload) return {"status": "ok"} # Immediate response ``` **Cause:** Retries after timeout or connection issues **Solution:** Make processing idempotent using `job_id`: ```python theme={null} processed_jobs = set() async def process_changes(payload): job_id = payload['job_id'] if job_id in processed_jobs: return processed_jobs.add(job_id) # Process... ``` **Cause:** Your endpoint returns a 4xx status code **Solution:** Meter treats 4xx as a permanent failure and does not retry. Check your endpoint for: * Invalid webhook secret (401) * Incorrect URL path (404) * Request validation errors (422) Fix the issue and delivery will resume on the next scheduled job. ## Next steps * Try [Pull-Based Monitoring](/guides/pull-based-monitoring) as an alternative * See [RAG Integration](/guides/rag-integration) for vector database updates * Check [Schedule API Reference](/api-reference/python/schedules) for webhook configuration * See [Webhook payload formats](/api-reference/rest/webhooks) for full field documentation ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Introduction Source: https://docs.meter.sh/introduction AI-powered web scraping with intelligent change detection to keep your data fresh # Welcome to Meter Meter is an AI-powered web scraping platform that helps you extract data from websites, monitor them for changes, and get notified only when meaningful content updates occur. **API-first extraction, fully automated.** Meter automatically discovers and extracts data from hidden APIs when available—delivering cleaner, faster, more reliable results. When APIs aren't available, it falls back to intelligent HTML extraction. You get the best of both worlds without any extra configuration. ## What makes Meter different? Meter solves the core problem with traditional web scraping: **wasted resources from re-processing unchanged data**. Whether you're building a RAG system, monitoring competitor prices, or tracking news articles, Meter ensures you only process what's actually new. Cloudflare, PerimeterX, DataDome—we handle it. Our antibot bypass means you can scrape sites that block traditional scrapers. No more 403s or CAPTCHAs. JavaScript-heavy sites? No problem. Meter automatically detects hidden APIs, handles authentication tokens, and extracts data directly from the source—no brittle DOM scraping required. Use AI to generate extraction strategies once. All future scrapes use fast, reliable CSS selectors—no recurring LLM costs. Detect meaningful content changes using content hashing and structural signatures. Stop wasting compute on layout updates and noise. Cut re-embedding costs for RAG systems by up to 95%. Only update your vector database when content actually changes. ## How it works Give Meter a URL and a plain English description: "Extract product names and prices" or "Get article headlines and authors." Meter's AI analyzes the page and chooses the optimal extraction method. For traditional pages, it creates CSS selectors. For JavaScript-heavy sites, it automatically discovers APIs and generates direct extraction code. Set up automated monitoring (hourly, daily, cron-based). Meter scrapes using the saved strategy—no LLM costs. Receive webhooks or poll for changes. Meter's diffing detects structural content changes, filtering out layout noise and timestamps. ## Use cases Keep your embeddings fresh without wasting tokens. Meter detects content changes and triggers re-embedding only for updated content, reducing costs by up to 95%. Track competitor prices, product availability, or market trends. Get notified instantly when prices change—ignore layout updates and ads. Monitor news sites, blogs, or forums for new articles. Detect new posts while filtering out timestamp changes and layout shifts. Scrape job listings and get alerts when new positions appear. Perfect for talent teams or job search automation. ## Key concepts Before diving in, familiarize yourself with these core concepts: * **[Strategy](/concepts/strategies)**: An AI-generated extraction plan that defines how to scrape a website * **[Job](/concepts/jobs)**: A single execution of a scrape using a strategy * **[Schedule](/concepts/schedules)**: Automated recurring scrapes at specified intervals * **[Workflow](/concepts/workflows)**: DAG-based pipelines that chain multiple strategies together * **[Content Diffing](/concepts/change-detection)**: Intelligent change detection using content hashing and structural signatures ## Ready to start? Generate your first strategy and run a scrape in 5 minutes Explore the Python SDK and REST API documentation Learn how to integrate Meter with your RAG pipeline See real-world examples and copy-paste code ## Current status Meter is currently in **early beta**. The API is functional and stable for core features, but breaking changes may occur. We're actively developing new features and improvements. * ✅ AI strategy generation * ✅ Scheduled monitoring (cron and interval) * ✅ Webhooks (push-based notifications) * ✅ Pull-based change detection * ✅ Content diffing (content hashing + structural signatures) * ✅ Antibot bypass (feature-gated—[contact us](mailto:mckinnon@meter.sh)) * ✅ LLM summaries (feature-gated—[contact us](mailto:mckinnon@meter.sh)) * ✅ Automatic API discovery for JavaScript-heavy sites * ✅ Workflows (DAG-based multi-step pipelines) * 🚧 Semantic similarity detection (roadmap) ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh) # Quick Start Source: https://docs.meter.sh/quickstart Generate your first strategy and run a scrape in 5 minutes # Quick Start This guide will get you up and running with Meter in under 5 minutes. You'll generate an extraction strategy, run your first scrape, and set up monitoring. **Prerequisites**: * Python 3.8 or later * A Meter account ([sign up here](https://meter.sh/login)) * Your API key from the [dashboard](https://meter.sh/dashboard) ## 1. Install the SDK Install the Meter Python SDK: ```bash theme={null} pip install meter-sdk ``` ## 2. Set up authentication Store your API key securely as an environment variable: ```bash theme={null} export METER_API_KEY="sk_live_your_api_key_here" ``` Get your API key from the [Meter dashboard](https://meter.sh/dashboard). Never commit API keys to version control. ## 3. Generate your first strategy Create a Python file and generate an extraction strategy: ```python theme={null} from meter_sdk import MeterClient import os # Initialize the client client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Generate a strategy for Hacker News result = client.generate_strategy( url="https://news.ycombinator.com", description="Extract post titles and scores", name="HN Front Page" ) # Print the results strategy_id = result["strategy_id"] print(f"Strategy created: {strategy_id}") print(f"\nPreview data ({len(result['preview_data'])} items):") for item in result['preview_data'][:3]: print(f" - {item}") ``` **What's happening here:** 1. Meter's AI analyzes the page structure 2. Generates CSS selectors for extracting titles and scores 3. Returns a preview of extracted data 4. Saves the strategy for reuse (no LLM costs on future scrapes) ## 4. Run a scrape job Use your saved strategy to run a scrape: ```python theme={null} # Create a job using the strategy job = client.create_job( strategy_id=strategy_id, url="https://news.ycombinator.com" ) # Wait for completion completed_job = client.wait_for_job(job["job_id"]) # Print results print(f"\nScraped {len(completed_job['results'])} items") for item in completed_job['results'][:5]: print(f" - {item}") ``` Jobs run asynchronously. The `wait_for_job()` method polls automatically until completion. ## 5. Set up monitoring Schedule automatic scrapes to monitor for changes: ```python theme={null} # Run every hour schedule = client.create_schedule( strategy_id=strategy_id, url="https://news.ycombinator.com", interval_seconds=3600 ) print(f"Schedule created: {schedule['schedule_id']}") print(f"Next run: {schedule['next_run_at']}") ``` Now Meter will automatically scrape Hacker News every hour. You can check for changes using the pull-based API or set up webhooks. ## 6. Check for changes Use the pull-based API to get changes: ```python theme={null} # Get changes for a schedule changes = client.get_schedule_changes( schedule_id=schedule['schedule_id'], mark_seen=True # Mark as seen after reading ) if changes['count'] > 0: print(f"\n{changes['count']} jobs with changes:") for change in changes['changes']: print(f" - Job {change['job_id']}: {change['item_count']} items") else: print("\nNo changes detected") ``` Set `mark_seen=False` to preview changes without marking them as read. ## Complete example Here's the complete code: ```python theme={null} from meter_sdk import MeterClient import os # Initialize client client = MeterClient(api_key=os.getenv("METER_API_KEY")) # Step 1: Generate strategy result = client.generate_strategy( url="https://news.ycombinator.com", description="Extract post titles and scores", name="HN Front Page" ) strategy_id = result["strategy_id"] print(f"✓ Strategy created: {strategy_id}") # Step 2: Run initial scrape job = client.create_job(strategy_id, "https://news.ycombinator.com") initial = client.wait_for_job(job["job_id"]) print(f"✓ Initial scrape: {initial['item_count']} items") # Step 3: Set up monitoring schedule = client.create_schedule( strategy_id=strategy_id, url="https://news.ycombinator.com", interval_seconds=3600 ) print(f"✓ Monitoring enabled: every hour") # Step 4: Check for changes changes = client.get_schedule_changes(schedule['schedule_id']) print(f"✓ Changes detected: {changes['count']} jobs") ``` ## Next steps Learn about strategies, jobs, and schedules Connect Meter to your vector database Set up real-time notifications Explore all SDK methods ## Troubleshooting **Possible causes:** * URL is not accessible * Description is too vague or complex * Page requires authentication **Solutions:** * Verify the URL loads in your browser * Make your description more specific: "Extract product names and prices from the grid" instead of "Get products" * For auth-required pages, contact support **Possible causes:** * Target website is slow or down * Strategy is incorrect **Solutions:** * Increase timeout: `client.wait_for_job(job_id, timeout=300)` * Check job status manually: `client.get_job(job_id)` * Refine strategy if results are incorrect **Problem:** `ModuleNotFoundError: No module named 'meter_sdk'` **Solution:** Install the SDK: `pip install meter-sdk` **Problem:** `401 Unauthorized` **Solutions:** * Verify your API key is correct * Check that `METER_API_KEY` environment variable is set * Ensure your API key hasn't expired ## Need help? Email me at [mckinnon@meter.sh](mailto:mckinnon@meter.sh)