Skip to main content

News Aggregation

Build a news aggregation system that monitors multiple sources and detects new articles.

Use case

Aggregate news from multiple sources, detect new articles, and feed them into your knowledge base or RAG system.

What you’ll build

  • Strategies for multiple news sources
  • Hourly monitoring of all sources
  • New article detection
  • RAG/vector database integration

Prerequisites

  • Meter API key
  • TODO: Vector database (Pinecone, Weaviate, etc.)
  • TODO: Embedding service (OpenAI, Cohere, etc.)

Step 1: Generate strategies for news sources

TODO: Add your news sources

Step 2: Set up monitoring for all sources

Step 3: Aggregate new articles

Step 4: Keyword filtering for specific topics

Use keyword filters to only retrieve articles matching specific topics. This is useful when monitoring news for specific keywords relevant to your business.

Filter syntax

Example: Monitor news for specific keywords

Complete example: Multi-topic news monitor

The filter applies to individual result items. If a job returns 50 articles but only 5 match your filter, you’ll only receive those 5 matching articles. Jobs with zero matching items are excluded entirely.

Step 5: Feed into RAG system

TODO: Integrate with your vector database

Complete example

TODO: Full implementation

Deployment

TODO: Deploy your aggregator
  • Run as background service
  • Use cron for periodic checks
  • Deploy to cloud (AWS, GCP, etc.)

Advanced features

TODO: Extend functionality
  • Article deduplication across sources
  • Trend detection
  • Topic clustering
  • Sentiment analysis integration

See also

Need help?

Email me at mckinnon@meter.sh