TelecomIQ is an end-to-end market intelligence system built for the Indian telecom space. It automatically scrapes live plans from Jio, Airtel, and Vi using headless Chromium, runs them through a change-detection engine, and surfaces insights on a modern React dashboard — complete with an AI pricing strategist powered by Azure OpenAI.
138+ plans tracked. 3 providers. 5 change types. 9 dashboard views. 1 AI brain.
Built on Playwright (headless Chromium) with multi-strategy extraction — uses a cascade of CSS selectors targeting plan card components, with a JS DOM fallback that pattern-matches ₹price + XGB + Xdays across any page layout. Rotates 6 user-agent strings to avoid bot detection. All 3 scrapers fire concurrently via Prefect's task submit API.
Every scraped plan is fingerprinted with an MD5 hash of (price + data_gb + validity_days + perks). On each scrape run:
- New hash seen →
new_planevent logged - Hash gone from live market →
removedevent logged - Price differs on same plan →
price_droporprice_hikelogged - Perks list changed →
perk_changelogged
The stats API deliberately excludes bulk new_plan events from the weekly change count — only real market movements are surfaced.
A three-stage pipeline feeds structured market data into Azure OpenAI:
Active plans (DB)
│
├─► Gap Analyzer — finds ₹50+ dead zones between competitors
│ calculates sweet_spot at 40% into each gap
│
├─► Elasticity Scorer — scores 5 price bands (budget → premium)
│ opportunity_score = value_score / (density + 1)
│
└─► LLM Strategist — LangChain + AzureChatOpenAI (temp=0.3)
Pydantic-forced structured output:
price · rationale · perks · positioning
advantage · risk · opportunity · confidence
HTML-formatted alert emails triggered when market movements cross configurable thresholds. Tracks a last_alerted_change_id watermark so no event is emailed twice. Supports per-event-type toggles (price drops, hikes, removals, perk changes) and a minimum ₹ delta filter.
| Page | What You See |
|---|---|
| Overview | KPIs, price tier distribution chart, recent changes feed |
| Plan Comparison | Side-by-side grid across all providers and plan types |
| Price History | Line chart of price evolution over time |
| Perks Matrix | OTT/perk availability heatmap (Netflix, Hotstar, Prime, etc.) |
| Plans by Duration | Validity-grouped analysis (7d quick recharges → annual plans) |
| Market Intelligence | Value metrics (GB/₹), plan clusters, competitive matrix, opportunities |
| AI Strategy | LLM recommendations with gap waterfall and elasticity charts |
| Chairman's Brief | Executive threat assessment and strategic imperatives |
| Email Alerts | Alert configuration, SMTP setup, alert history |
| Layer | Technology |
|---|---|
| Backend API | FastAPI + Uvicorn |
| Scraping | Playwright (headless Chromium), BeautifulSoup, lxml |
| Workflow Orchestration | Prefect |
| AI / LLM | LangChain + Azure OpenAI (gpt-4o-mini) |
| Database | SQLite via SQLAlchemy ORM |
| Frontend | React 18 + TypeScript + Vite |
| Charts | Recharts |
| UI Components | Radix UI + TailwindCSS |
| State Management | Zustand + TanStack React Query |
| Python smtplib (Gmail / any SMTP) | |
| Container | Docker + nginx |
| Process Manager | PM2 |
- Python 3.11+
- Node.js 18+
- An Azure OpenAI resource with a
gpt-4o-minideployment
git clone https://github.com/zeeshanparwez/telecom-competitor-intelligence-.git
cd telecom-competitor-intelligence
cp .env.example .env
# Fill in your Azure OpenAI credentialsdocker compose up --buildFrontend → http://localhost:2720 · Backend → http://localhost:2719
# Backend
cd backend
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
playwright install chromium
python -m api.main # starts on port 2719
# Frontend (new terminal)
cd frontend
npm install
npm run dev # starts on port 2720npm install # installs pm2
npm start # pm2 start ecosystem.config.cjs
npm run logs # tail all logs# .env (copy from .env.example)
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o-mini
AZURE_OPENAI_API_VERSION=2024-10-21
SQLITE_DB_PATH=../data/telecom_intel.db
API_PORT=2719
FRONTEND_URL=http://localhost:2720# From the dashboard sidebar → "Trigger Scrape" button
# Or via API:
curl -X POST http://localhost:2719/api/scrape/trigger
# Or via script (cron-ready):
./scripts/run_scrape.shcurl -X POST http://localhost:2719/api/pricing/recommend \
-H "Content-Type: application/json" \
-d '{"plan_type":"prepaid","circle":"PAN India","target_segment":"budget"}'
# Or via script:
./scripts/run_pricing.sh prepaid "PAN India" budget# Add to crontab -e — runs every day at 6am
0 6 * * * /path/to/telecom-competitor-intelligence/scripts/run_scrape.sh >> /var/log/telecomiq.log 2>&1make install # pip install + npm install
make install-playwright # install Chromium browser for scraping
make backend # start FastAPI server
make frontend # start Vite dev server
make dev # start both via PM2
make test # run pytest suite
make lint # run ESLint on frontend
make docker-up # docker compose up --build -d
make docker-down # docker compose down
make scrape # trigger a scrape runFull contract in docs/API_CONTRACT.md · Interactive docs at http://localhost:2719/docs
| Method | Endpoint | Description |
|---|---|---|
GET |
/health |
Service health + last scrape time |
GET |
/api/stats |
Plan counts + weekly market movements |
GET |
/api/plans |
Active plans (filterable by provider, type, circle) |
GET |
/api/changes |
Full market change log |
POST |
/api/scrape/trigger |
Launch a scrape run |
GET |
/api/scrape/status |
Latest scrape run status |
POST |
/api/pricing/recommend |
Generate AI pricing recommendation |
GET |
/api/pricing/latest |
Most recent recommendation |
GET |
/api/analytics/overview |
Provider stats + price band breakdown |
GET |
/api/analytics/value-metrics |
GB/₹ efficiency across all plans |
GET |
/api/analytics/opportunities |
Price gaps + underserved segments |
GET |
/api/analytics/competitive-matrix |
Value leadership by band + provider |
GET |
/api/alerts/config |
Get alert configuration |
POST |
/api/alerts/config |
Save alert configuration |
POST |
/api/alerts/test |
Send a test alert email |
telecom-competitor-intelligence/
│
├── backend/
│ ├── api/
│ │ ├── main.py # FastAPI app, CORS, lifespan
│ │ ├── schemas.py # Pydantic request/response models
│ │ └── routes/ # plans · changes · pricing · scrape · analytics · alerts
│ ├── db/
│ │ ├── models.py # ORM models + compute_plan_hash()
│ │ ├── connection.py # SQLAlchemy engine + SessionLocal
│ │ └── migrations/ # Reference SQL schema
│ ├── flows/
│ │ ├── scrape_flow.py # Prefect: concurrent scrape → persist
│ │ └── pricing_flow.py # Prefect: gap analysis → LLM → save
│ ├── pricing_engine/
│ │ ├── gap_analyzer.py # Finds ₹50+ price gaps, calculates sweet spots
│ │ ├── elasticity_scorer.py # Opportunity score per price band
│ │ └── llm_strategist.py # LangChain chain → structured PricingStrategy
│ ├── scrapers/
│ │ ├── base_scraper.py # Abstract base: Playwright + JS extraction + user-agent rotation
│ │ ├── jio_scraper.py
│ │ ├── airtel_scraper.py
│ │ └── vi_scraper.py
│ ├── services/
│ │ └── email_service.py # HTML email builder + SMTP sender
│ ├── tests/ # pytest suite (conftest, test_health, test_models)
│ ├── Dockerfile
│ └── requirements.txt
│
├── frontend/
│ ├── src/
│ │ ├── pages/ # 9 dashboard pages
│ │ ├── components/ # cards/ · charts/ · layout/
│ │ ├── hooks/ # React Query data-fetching hooks
│ │ ├── lib/ # api.ts · utils.ts · constants.ts
│ │ ├── store/ # Zustand filter state
│ │ └── types/ # TypeScript interfaces
│ ├── Dockerfile # nginx multi-stage build
│ ├── nginx.conf # SPA fallback + /api proxy
│ └── package.json
│
├── scripts/
│ ├── run_scrape.sh # Cron-ready scrape trigger
│ └── run_pricing.sh # Pricing recommendation CLI
│
├── data/ # SQLite DB lives here (gitignored)
├── docker-compose.yml
├── ecosystem.config.cjs # PM2 config
├── Makefile
├── .env.example
└── docs/API_CONTRACT.md
telecom_plans — live plan snapshot (is_active flag, plan_hash index)
plan_change_log — append-only market event log (new_plan · price_drop · price_hike · removed · perk_change)
pricing_recommendations — AI strategy history with confidence scores
scrape_runs — execution log per scrape (status · plans_found · changes_detected)
alert_config — SMTP config + per-event-type toggles + ₹ threshold
alert_log — email delivery history (sent · failed · test)
MIT © Zeeshan Parwez