Skip to content

AGENTS.md

Latest commit

 

History

History
456 lines (363 loc) · 16 KB

AGENTS.md

File metadata and controls

456 lines (363 loc) · 16 KB
Raw

AGENTS.md - wsn_server

This file provides essential information for AI coding agents working on the wsn_server project.

Project Overview

wsn_server is a Django-based server application for collecting, storing, and managing data from Wireless Sensor Networks (WSN) and IoT devices. It is developed by Spectraphilic for environmental monitoring research, particularly focused on Arctic and high-latitude environments.

Key Features

  • Multi-source data ingestion: Supports Waspmote motes (4G, Iridium satellite), Campbell CR6 data loggers, LI-COR gas analyzers, EddyPro processed files, and Sommer sensors
  • Dual database architecture: PostgreSQL for operational data, ClickHouse for time-series analytics
  • Asynchronous processing: Celery workers for handling incoming data streams
  • RESTful API: For data upload, querying, and quality control workflows
  • Dynamic schema handling: FlexModel extracts JSON fields into dedicated PostgreSQL columns on demand; ClickHouse tables auto-alter to add new columns

Technology Stack

Component Technology
Backend Framework Django 6.0 (Python 3.13)
Primary Database PostgreSQL
Analytics Database ClickHouse
Task Queue Celery 5.6 with Redis (singleton backend)
API Framework Django REST Framework 3.17
Process Management Supervisor
Web Server Nginx + Uvicorn (ASGI)
Monitoring Monit
Deployment Ansible
Frontend Build Vite 7.x (Node.js, optional)
Linting Ruff

Project Structure

wsn_server/
├── project/              # Django project configuration
│   ├── settings_django.py # Django default template (do not edit)
│   ├── settings_ansible.py # Generated by Ansible (do not edit/commit)
│   ├── settings.py       # Project-specific settings (edit this)
│   ├── settings_local.py # Local overrides (do not commit)
│   ├── settings_test.py  # Test-specific overrides
│   ├── urls.py           # Root URL configuration
│   ├── urls_ansible.py   # Generated by Ansible
│   ├── asgi.py           # ASGI application (with optional Channels)
│   ├── wsgi.py           # WSGI application
│   ├── celery.py         # Celery configuration
│   └── dbrouters.py      # Database routing (PostgreSQL vs ClickHouse)
│
├── api/                  # REST API application
│   ├── urls.py           # API endpoint definitions
│   ├── views_wsn.py      # WSN data upload/query views
│   ├── views_wsn_iridium.py # Alternate Iridium view (unused)
│   ├── views_qc.py       # Quality control API views
│   ├── serializers.py    # DRF serializers
│   └── permissions.py    # API permission classes
│
├── wsn/                  # Core WSN application
│   ├── models.py         # Frame, Metadata models (PostgreSQL)
│   ├── admin.py          # Django admin configuration
│   ├── tasks.py          # Celery tasks (async processing)
│   ├── clickhouse.py     # ClickHouse client wrapper
│   ├── upload.py         # Data upload utilities
│   ├── utils.py          # Shared utilities
│   ├── parsers/          # File parsers
│   │   ├── base.py       # Base parser classes
│   │   ├── waspmote.py   # Waspmote binary frame parser
│   │   ├── cr6.py        # Campbell CR6 CSV parser
│   │   ├── eddypro.py    # EddyPro output parser
│   │   ├── licor.py      # LI-COR .ghg file parser
│   │   ├── sommer.py     # Sommer sensor CSV parser
│   │   └── schemas.py    # Field type schemas for ClickHouse
│   └── management/commands/  # Custom Django management commands
│
├── apps/                 # Additional Django applications
│   ├── ch/               # ClickHouse models (managed + unmanaged)
│   ├── boot/             # Legacy (source removed, only __pycache__)
│   └── myadmin/          # Legacy (source removed, only __pycache__)
│
├── qc/                   # Quality Control application
│   ├── models.py         # Site, Node, Data models
│   ├── admin.py          # Django admin configuration
│   └── views.py          # Empty placeholder
│
├── ansible/              # Deployment configuration
│   ├── development.yml   # Local development playbook
│   ├── production.yml    # Production deployment playbook
│   ├── vars.yml          # Common Ansible variables
│   ├── hosts-local       # Local inventory
│   ├── hosts-production  # Production inventory
│   └── templates/        # Configuration templates
│
├── tests/                # Test suite
│   ├── conftest.py       # pytest fixtures and configuration
│   ├── test_qc.py        # Quality control API tests
│   └── wsn/              # WSN tests
│       ├── test_api.py   # API endpoint tests
│       ├── test_commands.py # Management command tests
│       └── test_parsers.py # Parser unit tests
│
├── etc/                  # Runtime configuration files
│   ├── requirements.txt  # Python dependencies
│   ├── supervisor.conf   # Supervisor configuration
│   ├── gunicorn.conf.py  # Gunicorn configuration
│   ├── nginx.conf        # Nginx configuration
│   ├── start.sh          # Start Supervisor script
│   └── stop.sh           # Stop Supervisor script
│
├── mods/                 # Modular Django settings/plugins
│   ├── admin/            # Admin URLs and settings
│   ├── dotenv/           # .envrc loading
│   ├── postgres/         # PostgreSQL settings
│   ├── uvicorn/          # Uvicorn logging settings
│   └── ...               # Other optional modules
│
├── var/                  # Runtime data (logs, uploads, static files)
│   ├── log/              # Application logs
│   ├── data/             # Archived frame data
│   ├── static/           # Collected static files
│   ├── media/            # User uploads
│   └── run/              # PID and socket files
│
├── data/                 # Sample/test data files
├── src/                  # Frontend source (admin.js for Vite)
├── config.toml           # Data import configuration
├── manage.py             # Django management script
├── Makefile              # Build automation
├── pytest.ini            # pytest configuration
├── ruff.toml             # Ruff linter configuration
├── package.json          # Node.js/Vite frontend build
└── vite.config.js        # Vite bundler configuration

Build and Development Commands

Local Development Setup

# Install system dependencies (Debian/Ubuntu)
apt install git make postgresql python3-venv rabbitmq-server redis
apt install ansible

# Create PostgreSQL database
su - postgres
createuser -e -P wsn
createdb -e -O wsn wsn

# Clone and install
git clone https://github.com/spectraphilic/wsn_server.git
cd wsn_server
make local

# Activate virtual environment and create superuser
source venv313/bin/activate
python manage.py createsuperuser

# Start development server
make start

Key Make Commands

Command Description
make local Install/update local development environment via Ansible
make local-requirements Update pip dependencies only
make install-server Install/update server environment via Ansible
make deploy-production Remote deployment via Ansible
make run Run development server with uvicorn (--reload)
make start Start Supervisor (runs Django + Celery)
make stop Stop Supervisor
make ctl Run supervisorctl for process management
make reload Reload all programs in Supervisor

Testing

# Run all tests
pytest

# Run with verbose output
pytest -v

# Run specific test file
pytest tests/test_qc.py
pytest tests/wsn/test_api.py
pytest tests/wsn/test_parsers.py

Frontend Build (Optional)

The project includes an optional Vite-based frontend build for admin JavaScript:

# Install Node.js dependencies
npm install

# Development server
npm run dev

# Production build
npm run build

Build outputs to var/build/. The src/admin.js entry point is configured in vite.config.js.

Code Style Guidelines

Python Style

  • Follow PEP 8
  • Use 4 spaces for indentation
  • Maximum line length: 100 characters (flexible)
  • Import order: standard library → third-party → Django → project
  • The project uses Ruff for linting; configuration is in ruff.toml

Import Organization

# Standard Library
import datetime
import json
from pathlib import Path

# Third-party requirements
import pytz
from celery import shared_task

# Django
from django.db import models
from django.conf import settings

# Project
from wsn.models import Frame
from wsn.parsers.base import CSVParser

Naming Conventions

Type Convention Example
Classes PascalCase FrameSerializer, CR6Parser
Functions/Methods snake_case parse_frame(), get_value()
Variables snake_case metadata, frame_count
Constants UPPER_SNAKE_CASE SENSORS, OPEN_KWARGS
Private _leading_underscore _parse_header()

Django Patterns

  • Models: Define Meta class, use __str__ methods
  • Views: Use class-based views for APIs (DRF)
  • Serializers: Explicit Meta classes with fields
  • Tasks: Celery tasks use @shared_task decorator with retry configuration

Testing Instructions

Test Configuration

Tests are configured in pytest.ini:

[pytest]
DJANGO_SETTINGS_MODULE = project.settings_test
addopts = --reuse-db --tb=short
pythonpath = apps
testpaths = tests/

Key behaviors:

  • Reuses the test database across runs (--reuse-db)
  • Adds apps/ to the Python path
  • Uses project.settings_test which sets CELERY_TASK_MAX_RETRIES = 0, CLICKHOUSE_NAME = 'test_wsn', and WSN_DATA_DIR = '/tmp'

Fixtures and Design (tests/conftest.py)

  • Session-scoped fixtures are used for speed
  • Non-transactional tests are preferred (transactional tests truncate all tables)
  • celery_session_worker is used instead of celery_worker to avoid database connection-closing issues (celery/celery#4511)
  • django_db_blocker is unblocked to allow Celery workers database access
  • Celery runs synchronously in tests (task_always_eager = True, task_eager_propagates = True) so errors appear as normal tracebacks

Key fixtures:

  • api: Unauthenticated APIClient wrapper around endpoints (/api/create/, /api/query/postgresql/, /api/query/clickhouse/, /api/iridium/, /getpost_frame_parser.php)
  • api_user: Authenticated client using a Django REST Framework Token for user api
  • django_db_setup: Creates the api user and token in the test DB using get_or_create

Writing Tests

import pytest
from rest_framework.test import APIClient

@pytest.fixture
def api_client():
    client = APIClient()
    client.credentials(HTTP_AUTHORIZATION='Token <token>')
    return client

def test_example(api_client, db):
    response = api_client.get('/api/query/postgresql/')
    assert response.status_code == 200

Database Dependencies

  • Use db fixture for tests that modify the database
  • Use django_db_setup for module-scoped database initialization
  • Tests reuse database (--reuse-db) by default
  • ClickHouse-dependent tests are skipped if port 9000 is unreachable (see tests/wsn/test_commands.py)

Security Considerations

API Authentication

  • Token Authentication: Used for general API access (Django REST Framework Token)
  • API Key Authentication: Used for quality-control endpoints (djangorestframework-api-key)
  • Permission Classes:
    • IsUserAPI restricts certain endpoints to the Django user named api
    • APIKeyTest / with_api_key() validate custom predicates against API keys
    • Default DRF permission is IsAdminUser

Sensitive Configuration

Store in environment variables (via .envrc file, not committed):

WSN_POSTGRESQL_PASSWORD=
WSN_CLICKHOUSE_PASSWORD=
WSN_CIPHER_KEY=          # For Waspmote frame decryption

Production Security

  • DEBUG = False in production
  • HTTPS enforced (configured via Ansible)
  • Secret key generated by Ansible (not committed)
  • API keys managed via Django admin

Database Architecture

PostgreSQL (Primary)

Models:

  • wsn.Metadata: Device/site information with JSONB tags field (GIN indexed)
  • wsn.Frame: Sensor data frames with timestamp, metadata reference, JSON data field, and dynamically extracted columns (bat, RECORD, TiltX_Avg, TiltY_Avg, etc.)
  • qc.Site/qc.Node/qc.Data: Quality control workflow data with QC boolean flags

Key features:

  • FlexModel abstract base for dynamic field extraction from JSON
  • TimeModelMixin for timestamp formatting
  • GIN index on Metadata.tags for fast JSON queries
  • Frame.create() upserts frames and merges split frames across multiple transmissions

ClickHouse (Analytics)

Managed models (apps/ch/models.py):

  • FinsefetenePostproc: Post-processed eddy covariance data
  • FinsefluxPostproc: Post-processed eddy covariance data
  • MobilefluxPostproc: Mobile flux station data (includes CH4)
  • Myr1Postproc/Myr2Postproc: Myr site post-processed data (includes CH4)

Unmanaged models (apps/ch/models_unmanaged.py):

  • High-frequency data tables (*_HFData)
  • Biometeorological data (*_Biomet)
  • Station status/diagnostics (*_StationStatus)
  • Read-only access to existing ClickHouse tables

Routing via ClickHouseRouter in project/dbrouters.py.

Data Ingestion Flow

Waspmote via 4G (Meshlium)

  1. POST to /getpost_frame_parser.php
  2. MeshliumView creates envelope with payload
  3. Celery tasks: archive.delay() (filesystem) + in_meshlium.delay() (parsing)
  4. in_meshlium parses binary frames, saves to PostgreSQL

Waspmote via Iridium

  1. POST to /api/iridium/
  2. IridiumView processes RockBLOCK payload
  3. Similar Celery workflow with additional Iridium metadata (IMEI, MOMSN, location)

Campbell CR6 / LI-COR / EddyPro / Sommer

  1. Manual or scripted import using import_file management command
  2. File parsed using appropriate parser class
  3. Data uploaded to PostgreSQL and/or ClickHouse based on configuration

Common Development Tasks

Adding a New Parser

  1. Create class in wsn/parsers/ inheriting from BaseParser or CSVParser
  2. Implement _load(), _parse_header(), _parse_time() methods
  3. Add tests with sample data files
  4. Update schema in schemas.py if needed for ClickHouse

Adding API Endpoints

  1. Add URL pattern in api/urls.py
  2. Create view in api/views_*.py
  3. Add serializer in api/serializers.py if needed
  4. Configure permissions appropriately
  5. Add tests

Adding Celery Tasks

from celery import shared_task
from celery_singleton import Singleton

@shared_task(
    acks_late=True,
    autoretry_for=(Exception,),
    max_retries=None,
    default_retry_delay=300,
    base=Singleton,  # For singleton tasks
    unique_on=['param_name'],
)
def my_task(param):
    pass

Deployment

Production deployment uses Ansible:

# Configure ansible/hosts-production
make deploy-production

Services managed:

  • Nginx (reverse proxy, SSL termination)
  • Uvicorn (ASGI server)
  • Celery workers
  • Supervisor (process management)
  • Monit (monitoring)

Settings Load Order

1. settings_django.py  : Defaults from Django (do not edit)
2. settings_ansible.py : Generated by Ansible (do not edit/commit)
3. settings.py         : Project-specific settings (edit this)
4. settings_local.py   : Local settings (do not commit)

Useful Resources