Skip to content

CLAUDE.md

Latest commit

 

History

History
358 lines (265 loc) · 12.9 KB

CLAUDE.md

File metadata and controls

358 lines (265 loc) · 12.9 KB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

StoryMapJS is a tool for telling stories through the context of places on a map. The project consists of:

  • JavaScript library (src/js/) - the client-side StoryMap viewer built with Leaflet
  • Python Flask backend (storymap/) - web server for the StoryMap editor with Google OAuth authentication
  • Build system - Webpack for JS bundling and LESS compilation

Development Setup

JavaScript Library Development

Build the JavaScript library:

npm install
npx webpack -c webpack.dev.js

For production builds:

npm run build        # Build for production
npm run dist         # Clean, build, and copy to dist/
npm run stage        # Build and stage to a specific version
npm run stage_latest # Build and stage to 'latest'
npm run stage_dev    # Build and stage to 'dev'

Test changes by running a local web server and navigating to templates, e.g., http://localhost:8000/src/template/arya.html

Python Server Development

The Python Flask application requires Docker and LocalStack for S3 emulation.

Prerequisites:

  • Docker installed
  • AWS CLI installed with local profile configured in ~/.aws/credentials: 
    [local]
region=us-east-1
endpoint-url=http://localhost:4566
aws_access_key_id=localstack
aws_secret_access_key=localstack

Initial setup:

# 1. Generate SSL certificates
scripts/makecerts.sh

# 2. Create .env file from template
cp dotenv.example .env
# Fill in GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET (required for login)

# 3. Build and start services
docker compose build
docker compose up

# 4. In separate terminals, create S3 buckets and database tables
scripts/makebuckets.sh
scripts/create-tables.sh  # Password: storymap

Access the application at https://localhost (accept self-signed certificate).

Testing

The project uses hatch for test environment management with pytest.

Available test environments:

  1. Unit Tests (no Docker required):
hatch run unit:test              # Run unit tests
hatch run unit:test-cov          # Run with coverage report
hatch run unit:test-watch        # Run in watch mode
  1. Integration Tests (requires Docker Compose running):
# First, ensure Docker stack is running:
docker compose up

# Then run integration tests:
hatch run integration:test       # Run integration tests
hatch run integration:test-slow  # Run with detailed output
  1. Development Environment (all tools):
hatch run dev:unit               # Run unit tests
hatch run dev:integration        # Run integration tests
hatch run dev:all                # Run all tests
hatch run dev:lint               # Lint code with ruff
hatch run dev:format             # Format code with ruff

Test markers:

  • @pytest.mark.unit - Unit tests (fast, no external dependencies)
  • @pytest.mark.integration - Integration tests (require Docker stack)
  • @pytest.mark.slow - Slow-running tests

Running specific tests:

hatch run unit:test tests/unit_tests.py::test_specific_function
hatch run integration:test -k test_save_from_data

Note: Current test suite is minimal and needs expansion.

Architecture

JavaScript Library (src/)

The library is organized into modular components:

  • src/js/core/ - Core StoryMap logic and data structures
  • src/js/map/ - Map rendering (primarily Leaflet-based)
  • src/js/slider/ - Slide navigation UI components
  • src/js/media/ - Media handling (images, video, etc.)
  • src/js/ui/ - UI components (navigation, controls)
  • src/js/language/ - Internationalization (i18n) with locale JSON files
  • src/js/animation/ - Animation utilities
  • src/less/ - LESS stylesheets

Entry point: src/js/index.js

Build output: dist/js/storymap.js (exposed as KLStoryMap global)

Python Application (storymap/)

Flask-based web application with the following key modules:

  • storymap/api.py - Main Flask application with all API routes and editor endpoints
  • storymap/storage.py - S3 storage abstraction layer using boto3
  • storymap/connection.py - PostgreSQL database connection and user management
  • storymap/googleauth.py - Google OAuth authentication
  • storymap/tasks.py - Huey task queue for async operations (e.g., cleanup)
  • storymap/core/settings.py - Configuration loaded from environment variables
  • storymap/core/wsgi.py - WSGI application entry point

The application uses:

  • PostgreSQL (port 5432) - User data and StoryMap metadata stored as JSONB
  • LocalStack (port 4566) - Local S3 emulation for development
  • Huey - Task queue (separate container in docker-compose)
  • Gunicorn - WSGI server with SSL (port 443 → 5000)

Data Flow

  1. Users authenticate via Google OAuth (storymap/googleauth.py)
  2. User data stored in PostgreSQL users table with JSONB storymaps field
  3. StoryMap content (JSON, images) stored in S3 buckets:
    • uploads.knilab.com - User uploads
    • cdn.knilab.com - Published storymaps
  4. Editor UI loads from storymap/templates/ and compiled static from compiled/
  5. Published storymaps load the library from CDN_URL (configured in .env)

Environment Configuration

Critical environment variables (see dotenv.example):

  • FLASK_SECRET_KEY - Flask session security
  • GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET - OAuth credentials
  • AWS_STORAGE_BUCKET_NAME, AWS_STORAGE_BUCKET_KEY - S3 buckets
  • AWS_ENDPOINT_URL - Set to http://localstack:4566 for local dev
  • CDN_URL - URL for served static assets (can point to deployed CDN)
  • STATIC_URL - URL for application static files
  • FLASK_SETTINGS_MODULE - Settings module path (storymap.core.settings)
  • ADMINS - Space-separated list of admin user IDs

Docker Compose Services

  • app - Flask application (gunicorn with SSL on port 443)
  • pg - PostgreSQL 12.17 database
  • localstack - S3 emulation for local development
  • huey - Background task worker

Volumes are mounted for live code reloading: storymap/, dist/, compiled/

Key Technical Details

Static Asset Serving

The Flask app serves static JS/CSS from the compiled/ directory. For local development, it's easiest to set CDN_URL environment variable to a deployed CDN rather than building and serving locally:

CDN_URL=https://cdn.knightlab.com/libs/storymapjs/latest/

Webpack Configuration

Multiple webpack configs for different environments:

  • webpack.common.js - Shared base configuration
  • webpack.dev.js - Development build
  • webpack.prd.js - Production build (used by npm run build)
  • webpack.stg.js - Staging build

Important Constraints

  • Werkzeug version pinned to 3.0.1 - Versions >= 3.1 break image uploads in the editor
  • Python 3.12 is the target version
  • Node version specified in .nvmrc

Database Schema

The users table structure:

CREATE TABLE users (
  id serial PRIMARY KEY,
  uid varchar(32),
  uname varchar(100),
  migrated smallint,
  storymaps jsonb,
  CONSTRAINT unique_uid UNIQUE (uid)
);

AWS/S3 Operations

Use AWS CLI with LocalStack for local development:

aws --profile local --endpoint-url=http://localhost:4566 s3 ls uploads.knilab.com/storymapjs/

Language Translations

To add a new language translation, create a file like src/js/language/locale/xx.json where xx is the ISO 639-1 two-letter language code. Copy an existing translation file and translate the values (not the keys).

Map Features

StoryMapJS supports various map modes and customization:

  • map_as_image - Display image with markers (vs cartography mode with connecting lines)
  • GigaPixel - Support for high-resolution images
  • Custom markers - Use custom images/icons via use_custom_markers option
  • Line customization options: line_follows_path, line_color, line_weight, line_dash, etc.
  • calculate_zoom - Set to false to manually control zoom levels

Userinfo Endpoint

The /userinfo/ endpoint (https://storymap.knightlab.com/userinfo/) provides account and storymap information for troubleshooting user issues.

Storage Error Handling and Retry Logic

The storage layer (storymap/storage.py) implements robust error handling for S3 operations:

Boto3 Retry Configuration

The S3 client is configured with automatic retry logic for transient errors:

  • Retry mode: standard (uses exponential backoff with jitter)
  • Max attempts: 5 (1 initial attempt + 4 retries)
  • Connect timeout: 10 seconds
  • Read timeout: 60 seconds

What gets retried automatically:

  • Connection errors (including connection resets)
  • Read/connect timeouts
  • Throttling errors (429, 503)
  • Server errors (500, 502, 503, 504)

Exception Handling Hierarchy

  1. ClientError - S3-specific errors (NoSuchBucket, AccessDenied)
  2. ConnectionClosedError/ReadTimeoutError/ConnectTimeoutError - Connection interruptions (connection resets)
  3. EndpointConnectionError - Endpoint unreachable
  4. BotoCoreError - Other boto errors
  5. Exception - Unexpected errors

All exceptions are converted to StorageException with:

  • User-friendly message separated by | from instructions
  • Detailed error information in error_detail field for debugging

Monitoring Retries

To see retry attempts in production logs, enable boto3 debug logging by uncommenting in storage.py:

boto3.set_stream_logger('boto3.resources', logging.INFO)

Look for [STORAGE] prefixed log lines to identify specific exceptions:

grep "\[STORAGE\]" logs.txt

S3 Key Size Errors

Intermittent production issues during save operations related to S3 object key length:

KeyTooLongError

Error Code: KeyTooLongError Cause: S3 object key exceeds AWS limit of 1024 bytes Occurrence: Intermittent production issue during save operations

S3 object keys are constructed from {AWS_STORAGE_BUCKET_KEY}/{user_id}/{storymap_id}/.... If user IDs or storymap IDs are extremely long (corrupted data, malformed UUIDs, encoded content), the key can exceed the limit.

Diagnostic Logging: When this error occurs, storage.py logs:

  • Function name where error occurred
  • Function arguments
  • Key length in bytes (compared to 1024 byte max)
  • First 200 characters of the key

User Message: "The StoryMap identifier is too long for storage. This is usually caused by corrupted data. Please contact KnightLab support with the StoryMap ID."

Investigation: Check production logs for [STORAGE] KeyTooLongError to identify the specific keys that are too long.

RequestHeaderSectionTooLarge

Error Code: RequestHeaderSectionTooLarge Cause: HTTP request headers exceed S3's 8KB limit Occurrence: Intermittent production issue during save operations

This error indicates the total size of all HTTP headers in the PutObject request exceeds AWS S3's limit. Since the code doesn't set custom metadata, possible causes:

  1. Long S3 object keys - Keys under 1024 bytes but still very long (700-900 bytes)
  2. Large AWS signature - Signature V4 authorization header is typically 500-1000 bytes
  3. Cumulative header size - All headers (Host, Authorization, Content-Type, ACL, Cache-Control, Key, etc.) combined

Diagnostic Logging: When this error occurs, storage.py logs:

  • Function name where error occurred
  • Function arguments
  • Key length in bytes
  • First 200 characters of the key

User Message: "The StoryMap data exceeds size limits for storage. Please try reducing the amount of content, especially in text fields."

Investigation:

  • Check production logs for [STORAGE_ERROR] to see all storage errors
  • Check logs for [SAVE_ATTEMPT] and [SAVE_SUCCESS] to correlate failures with retries
  • See DEBUGGING_INTERMITTENT_SAVES.md for complete diagnostic strategy
  • IMPORTANT: We have NOT confirmed this error is related to user-reported intermittent save issues

Preventive Fix Applied

To prevent one class of potential errors, we limited StoryMap ID length. Key construction in storage.py:

  • key_prefix(*args) - '{AWS_STORAGE_BUCKET_KEY}/{args joined by /}/'
  • key_name(*args) - '{AWS_STORAGE_BUCKET_KEY}/{args joined by /}'

Potential Issue: StoryMap IDs were created from user-provided titles using slugify.slugify(title) with no length limit. Very long titles could result in keys exceeding AWS limits.

Preventive Fix (api.py:394-401): StoryMap ID generation now limits the slugified title to 200 characters maximum:

MAX_ID_LENGTH = 200
id_base = slugify.slugify(title)
if len(id_base) > MAX_ID_LENGTH:
    id_base = id_base[:MAX_ID_LENGTH]

This reserves sufficient space for:

  • Bucket key (~20 bytes)
  • User ID (~100 bytes)
  • Path components like _images/ (~50 bytes)
  • File names like draft.json (~20 bytes)
  • Suffix digits for uniqueness (~10 bytes)
  • Safety margin (~624 bytes remaining out of 1024 byte limit)

Testing Error Conditions

See TESTING_STORAGE_ERRORS.md for documentation on forcing storage errors to test the user experience.