Magazine

A distributed persistent queue for homogeneous data management in Java.

Overview

Magazine is a Java library that streamlines the management of homogeneous data requiring persistence for a duration and on-demand consumption. Inspired by the mechanics of a rifle magazine, it provides a simple and intuitive API to load, fire, and reload data, backed by a pluggable, distributed storage layer.

flowchart TD
    A1["MagazineManager"] -- "Manages instances of" --> A0["Magazine‹T›"]
    A0 -- "Delegates operations to" --> A2["BaseMagazineStorage‹T›"]
    A2 -- "Returns / Accepts" --> A3["MagazineData‹T›"]
    A2 -- "Manages state using" --> A4["MetaData"]
    A2 -- "Implements" --> A5["Sharding"]
    A2 -- "Implements" --> A6["Concurrency Control & De-duplication"]

Loading

Features

Feature	Description
Load / Fire / Reload	Queue-like semantics with pointer-based reads
Sharding	Configurable shard count for horizontal throughput
De-duplication	Optional distributed lock-based de-dup on write
Pluggable Backends	Aerospike (production-ready), HBase (planned)
Magazine Manager	Orchestrate multiple heterogeneous magazines

Quick Start

1. Add Dependency

Maven

<dependency>
    <groupId>com.phonepe</groupId>
    <artifactId>magazine</artifactId>
    <version>1.0.0</version>
</dependency>

Gradle

implementation 'com.phonepe:magazine:1.0.0'

2. Create a Magazine (Aerospike Backend)

import com.aerospike.client.AerospikeClient;
import com.phonepe.magazine.*;
import com.phonepe.magazine.impl.aerospike.*;
import com.phonepe.magazine.scope.MagazineScope;

// Connect to Aerospike
IAerospikeClient client = new AerospikeClient("localhost", 3000);

// Configure storage
AerospikeStorageConfig config = AerospikeStorageConfig.builder()
        .namespace("test")
        .dataSetName("magazine_data")
        .metaSetName("magazine_meta")
        .shards(64)
        .recordTtl(30 * 24 * 60 * 60)      // 30 days
        .metaDataTtl(2 * 30 * 24 * 60 * 60) // 60 days
        .build();

// Build storage
AerospikeStorage<String> storage = AerospikeStorage.<String>builder()
        .aerospikeClient(client)
        .storageConfig(config)
        .enableDeDupe(true)
        .farmId("dc1")
        .clazz(String.class)
        .clientId("my-service")
        .scope(MagazineScope.LOCAL)
        .build();

// Create magazine
Magazine<String> magazine = Magazine.<String>builder()
        .baseMagazineStorage(storage)
        .magazineIdentifier("notifications")
        .build();

3. Use the Magazine

// Load data
magazine.load("Hello, Magazine!");

// Fire (consume) next item
MagazineData<String> fired = magazine.fire();
System.out.println(fired.getData()); // "Hello, Magazine!"

// Reload (re-enqueue without incrementing load counter)
magazine.reload("Hello, Magazine!");

// Delete a specific record
magazine.delete(fired);

// Get metadata (counters & pointers per shard)
Map<String, MetaData> meta = magazine.getMetaData();

// Peek at specific shard/pointer combinations
Map<Integer, Set<Long>> shardPointers = Map.of(0, Set.of(1L, 2L));
Set<MagazineData<String>> peeked = magazine.peek(shardPointers);

4. Manage Multiple Magazines

MagazineManager manager = new MagazineManager("my-service");
manager.refresh(List.of(magazine));

// Retrieve by identifier
Magazine<String> m = manager.getMagazine("notifications");
m.fire();

API Overview

Class	Method	Description
Magazine<T>	load(T data)	Enqueue data
	fire()	Dequeue next item
	reload(T data)	Re-enqueue (no load-counter increment)
	delete(MagazineData<T>)	Remove a specific record
	getMetaData()	Retrieve per-shard counters & pointers
	peek(Map<Integer,Set<Long>>)	Read without consuming
MagazineManager	refresh(List<Magazine<?>>)	Register / update magazines
	getMagazine(String)	Retrieve magazine by identifier

Documentation

Full documentation is available in the docs/ directory:

• Getting Started
• Usage Guide
• API Reference
• Defaults & Configuration
• Error Codes
• Backends: Aerospike · HBase

Architecture

Contributing

We welcome contributions! Please read our Contributing Guide and Code of Conduct before submitting a pull request.

License

Licensed under the Apache License 2.0.

Copyright © 2025 PhonePe India Pvt. Ltd.