CHANGELOG.md

Changelog

All changes we make to the assignment code or PDF will be documented in this file.

[26.1.4] - 2026-05-01

• code: sharded model tests use tensors with even leading dimensions, allowing alternate solutions

[26.1.3] - 2026-04-28

• writeup: write proper FA3-style backward pass with two passes, one for dK, dV, and a separate for dQ
• writeup: enforce consistency in backward and forward algorithm notation
• writeup: encourage using the DDP test for the DDP implementation

[26.1.2] - 2026-04-28

• code: fix package existence bug for __init__.py
• code: relax equality for FSDP testing
• writeup: correct test file names for DDP

[26.1.1] - 2026-04-20

• handout: Remove H100-specific references in favor of B200 use and TMA description

[26.1.0] - 2026-04-16

• handout: clean up language, fix a few inconsistencies
• handout: fix triton code compile issue
• handout: reduce work for profiling bits

[26.0.0] - 2026-04-15

• handout: more detailed Nsight Systems profiler
• handout: replace leaderboard task with full forward and backward pass on two B200s
• handout: add section on activation checkpointing
• handout: remove excessive benchmarking, multi-GPU sweeps
• handout: add FSDP section
• handout: remove bucketed DDP

[1.0.5] - 2025-07-03

Fixed

• code: typos regarding FlashAttention2 in the adapters

Added

• code: uv build backend for building the package

[1.0.4] - 2025-04-27

Fixed

• handout: details for leaderboard submission
• handout: replace Pytorch profiler with Nsight for DDP

Added

• handout: clarify which attention implementations to benchmark

[1.0.3] - 2025-04-24

Fixed

• handout: memory_profiling (b) should sweep over context length, not model size
• handout: keyword argument in FA2 Triton starter code from tile_shape -> block_shape
• handout: minor typo in flash_backward problem statement
• handout: launch all-gather example using uv run

[1.0.2] - 2025-04-22

Added

• handout: clarify interface for flash autograd function
• handout: clarify submission for attention benchmarking

Fixed

• handout: fix small notation issues in the flash algorithm
• handout: fix math error and explanation for flash backward savings
• handout: change parameters used for attention benchmarking to be sensible

Removed

• handout: some memory benchmarking that is incompatible with modern PyTorch

[1.0.1] - 2025-04-17

Added

• code: Include tests for logsumexp in flash forward implementation
• handout: Clarify interface for flash autograd function
• code: Test causal=True for forward as well as backward

[1.0.0] - 2025-04-16

Added

• handout/code: add FlashAttention2
• handout: add proper profiling with Nsight Systems
• handout: add communication accounting
• code: tests for additional content

Changed

• handout: greatly improve demo example for Triton
• handout: remove busywork for communication

Fixed

• handout: clarify that ddp_bucketed_benchmarking doesn't require the full grid of runs.

[0.0.4] - 2024-04-23

Added

Changed

• code: remove try-finally blocks in DDP tests.

Fixed

• handout: remove outdated mention of a problem that doesn't exist on the assignment
• handout: fix Slurm environment variables in examples.
• handout: clarify assumptions in ddp_bucketed_benchmarking (b).

[0.0.3] - 2024-04-21

Added

Changed

• code: remove humanfriendly from requirements.txt, add matplotlib
• handout: modify problem distributed_communication_multi_node to specify that multinode measurements should be 2x1, 2x2, and 2x3.
• handout: clarify that torch.cuda.synchronize() is necessary for timing collective communication ops, even when they are called with async_op=False.

Fixed

• handout: fixed cut off text in problem memory_profiling (a)
• handout: fixed mismatch between slurm config and description text in section 3.2
• code: fix ToyModelWithTiedWeights to actually tie weights.
• handout: fix typo in bucketed DDP test command, should be pytest tests/test_ddp.py
• handout: fix deliverable of ddp_overlap_individual_parameters_benchmarking (a) to not ask for communication time, only end-to-end step time.
• handout: clarify analysis in optimizer_state_sharding_accounting (a).

[0.0.1] - 2024-04-17

Added

• handout: added a short question about variability on problem benchmarking_script

Changed

Fixed

• handout: fixed typo in problem triton_rmsnorm_forward. The adapters should return the classes, not the .apply attribute.
• code: added -e flag to ./cs336-systems/'[test]'
• handout: clarified recommendation about the timeit module
• handout: clarified question about kernel with highest CUDA total

[0.0.0] - 2024-04-16

Initial release.