tests: Add test runner scripts - #1248
Conversation
Running functional tests, nccl-tests, and sanitizer checks currently requires each developer to maintain their own ad-hoc scripts with hardcoded paths. This makes it easy to miss validation steps before submitting PRs and difficult for AI agents to run tests autonomously. Add a tests/scripts/ directory with reusable, Slurm-agnostic runners that cover the common test workflows (functional, nccl-tests, ASAN, Valgrind). Environment is configured via a gitignored env.sh so user-specific paths stay out of the repo. A Slurm batch template is provided for clusters that use it. Also includes an AGENTS.md with detailed instructions for AI agents on how to run each test type and interpret results. Signed-off-by: Bibrak Qamar Chandio <bibracha@amazon.com>
e7bcb9b to
d37c590
Compare
|
bot:aws:retest |
1 similar comment
|
bot:aws:retest |
| NUM_NODES=${NUM_NODES:-1} | ||
| RANKS_PER_NODE=${RANKS_PER_NODE:-8} | ||
| NUM_GPUS_PER_RANK=${NUM_GPUS_PER_RANK:-1} | ||
| EXTRA_TEST_ARGS=${EXTRA_TEST_ARGS:-"-n 15 -w 10 -b 1K -e 16G -f 2 -c 1 -R 0"} |
There was a problem hiding this comment.
-n 15 -w 10: we usually run with different options -- why are these default?
| export NCCL_BUFFSIZE=${NCCL_BUFFSIZE:-8388608} | ||
| export NCCL_P2P_NET_CHUNKSIZE=${NCCL_P2P_NET_CHUNKSIZE:-524288} |
There was a problem hiding this comment.
We no longer recommend setting these options.
|
|
||
| # Defaults | ||
| MODE=${MODE:-regular} | ||
| PROTOCOL=${PROTOCOL:-RDMA} |
There was a problem hiding this comment.
Shouldn't the default protocol be empty -- let plugin pick?
There was a problem hiding this comment.
Should this file have nccl-tests in the name? It isn't applicable to the functional tests, whose run scripts are also in this folder.
| - Exit code 0 (the script propagates mpirun's exit code directly) | ||
| - Output contains `Results: N/N passed` | ||
| - No `NCCL_OFI_WARN` in output | ||
| - No segfaults or aborts |
There was a problem hiding this comment.
Nit: that should also be covered by "exit code 0"
| fi | ||
|
|
||
| # Check correctness in output files | ||
| OUTPUTS=$(find "$CHECK_DIR" -name "output.txt" -o -name "slurmout_*.txt" 2>/dev/null) |
There was a problem hiding this comment.
In run-slurm.sh.example, the default output file name is slurm-%j.txt. So, I think we should find slurm-*.txt instead of slurmout_*.txt.
| OUTPUTS=$(find "$CHECK_DIR" -name "output.txt" -o -name "slurmout_*.txt" 2>/dev/null) | ||
| if [[ -z "$OUTPUTS" ]]; then | ||
| echo "WARNING: No output files found in $CHECK_DIR" | ||
| exit 0 |
There was a problem hiding this comment.
If no output is found, then should we exit 1?
| BW=$(grep "Avg bus bandwidth" "$LOGFILE" | awk '{print $NF}') | ||
| echo "Avg bus bandwidth: ${BW:-N/A}" |
There was a problem hiding this comment.
In order for line 144 to be useful, we should change line 143 to
BW=$(grep "Avg bus bandwidth" "$LOGFILE" | awk '{print $NF}' || true)
|
bot:aws:retest |
5 participants
Description of changes:
Running functional tests, nccl-tests, and sanitizer checks currently requires each developer to maintain their own ad-hoc scripts with hardcoded paths. This makes it easy to miss validation steps before submitting PRs and difficult for AI agents to run tests autonomously.
Add a tests/scripts/ directory with reusable, Slurm-agnostic runners that cover the common test workflows (functional, nccl-tests, ASAN, Valgrind). Environment is configured via a gitignored env.sh so user-specific paths stay out of the repo. A Slurm batch template is provided for clusters that use it.
Also includes an AGENTS.md with detailed instructions for AI agents on how to run each test type and interpret results.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.