Skip to content

Commit 4538178

Browse files
authored
Enhance ReplicationTutorial with workflow tools and practices
Added sections on other workflow management tools and best practices for experiment management, including unique experiment IDs, logging, error handling, resource management, and results organization.
1 parent 77a7774 commit 4538178

1 file changed

Lines changed: 100 additions & 1 deletion

File tree

ReplicationTutorial.md

Lines changed: 100 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -487,7 +487,106 @@ def main():
487487
if __name__ == '__main__':
488488
main()
489489
```
490-
#### Workflow Management Tools
490+
491+
#### Other Workflow Management Tools
492+
493+
* [Snakemake](https://snakemake.readthedocs.io/en/stable/)
494+
* [Sacred](https://sacred.readthedocs.io/en/stable/)
495+
* [SLURM]
496+
497+
#### Best Practices
498+
499+
1. Unique Experiment IDs
500+
501+
Always generate unique identifiers:
502+
503+
```python
504+
import uuid
505+
from datetime import datetime
506+
507+
# Timestamp-based
508+
exp_id = f"exp_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
509+
510+
# UUID-based
511+
exp_id = f"exp_{uuid.uuid4().hex[:8]}"
512+
513+
# Parameter-based
514+
exp_id = f"lr{lr}_bs{bs}_seed{seed}"
515+
```
516+
517+
1. Logging and Checkpointing
518+
519+
```python
520+
import logging
521+
from pathlib import Path
522+
523+
def setup_logging(exp_dir):
524+
"""Setup logging for experiment"""
525+
log_file = exp_dir / 'experiment.log'
526+
527+
logging.basicConfig(
528+
level=logging.INFO,
529+
format='%(asctime)s - %(levelname)s - %(message)s',
530+
handlers=[
531+
logging.FileHandler(log_file),
532+
logging.StreamHandler()
533+
]
534+
)
535+
536+
# In experiment script
537+
setup_logging(exp_dir)
538+
logging.info(f"Starting experiment with params: {params}")
539+
```
540+
541+
1. Error Handling
542+
543+
```python
544+
def safe_run_experiment(params):
545+
"""Run experiment with error handling"""
546+
try:
547+
result = run_experiment(params)
548+
return {'success': True, 'result': result}
549+
except Exception as e:
550+
logging.error(f"Experiment failed: {e}")
551+
return {'success': False, 'error': str(e)}
552+
```
553+
554+
1. Resource Management
555+
556+
```python
557+
import psutil
558+
import time
559+
560+
def monitor_resources(interval=60):
561+
"""Monitor CPU and memory usage"""
562+
while True:
563+
cpu = psutil.cpu_percent(interval=1)
564+
mem = psutil.virtual_memory().percent
565+
logging.info(f"CPU: {cpu}%, Memory: {mem}%")
566+
time.sleep(interval)
567+
568+
# Run in background thread
569+
import threading
570+
monitor_thread = threading.Thread(target=monitor_resources, daemon=True)
571+
monitor_thread.start()
572+
```
573+
574+
1. Results Organization
575+
576+
```
577+
experiments/
578+
├── exp_001_20250101_120000/
579+
│ ├── params.json
580+
│ ├── metrics.json
581+
│ ├── model.pkl
582+
│ ├── logs/
583+
│ │ └── training.log
584+
│ └── plots/
585+
│ └── learning_curve.png
586+
├── exp_002_20250101_130000/
587+
│ └── ...
588+
└── summary.csv
589+
```
491590

492591
## Replication Package
493592

0 commit comments

Comments
 (0)