Skip to content

Commit fc1c8cb

Browse files
bobbypatonbobbypaton
committed
Update benchmarks with gXTB, UMA, and RF results
- Update README benchmarking section with correct values from new CSV (526 molecules across xTB/gXTB/UMA, plus RF baseline) - Add model_benchmarks/README.md with detailed per-method tables - Add benchmark text outputs (gxtb_run, uma_run, rf_run, xtb_run) - Remove redundant gxtb warning print from predictions - Mention examples directory in main README
1 parent 9d3cb95 commit fc1c8cb

7 files changed

Lines changed: 466 additions & 28 deletions

File tree

‎README.md‎

Lines changed: 25 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,8 @@ python -m deltahf predict \
190190
-o predictions.csv
191191
```
192192

193+
See the [examples/](examples/) directory for additional scripts covering batch prediction, model comparison, and atom classification.
194+
193195
## Benchmarking
194196

195197
### Running Benchmarks
@@ -210,21 +212,21 @@ python benchmark.py --methods uma --append
210212
python benchmark.py --methods rf --append # requires: pip install molpipeline
211213
```
212214

213-
Results are cached per method × n_conformers in `.benchmark_cache/`, so re-runs skip expensive optimizations. See `benchmark_results.csv` for the full output.
215+
Results are cached per method × n_conformers in `.benchmark_cache/`, so re-runs skip expensive optimizations. See [`model_benchmarks/`](model_benchmarks/) for detailed results and the full CSV output.
214216

215217
### Key Findings
216218

217219
A benchmark across all eight models and three methods (n_conformers = 1, 531 molecules) reveals four main findings:
218220

219221
1. **Bond-order classification outperforms hybridization** — Using maximum bond order (1/2/3) from the Kekulized structure instead of RDKit hybridization labels improves accuracy at both the coarse level (`bondorder` vs `hybrid`) and the fine-grained level (`bondorder_ext` vs `extended`). The best model overall is `bondorder_ext`, combining bond-order labels with per-carbon H-counts.
220222

221-
2. **Model chemistry matters far more than parameterisation** — Upgrading from xTB to gXTB (wB97M-V/def2-TZVPPD single-points on xTB geometries) reduces RMSD by ~42% (6.58 → 3.82 kcal/mol for `bondorder_ext`). Upgrading to UMA (MLIP, GPU) reduces it by a further ~10% (3.82 → 3.45 kcal/mol).
223+
2. **Model chemistry matters far more than parameterisation** — Upgrading from xTB to gXTB (wB97M-V/def2-TZVPPD single-points on xTB geometries) reduces RMSD by ~42% (6.53 → 3.82 kcal/mol for `bondorder_ext`). Upgrading to UMA (MLIP, GPU) reduces it by a further ~32% (3.82 → 2.59 kcal/mol).
222224

223-
3. **Number of conformers has minimal impact** — On the original CHNO dataset, increasing from n=1 to n=5 gave negligible accuracy gains (<1% RMSD change) at a cost of 3–4× more computation. Use `--n-conformers 1` (the default).
225+
3. **Number of conformers has minimal impact for xTB and gXTB** — Increasing from n=1 to n=5 gives negligible accuracy change at 3–4× the cost. For UMA, n=5 actually *degrades* accuracy (RMSD 2.59 → 3.11 for `bondorder_ext`), likely because UMA's more flexible potential finds lower-energy geometries that have isomerized. Use `--n-conformers 1` (the default).
224226

225227
4. **xTB + bondorder_ext matches or exceeds the published DFT-B baseline** — On the 102-molecule Cawkwell2021 subset, xTB + `bondorder_ext` (RMSD = 6.91 kcal/mol) approaches the DFT-B + `element_bo` result (RMSD = 6.08 kcal/mol) from Cawkwell et al.<sup>1</sup>, while gXTB and UMA substantially surpass it.
226228

227-
5. **The physics prior from xTB is essential** — On the original 313-molecule CHNO training set, a pure cheminformatics baseline (Morgan fingerprint + Random Forest) achieved an in-sample RMSD of 8.89 kcal/mol but a cross-validated RMSD of 22.9 kcal/mol — dramatically worse than even the simplest xTB model (`element`, CV RMSD = 11.4 kcal/mol). The atom equivalent approach sidesteps this by encoding the quantum mechanical energy decomposition directly.
229+
5. **The physics prior from xTB is essential** — A pure cheminformatics baseline (Morgan fingerprint + Random Forest) achieved an in-sample RMSD of 11.70 kcal/mol but a cross-validated RMSD of 30.9 kcal/mol — dramatically worse than even the simplest xTB model (`element`, CV RMSD = 11.9 kcal/mol). The atom equivalent approach sidesteps this by encoding the quantum mechanical energy decomposition directly.
228230

229231
> **Note on the `neighbour` model:** The `neighbour` model shows competitive training-set RMSD but produces an extremely large cross-validation RMSD in some CV folds (near-singular design matrix). It is not recommended for practical use.
230232
@@ -234,15 +236,15 @@ Full training set (531 molecules, C,H,N,O,F,S,Cl), n_conformers = 1:
234236

235237
| Model | Params | xTB RMSD | xTB MAD | gXTB RMSD | gXTB MAD | UMA RMSD | UMA MAD |
236238
|-------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
237-
| `element` | 7 | 11.53 | 8.16 | 4.36 | 3.17 | 4.11 | 2.44 |
238-
| `element_bo` | 11 | 9.12 | 5.78 | 4.06 | 2.80 | 3.88 | 2.10 |
239-
| `hybrid` | 13 | 8.93 | 5.51 | 4.10 | 2.87 | 3.59 | 1.87 |
240-
| `bondorder` | 13 | 7.66 | 5.02 | 3.95 | 2.75 | 3.58 | 1.85 |
241-
| `bondorder_ar` | 16 | 7.63 | 4.98 | 3.93 | 2.74 | 3.55 | 1.82 |
242-
| `extended` | 18 | 8.45 | 5.38 | 3.98 | 2.76 | 3.51 | 1.83 |
243-
| **`bondorder_ext`** | **19** | **6.58** | **4.29** | **3.82** | **2.67** | **3.45** | **1.84** |
239+
| `element` | 7 | 11.54 | 8.16 | 4.37 | 3.18 | 3.27 | 2.32 |
240+
| `element_bo` | 11 | 9.12 | 5.78 | 4.07 | 2.81 | 2.99 | 1.98 |
241+
| `hybrid` | 13 | 8.92 | 5.50 | 4.11 | 2.88 | 2.78 | 1.74 |
242+
| `bondorder` | 13 | 7.63 | 5.00 | 3.95 | 2.75 | 2.73 | 1.71 |
243+
| `bondorder_ar` | 16 | 7.60 | 4.96 | 3.94 | 2.74 | 2.71 | 1.67 |
244+
| `extended` | 18 | 8.44 | 5.37 | 3.98 | 2.76 | 2.69 | 1.69 |
245+
| **`bondorder_ext`** | **19** | **6.53** | **4.26** | **3.82** | **2.67** | **2.59** | **1.66** |
244246

245-
All values in kcal/mol. Params = active parameters after dropping zero-count types. Adj. R² and CV RMSD available in `benchmark_results.csv`.
247+
All values in kcal/mol. Params = active parameters after dropping zero-count types. Adj. R² and CV RMSD available in [`model_benchmarks/benchmark_results.csv`](model_benchmarks/benchmark_results.csv).
246248

247249
### Results: Comparison with DFT-B Literature Baseline
248250

@@ -253,22 +255,24 @@ Cawkwell2021 subset (102 molecules), n_conformers = 1. The DFT-B baseline<sup>1<
253255
| DFT-B (lit.)<sup>1</sup> | `element` | 4 | 7.59 | 25.48 |
254256
| DFT-B (lit.)<sup>1</sup> | `element_bo` | 7 | 6.08 | 15.01 |
255257
| xTB | `bondorder_ext` | 15 | 6.91 | 23.66 |
256-
| gXTB | `bondorder_ext` | 15 | 3.74 | 10.52 |
258+
| gXTB | `bondorder_ext` | 15 | 3.75 | 10.52 |
257259
| **UMA** | **`bondorder_ext`** | **15** | **2.70** | **8.95** |
258260

259261
### Results: Effect of n_conformers
260262

261-
`bondorder_ext` model, original CHNO training set (310 molecules). gXTB timing is from fresh runs (uncached).
263+
`bondorder_ext` model, full training set (526 molecules). Timings are from fresh runs (uncached).
264+
265+
| n_conformers | xTB RMSD | gXTB RMSD | UMA RMSD | xTB time | gXTB time | UMA time |
266+
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
267+
| 1 | 6.53 | 3.82 | 2.59 | 54 s | 69 s | 285 s |
268+
| 3 | 6.51 | 3.81 | 2.58 | 104 s | 119 s | 707 s |
269+
| 5 | 6.52 | 3.83 | 3.11\* | 151 s | 166 s | 1095 s |
262270

263-
| n_conformers | xTB RMSD | gXTB RMSD | UMA RMSD | gXTB wall time |
264-
|:---:|:---:|:---:|:---:|:---:|
265-
| 1 | 4.59 | 3.15 | 2.48 | 78 s |
266-
| 3 | 4.61 | 3.15 | 2.48 | 163 s |
267-
| 5 | 4.65 | 3.15 | 2.43 | 245 s |
271+
\*UMA n=5 degrades due to isomerization of some conformers during UMA geometry optimization (max deviation jumps from 20 to 41 kcal/mol).
268272

269273
### Recommendations
270274

271-
- **Use `--n-conformers 1`** (the default) — additional conformers give <1% RMSD improvement at 3–4× cost
275+
- **Use `--n-conformers 1`** (the default) — additional conformers give negligible improvement for xTB/gXTB and can degrade UMA results
272276
- **Use `--model bondorder_ext`** for best accuracy
273277
- **Use `--optimizer uma`** for best accuracy if a GPU is available
274278
- **Use `--use-gxtb`** for substantially improved accuracy over xTB on CPU alone
@@ -295,7 +299,7 @@ For each molecule, deltahf performs the following steps:
295299
- [tqdm](https://tqdm.github.io/) — progress bars
296300

297301
**Optional:**
298-
- [gxtb](https://github.com/grimme-lab/gxtb) — Higher-level DFT energies (wB97M-V/def2-TZVPPD). **Must be installed manually from source** (not available via pip/conda). **WARNING:** gxtb and xTB energies are on completely different scales and cannot be mixed - see [GXTB_USAGE.md](GXTB_USAGE.md) for proper usage.
302+
- [gxtb](https://github.com/grimme-lab/gxtb) — **Must be installed manually from source** (not available via pip/conda). **WARNING:** gxtb and xTB energies are on completely different scales and cannot be mixed - see [GXTB_USAGE.md](GXTB_USAGE.md) for proper usage.
299303

300304
## Testing
301305

‎deltahf/__main__.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -288,7 +288,6 @@ def _verify_gxtb(use_gxtb: bool) -> None:
288288
print(" gxtb must be installed manually from source (not available via conda/pip).")
289289
sys.exit(1)
290290
print(" Using gxtb energies (wB97M-V/def2-TZVPPD approximation)")
291-
print(" WARNING: gxtb and xTB energies are on different scales - do not mix workflows!")
292291
print()
293292

294293

‎model_benchmarks/README.md‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
# Model Benchmarks
2+
3+
This directory contains the benchmark script, results CSV, and text outputs from evaluating `deltahf` across all eight atom equivalent models, three model chemistries (xTB, gXTB, UMA), and three conformer counts (1, 3, 5).
4+
5+
---
6+
7+
## Commands
8+
9+
All commands were run from the `model_benchmarks/` directory.
10+
11+
```bash
12+
# xTB (CPU, 8 threads)
13+
python benchmark.py --methods xtb --xtb-threads 8
14+
15+
# gXTB (CPU, 8 threads) — appends to existing CSV
16+
python benchmark.py --methods gxtb --append --xtb-threads 8
17+
18+
# UMA (GPU) — appends to existing CSV
19+
python benchmark.py --methods uma --append
20+
21+
# RF baseline (no quantum chemistry) — appends to existing CSV
22+
python benchmark.py --methods rf --append
23+
```
24+
25+
Approximate wall times (531 molecules, n_conformers = 1/3/5):
26+
27+
| Method | n=1 | n=3 | n=5 | Hardware |
28+
|--------|----:|----:|----:|----------|
29+
| xTB | 54 s | 104 s | 151 s | CPU (8 threads) |
30+
| gXTB | 69 s | 119 s | 166 s | CPU (8 threads) |
31+
| UMA | 285 s | 707 s | 1095 s | GPU (CUDA) |
32+
| RF | <1 s | — | — | CPU |
33+
34+
---
35+
36+
## Dataset
37+
38+
**Input:** `deltahf/data/training_data.csv` — 531 molecules (C, H, N, O, F, S, Cl).
39+
40+
**Successfully processed:** 526 / 531 molecules (5 failed: 3 conformer generation failures, 1 sulfur compound, 1 isomerization).
41+
42+
Two evaluation subsets are reported:
43+
44+
| Subset | Molecules | Description |
45+
|--------|:---------:|-------------|
46+
| `all` | 526 | Full training set (all successfully processed molecules) |
47+
| `cawkwell` | 102 | Cawkwell2021 subset only (enables comparison with published DFT-B baseline) |
48+
49+
**Literature baseline:** Cawkwell et al. (2021) reported `element` RMSD = 7.59, `element_bo` RMSD = 6.08 kcal/mol using DFT-B (DFTB+ geometries and energies).
50+
51+
---
52+
53+
## Results: Full Training Set (526 molecules, n_conformers = 1)
54+
55+
### xTB
56+
57+
| Model | Params | Adj. R² | RMSD | MAD | Max dev | CV RMSD |
58+
|-------|-------:|--------:|-----:|----:|--------:|--------:|
59+
| element | 7 | 0.9533 | 11.54 | 8.16 | 71.76 | 11.87 |
60+
| element_bo | 11 | 0.9706 | 9.12 | 5.78 | 68.70 | 9.59 |
61+
| hybrid | 13 | 0.9717 | 8.92 | 5.50 | 52.96 | 9.59 |
62+
| bondorder | 13 | 0.9793 | 7.63 | 5.00 | 39.88 | 8.18 |
63+
| **bondorder_ext** | **19** | **0.9847** | **6.53** | **4.26** | **39.40** | **7.09** |
64+
| bondorder_ar | 16 | 0.9794 | 7.60 | 4.96 | 39.96 | 8.20 |
65+
| extended | 18 | 0.9744 | 8.44 | 5.37 | 52.66 | 9.04 |
66+
| neighbour | 28 | 0.9778 | 7.79 | 5.00 | 46.03 | 8.79 |
67+
68+
### gXTB
69+
70+
| Model | Params | Adj. R² | RMSD | MAD | Max dev | CV RMSD |
71+
|-------|-------:|--------:|-----:|----:|--------:|--------:|
72+
| element | 7 | 0.9933 | 4.37 | 3.18 | 22.56 | 4.47 |
73+
| element_bo | 11 | 0.9941 | 4.07 | 2.81 | 21.25 | 4.19 |
74+
| hybrid | 13 | 0.9940 | 4.11 | 2.88 | 20.31 | 4.31 |
75+
| bondorder | 13 | 0.9945 | 3.95 | 2.75 | 21.41 | 4.17 |
76+
| **bondorder_ext** | **19** | **0.9948** | **3.82** | **2.67** | **19.99** | **4.11** |
77+
| bondorder_ar | 16 | 0.9945 | 3.94 | 2.74 | 21.45 | 4.17 |
78+
| extended | 18 | 0.9943 | 3.98 | 2.76 | 20.37 | 4.23 |
79+
| neighbour | 28 | 0.9949 | 3.75 | 2.62 | 24.48 | 4.27 |
80+
81+
### UMA
82+
83+
| Model | Params | Adj. R² | RMSD | MAD | Max dev | CV RMSD |
84+
|-------|-------:|--------:|-----:|----:|--------:|--------:|
85+
| element | 7 | 0.9962 | 3.27 | 2.32 | 21.27 | 3.37 |
86+
| element_bo | 11 | 0.9968 | 2.99 | 1.98 | 21.00 | 3.12 |
87+
| hybrid | 13 | 0.9972 | 2.78 | 1.74 | 20.89 | 2.93 |
88+
| bondorder | 13 | 0.9974 | 2.73 | 1.71 | 20.10 | 2.88 |
89+
| **bondorder_ext** | **19** | **0.9976** | **2.59** | **1.66** | **20.06** | **2.78** |
90+
| bondorder_ar | 16 | 0.9974 | 2.71 | 1.67 | 20.28 | 2.87 |
91+
| extended | 18 | 0.9974 | 2.69 | 1.69 | 20.89 | 2.88 |
92+
| neighbour | 28 | 0.9976 | 2.56 | 1.61 | 21.32 | 2.85 |
93+
94+
### Cross-method comparison (bondorder_ext, best overall)
95+
96+
| Method | RMSD | MAD | Max dev | CV RMSD | Adj. R² |
97+
|--------|-----:|----:|--------:|--------:|--------:|
98+
| xTB | 6.53 | 4.26 | 39.40 | 7.09 | 0.9847 |
99+
| gXTB | 3.82 | 2.67 | 19.99 | 4.11 | 0.9948 |
100+
| UMA | 2.59 | 1.66 | 20.06 | 2.78 | 0.9976 |
101+
102+
---
103+
104+
## Results: Cawkwell2021 Subset (102 molecules, n_conformers = 1)
105+
106+
This subset enables direct comparison with the published DFT-B results from Cawkwell et al. (2021).
107+
108+
### Comparison with literature baseline
109+
110+
| Method | Model | Params | RMSD | Max dev |
111+
|--------|-------|-------:|-----:|--------:|
112+
| DFT-B (lit.) | element | 4 | 7.59 | 25.48 |
113+
| DFT-B (lit.) | element_bo | 7 | 6.08 | 15.01 |
114+
| xTB | bondorder_ext | 15 | 6.91 | 23.66 |
115+
| gXTB | bondorder_ext | 15 | 3.75 | 10.52 |
116+
| **UMA** | **bondorder_ext** | **15** | **2.70** | **8.95** |
117+
118+
### Full model breakdown (Cawkwell subset)
119+
120+
| Model | xTB RMSD | gXTB RMSD | UMA RMSD |
121+
|-------|:--------:|:---------:|:--------:|
122+
| element | 16.97 | 5.31 | 3.91 |
123+
| element_bo | 13.64 | 4.30 | 3.16 |
124+
| hybrid | 12.23 | 4.73 | 3.18 |
125+
| bondorder | 8.83 | 3.90 | 2.95 |
126+
| bondorder_ext | 6.91 | 3.75 | 2.70 |
127+
| bondorder_ar | 8.75 | 3.85 | 2.93 |
128+
| extended | 11.84 | 4.59 | 3.15 |
129+
| neighbour | 7.48 | 3.56 | 2.86 |
130+
131+
---
132+
133+
## Results: Effect of n_conformers
134+
135+
`bondorder_ext` model, full training set (526 molecules). All timings from fresh (uncached) runs.
136+
137+
### RMSD (kcal/mol)
138+
139+
| n_conformers | xTB | gXTB | UMA |
140+
|:---:|:---:|:---:|:---:|
141+
| 1 | 6.53 | 3.82 | 2.59 |
142+
| 3 | 6.51 | 3.81 | 2.58 |
143+
| 5 | 6.52 | 3.83 | 3.11* |
144+
145+
### Max deviation (kcal/mol)
146+
147+
| n_conformers | xTB | gXTB | UMA |
148+
|:---:|:---:|:---:|:---:|
149+
| 1 | 39.40 | 19.99 | 20.06 |
150+
| 3 | 39.35 | 19.99 | 20.05 |
151+
| 5 | 39.33 | 20.02 | 40.85* |
152+
153+
\*UMA n=5 shows degraded accuracy. The max deviation doubles from ~20 to ~41 kcal/mol, and RMSD increases from 2.59 to 3.11. This is likely because UMA's more flexible potential energy surface finds lower-energy conformer geometries that have actually isomerized (changed connectivity) for some molecules, but pass the connectivity check. The xTB and gXTB optimizers do not exhibit this behaviour.
154+
155+
**Recommendation:** Use `--n-conformers 1` (the default) for all methods.
156+
157+
---
158+
159+
## RF Baseline (No Quantum Chemistry)
160+
161+
A Morgan fingerprint + Random Forest baseline was evaluated to quantify the value of the physics-based xTB energy. This model uses no quantum chemistry — only 2D molecular structure.
162+
163+
| Subset | Molecules | In-sample RMSD | CV RMSD |
164+
|--------|:---------:|:--------------:|:-------:|
165+
| all | 531 | 11.70 | 30.92 |
166+
| cawkwell | 102 | 11.56 | 31.13 |
167+
168+
The RF baseline's CV RMSD (30.9 kcal/mol) is ~2.6× worse than even the simplest xTB atom equivalent model (`element`, CV RMSD = 11.9 kcal/mol) and ~11× worse than UMA + `bondorder_ext` (CV RMSD = 2.78 kcal/mol). This demonstrates that the xTB/gXTB/UMA energy provides an essential physics prior that simple cheminformatics descriptors cannot replace.
169+
170+
---
171+
172+
## Notes on the `neighbour` model
173+
174+
The `neighbour` model achieves competitive training-set RMSD (often the best or near-best for each method), but its cross-validation RMSD is unreliable on the smaller Cawkwell subset: CV RMSD values of 994 (xTB), 18665 (gXTB), and 18664 (UMA) indicate a near-singular design matrix in some CV folds. On the full 526-molecule training set the CV RMSD is reasonable (8.79 for xTB, 4.27 for gXTB, 2.85 for UMA), but the instability on smaller datasets makes it unsuitable for general use. The `bondorder_ext` model is recommended instead.
175+
176+
---
177+
178+
## Discussion: Best Model Choice
179+
180+
The `bondorder_ext` model consistently achieves the best or near-best accuracy across all three methods:
181+
182+
- **xTB:** bondorder_ext is clearly best (RMSD 6.53 vs next-best bondorder at 7.63)
183+
- **gXTB:** bondorder_ext is best by a narrow margin (RMSD 3.82 vs neighbour at 3.75, but neighbour has CV instability)
184+
- **UMA:** bondorder_ext is best among stable models (RMSD 2.59 vs neighbour at 2.56, but neighbour has CV instability on smaller datasets)
185+
186+
The key advantage of bond-order classification over hybridization is that it distinguishes atoms by their bonding environment more finely. For example, a carbonyl carbon (C=O, bond order 2) is classified differently from an aromatic carbon (also sp2 in RDKit), and the hydrogen-count extension further differentiates methyl (CH3) from methylene (CH2) from methine (CH) carbons.
187+
188+
For production use, **`bondorder_ext` + UMA** (GPU) gives the best accuracy (RMSD 2.59 kcal/mol, CV RMSD 2.78). On CPU, **`bondorder_ext` + gXTB** (RMSD 3.82, CV RMSD 4.11) is the best option, and **`bondorder_ext` + xTB** (RMSD 6.53, CV RMSD 7.09) is the most accessible with no optional dependencies.
189+
190+
---
191+
192+
## Files
193+
194+
| File | Description |
195+
|------|-------------|
196+
| `benchmark.py` | Benchmark script |
197+
| `benchmark_results.csv` | Full results (149 rows: 8 models × 3 methods × 3 n_conformers × 2 subsets + DFT-B + RF) |
198+
| `xtb_run` | Text output from xTB benchmark run |
199+
| `gxtb_run` | Text output from gXTB benchmark run |
200+
| `uma_run` | Text output from UMA benchmark run |
201+
| `rf_run` | Text output from RF baseline run |
202+
| `.benchmark_cache/` | Cached pipeline results (enables fast re-runs) |
203+
| `old/` | Previous benchmark results (prior to training data expansion) |

0 commit comments

Comments
 (0)