# Independent reference-query pilot evaluation

- Run window UTC: 2026-10-28T13:18:42.117274+00:00 to 2026-10-28T13:18:42.482437+00:00
- Corpus: 52 installed system-doc files; no user/project files indexed.
- Queries: 21 natural-language requests with predeclared, source-based relevance grades.
- Judgments: hand-curated file-level reference labels (2=primary, 1=supporting), not independent human labels.
- Comparison: BM25-style length-normalized term scoring versus a TF-IDF baseline.

## Aggregate results

| Ranking method | Precision@5 | Recall@5 | MRR | nDCG@5 | Median query ms |
|---|---:|---:|---:|---:|---:|
| BM25 | 0.390 | 0.746 | 0.790 | 0.675 | 0.2031 |
| TF-IDF | 0.276 | 0.532 | 0.639 | 0.434 | 0.1448 |

## Per-query results

| Query ID | BM25 P@5 | TF-IDF P@5 | BM25 nDCG@5 | TF-IDF nDCG@5 |
|---|---:|---:|---:|---:|
| shell-grammar | 0.60 | 0.40 | 0.81 | 0.54 |
| shell-builtins | 0.40 | 0.20 | 0.62 | 0.39 |
| shell-completion | 0.40 | 0.20 | 0.96 | 0.52 |
| git-restore | 0.40 | 0.40 | 0.88 | 0.43 |
| git-revision | 0.40 | 0.00 | 1.00 | 0.00 |
| git-packaging | 0.00 | 0.40 | 0.00 | 0.66 |
| openssl-csr | 0.60 | 0.40 | 0.70 | 0.39 |
| openssl-inspect-cert | 0.40 | 0.40 | 0.39 | 0.43 |
| openssl-provider | 0.40 | 0.40 | 0.70 | 0.82 |
| python-venv | 0.40 | 0.40 | 0.96 | 0.96 |
| systemd-unit | 0.60 | 0.00 | 0.75 | 0.00 |
| systemd-environment | 0.40 | 0.20 | 0.59 | 0.09 |
| systemd-logs | 0.20 | 0.40 | 0.24 | 0.58 |
| ssh-config | 0.40 | 0.20 | 0.63 | 0.08 |
| ssh-tunnel | 0.60 | 0.20 | 0.82 | 0.73 |
| ssh-dns | 0.40 | 0.20 | 0.49 | 0.83 |
| rsync-transfer | 0.20 | 0.20 | 0.41 | 0.28 |
| curl-http | 0.40 | 0.40 | 0.82 | 0.39 |
| archive-tar | 0.20 | 0.20 | 0.83 | 0.28 |
| archive-gzip | 0.60 | 0.60 | 0.82 | 0.71 |
| find-predicates | 0.20 | 0.00 | 0.73 | 0.00 |

## Human-review status

A blind review pack has been generated at `independent-review-pack.csv`. It contains the query and shuffled unique candidate documents returned by either ranking method, with blank relevance and notes fields and no ranker labels.
No independent human review has been performed by this run. The source-based grades are a provisional reference standard, and aggregate quality metrics are only as reliable as those grades.

## Decision rule

- Do not choose a ranking method based on this pilot alone; inspect per-query failures and obtain independent grades for the blind pack.
- The compact-array versus set benchmark is a storage/latency choice, separate from BM25 versus TF-IDF ranking quality.
- Keep the method configurable and preserve the per-query output for reproducibility.