lucerna eval

Evaluate search quality against a set of labelled queries. Useful for tuning embedding models and search parameters. Defaults to the current working directory.

lucerna eval queries.jsonl --k 1,5,10
lucerna eval queries.jsonl --dir /path/to/project --k 1,5,10

Query file format

One JSON object per line:

{ "query": "function that verifies JWT tokens", "expectedFile": "src/auth.ts", "expectedSymbol": "verifyToken" }
{ "query": "database connection pool", "expectedFile": "src/db/pool.ts" }
{ "query": "retry logic with exponential backoff", "expectedFile": "src/utils/retry.ts", "expectedSymbol": "withRetry" }

expectedSymbol is optional — omit it to match on file only.

Options

Option	Description
--dir <path>	Project root directory (default: current working directory)
--k <numbers>	Comma-separated k values to evaluate (default: 1,5,10)
--format raw|json|pretty-json	Output format (default: raw)
--no-semantic	Lexical search only

Example output

Evaluation results — 24 queries

  Recall@1  : 54.2%  (13/24)
  Recall@5  : 87.5%  (21/24)
  Recall@10 : 95.8%  (23/24)

Per-query breakdown:
  [@1:✓  @5:✓  @10:✓]  "function that verifies JWT tokens"  →  src/auth/middleware.ts::verifyToken
  [@1:✗  @5:✓  @10:✓]  "database connection pool"  →  src/db/pool.ts
  [@1:✗  @5:✗  @10:✓]  "retry logic with exponential backoff"  →  src/utils/retry.ts::withRetry
  [@1:✓  @5:✓  @10:✓]  "user repository find by email"  →  src/db/UserRepository.ts::findByEmail