Ground truth for evaluating a retrieval system

The problem

A team builds retrieval into its product: a corpus, a chunking strategy, an embedding model, sometimes a reranker. And then comes the question of “does it actually work?”. Without qrels, you see three behaviours - and none of them holds up.

1. Measure nothing. Chunk size, top_k, the embedding model - all tuned by eye. It's like walking through fog without a compass.
2. Measure end to end. An automated evaluation of the final answer shows a drop. But it can't tell a retrieval step that missed the right chunk from a generation step that misused it. And you end up optimising the wrong layer.
3. Annotate by hand. Sorting some fifty candidates takes an expert eight to ten minutes. A hundred questions, and that's a few days of their time. The real problem, again, is finding the person: annotation comes after everything else, takes weeks to land, and nobody does it again once the corpus moves. This is the part everyone underestimates.

The reference set also has a property the components it evaluates don't: it stays valid when you change embedding model, chunking, or reranker. It's a durable asset - where the retriever it measures, by contrast, is bound to be replaced.

The deliverable

The deliverable has three parts:

• the files;
• an interactive app for your experts to evaluate;
• optionally, fine-tuned models.

The files

An archive, sent as a download link. No integration, no SDK, no environment to set up - Simple, basic.

qrels.csv - the reference set

One row per (question, evaluated candidate chunk) pair - including the candidates judged not relevant. Their presence proves the candidate pool was actually examined, and they make useful negatives for training a reranker.

qrels.trec

The same content in the standard query_id 0 chunk_id grade format, directly consumable by ir_measures and pytrec_eval.

uncertain.csv

Questions on which the system failed to converge, or whose best candidate scores low - typically an ambiguous question, or one with no answer in the corpus. We'd rather bound the uncertainty and tell you, than show a uniform confidence that isn't there.¹

report.html and metrics.json

A reliability index, overall and per question; the distribution of the number of relevant chunks per question; the estimated coverage of the candidate pool²; and the exact command line to recompute your own metrics. The same values come back in metrics.json, in structured form, so you can track one run against the next.

fulldata.jsonl

The content of qrels.csv, enriched with the text of the questions, the content of the chunks and the reliability index of each score. Everything sits in a single file: it is the one the validation app reads, without needing your original files.

Optional: no questions? We generate them

No set of questions yet? As an option, from the Discovery plan, we generate synthetic questions from your corpus, covering several axes: task type, complexity, phrasing style. The generation method is state of the art, and the questions come with a report that rates the quality of each one: you know which to keep, review or drop.

An interactive app for your experts to evaluate

Included in the Starter plan, at no extra cost. A local web application that presents the reference set to your domain experts so they can validate the judgments, or correct them where needed. Since the processing is reliable, there is normally little to change: the job is to validate, not to redo the annotation.

It ships as a Docker image, to deploy on your own infrastructure and open to the experts of your choice. It is optional: the reference set works without it.

To run it on your fulldata.jsonl file:

docker pull goldset-verification:latest
docker run --rm -p 8080:8080 \
  -v /path/to/fulldata.jsonl:/data/fulldata.jsonl:ro \
  goldset-verification:latest /data/fulldata.jsonl

Optional: fine-tuned models

The reference set isn't only for measuring: it also makes a very good training set. As an option, we fine-tune three models on your corpus: an embedding model, a reranker and a late-interaction model (like ColBERT). We take care of data augmentation, to guarantee performance that generalises beyond the questions in the reference set.

Every delivery comes with a comparison of the performance gain against a standard retriever (hybrid BM25 + embedding), measured on your reference set.

Two delivery modes, either or both:

• the model weights, which you deploy wherever you want, offline included;
• the fine-tuned models, hosted, with their API, run by us on sovereign infrastructure in the European Union, with no logging and no retention of requests, with a one-year commitment and one retraining a year included. After 15 months of subscription (the committed year, plus 3 months), the model weights are yours: we hand them over. A no-commitment option, cancellable any month, also exists, without ownership of the weights (see Pricing).

Available from the Starter plan. Retraining available after the corpus changes, included once a year with the hosted API. Prices: see Pricing.

Measured reliability

A ground-truth reference set is only worth something if its reliability is measured. So we separate three claims, each established on its own.

The third claim is the most important in practice. What is a reference set actually for? Not to rule on a single document, but to say whether one configuration is better than another. And that's also the level at which this kind of judgment is most reliable. No annotation method - human ones included - guarantees the accuracy of any single label taken on its own.

Data handling

Nothing is kept beyond delivery. Each run executes inside dedicated infrastructure, hosted in the European Union, created for that one job and destroyed as soon as the results are delivered. Corpus, questions, intermediate representations and working files disappear with it. Deliberately, and verifiably.

• A confidentiality agreement is signed before any transfer, on simple request.
• Your data is only ever used to produce your deliverable. It is never used to train or fine-tune any model - except, as an option and at your request, the fine-tuned model delivered to you. It is never reused for any other client.
• The language models used are, without exception, run by us inside the European Union. No third-party model API is involved. This is not an option enabled on request: it is the only configuration in which we operate the service. They run on the infrastructure of one of our providers: Verda (Helsinki, Finland), OVHcloud (Roubaix, France), Scaleway (Paris, France), or Hetzner (Gunzenhausen, Germany). None of your content is ever submitted to a model operated outside the Union, and therefore never to a provider subject to the US CLOUD Act.
• Running on your own infrastructure is available in the Enterprise plan, including offline (see Pricing).
• Only the deliverable persists, for the lifetime of the download link (X days), then it's deleted too. So any later re-run starts from a fresh submission - that's the price of keeping nothing.
• One exception, if you choose the hosted API: we keep the fine-tuned model that serves it, and nothing else. Requests are neither logged nor retained.

Why Goldset?

Comparison with alternatives

You could object that free solutions exist. They do - the real question is what they produce. An open generator prompts a model to write a question from each chunk: the resulting pair is relevant by construction, but it carries only one relevant chunk per question, whereas a real corpus has several that answer it. Recall built on that is distorted: finding the single source chunk gives perfect recall, while any other relevant chunk retrieved counts as a miss.³

Market landscape

The previous table contrasted families of methods; this one names actual tools. Most belong to the same family as the “open generator” - a set of questions generated from the corpus, at most one relevant chunk per question - and a few automate relevance evaluation without building that question set themselves. None cover both steps with a measured reliability figure.

Each of these tools covers part of the need; none covers all of it. What makes Goldset different:

• Each question is evaluated against several candidate chunks from your corpus, not just one. That's what makes the measured recall meaningful.
• Each chunk gets a relevance grade and a reliability index.
• The labels reproduce human judgment (nDCG@10 = X% on public benchmarks), without taking up your experts' time.
• The deliverable is ready to use, with no integration.
• Processing is hosted in the European Union.

The difference is the combination of all five points - not any one of them alone.

Pricing

* Beta phase: turnaround times are indicative. We come back to you with an adjusted turnaround after the order.

A few purchase scenarios

• Try before committing. The free demonstration, then Discovery on 20 questions (€900). Convinced, you move to Starter within thirty days: €1,600 more. Total: €2,500 excl. tax for 100 questions - the Starter price, with a trial on top.
  What you gain: you know, before committing a budget, whether the deliverable holds up on your own documents. Then you can measure your assistant and judge every change on numbers.
• An internal assistant, no questions yet. A corpus of 8,000 chunks, no set of questions: Starter (€2,500) + generation of 100 questions at €10 (€1,000). Total: €3,500 excl. tax.
  What you gain: starting from scratch without taking up your experts' time, and measuring your assistant's quality from day one.
• Choosing between two architectures. 300 questions taken from your tickets, a 60,000-chunk corpus, your current retriever and one candidate to compare: Standard (€5,000) + one additional retriever (€750). Total: €5,750 excl. tax.
  What you gain: an architecture decision made on numbers measured on your data, not on gut feeling or a public benchmark that looks nothing like your corpus.
• Improving retrieval without managing infrastructure. Starter (€2,500) + hosted fine-tuned models and API (€300/month). First year: €6,100 excl. tax, one retraining included; the weights are yours from month 15.
  What you gain: an assistant built on models adapted to your corpus, with the gain measured against a standard retriever, no ML team to hire and no infrastructure to run - and models that end up being yours.
• A corpus that changes every quarter. Standard on a yearly subscription: the initial reference set and three refreshes, i.e. 4 runs for the price of 3 (3 × €5,000). Yearly total: €15,000 excl. tax. If everything has to run on your side, that's the Enterprise plan, on quote.
  What you gain: every quarter, you know whether your assistant's quality holds up as the documents change, and you spot a drop before your users do.

Frequently asked questions

How do I know your reference set is correct?

Our labels are validated on public benchmarks with human judgments - and on a domain-specific corpus annotated by domain experts, with the same performance. That means the method generalises beyond benchmarks. You can check it on your own corpus if you want to, without it being necessary, with the validation app included from the Starter plan: fifty annotations from one of your experts is enough, and we give you the agreement rate between them and us. And the deliverable isn't binary: every chunk carries a relevance grade and a reliability index, and the questions that don't converge are isolated in a separate file.

Isn't evaluation by a language model biased?

Taken naively, yes - and that's documented. That's precisely the problem our protocol addresses, and the reason a single call to a model isn't enough. The choice of models isn't tuned case by case; it systematically targets the best quality available, spread across several independent families, which mechanically limits the risk that a judgment favours one particular system. What we show you are the measured effects: the reliability index, the flagged borderline cases and, if you wish, agreement with your own experts.

How many questions are needed?

Fifty for a first signal. A hundred to two hundred to compare two configurations with an interpretable gap. Five hundred and above to segment by question type and pinpoint where failures occur - which is usually where the useful information is.

We don't have a set of questions.

Start with your real questions, even a small number: usage logs, tickets, support requests. These are the best ones - automatically generated questions are always too well-formed compared to what real users actually do. Failing that, we can generate them, explicitly covering several axes: task type, complexity, phrasing style - as an option from the Discovery plan, with a report rating the quality of each generated question.

What happens when the corpus changes?

The reference set stays valid for the unchanged part. To bring it up to date, we rerun the full processing on your current corpus: since we keep nothing, that is what guarantees an up-to-date result. At the plan price, or on a yearly subscription: 4 runs a year for the price of 3.

What file format?

Two UTF-8 encoded CSV files. questions.csv with the columns id and question; chunks.csv with id, text, and optionally source. If the files don't validate, you receive the details of the offending rows within 24 business hours.

Notes