The problem
A team builds retrieval into its product: a corpus, a chunking strategy, an embedding model, sometimes a reranker. The question then becomes whether the whole thing works. In the absence of qrels, three behaviours are observed, and none of them holds up.
- Measure nothing. Chunk size,
top_k, and the embedding model are tuned by guesswork. Every architectural decision is a bet whose outcome stays invisible. - Measure end to end. An automated evaluation of the final answer shows a drop, without distinguishing a retrieval step that missed the right chunk from a generation step that misused it. Optimisation then targets the wrong layer.
- Annotate by hand. Sorting some fifty candidates takes an expert eight to ten minutes. Across a hundred questions, that is a few days of their time. The real obstacle is their availability: annotation comes after their day-to-day work, takes weeks to complete, and is never repeated once the corpus changes.
The reference set also has a property the components it evaluates do not: it stays valid when you change embedding model, chunking, or reranker. It is a durable asset, where the retriever it measures is bound to be replaced.
The deliverable
An archive, delivered as a download link. No integration, no SDK, no environment to set up.
qrels.csv — the reference set
One row per (question, evaluated candidate chunk) pair, including candidates judged not relevant: their presence attests that the candidate pool was actually examined, and they serve as usable negatives for training a reranker.
| Column | Type | Description |
|---|---|---|
query_id | str | Identifier taken from questions.csv |
chunk_id | str | Identifier taken from chunks.csv |
rank | int | Chunk's rank for this question, starting at 1 |
grade | int | Relevance grade: 2 highly relevant, 1 relevant, 0 not relevant |
score | float | Continuous relevance score, for fine-grained tie-breaking |
confidence | float | Reliability index of the judgment on this chunk, in [0,1] |
qrels.trec
The same content in the standard query_id 0 chunk_id grade format,
directly consumable by ir_measures and pytrec_eval.
uncertain.csv
Questions on which the system failed to converge, or whose best candidate scores low — typically an ambiguous question, or one with no answer in the corpus. We prefer to bound the uncertainty and flag it to you rather than display a uniform confidence that does not exist.1
report.html and metrics.json
Overall and per-question reliability index; distribution of the number of relevant
chunks per question; estimated coverage of the candidate pool2; and the exact command
to compute your own metrics. The same values are repeated in structured form in
metrics.json, to track how a run evolves from one to the next.
Data handling
Nothing is kept beyond delivery. Each run executes inside a dedicated infrastructure, hosted in the European Union, created for that one job and destroyed as soon as the results are delivered. Corpus, questions, intermediate representations and working files vanish with it.
- A confidentiality agreement is signed before any transfer, with no prior negotiation.
- Your data is only ever used to produce your deliverable. It is never used to train or fine-tune any model.
- The language models used are, without exception, run by us inside the European Union. No third-party model API is involved. This is not an option enabled on request: it is the only configuration in which we operate the service. They run on the infrastructure of one of our providers: Verda (Helsinki, Finland), OVHcloud (Roubaix, France), Scaleway (Paris, France), or Hetzner (Gunzenhausen, Germany). None of your content is ever submitted to a model operated outside the Union, and therefore never to a provider subject to the US CLOUD Act.
- Running on your own infrastructure is available in the Enterprise plan, including offline (see Pricing).
- Only the deliverable persists, for the lifetime of the download link (X days), after which it too is deleted. Any later re-run therefore requires a new submission.
Pricing
| Plan | Price | Questions | Corpus | Includes | Turnaround* |
|---|---|---|---|---|---|
| Discovery | 900 | 20 | 2,000 | graded top-10, summary report. Amount fully credited toward the next plan ordered within thirty days | 72 h |
| Starter | 2,500 | 100 | 20,000 | graded top-10, full reliability report, uncertain cases flagged | 5 days |
| Standard | 7,500 | 500 | 100,000 | graded top-20, your retriever scored against the reference set, gap to the state of the art, one-hour debrief | 10 days |
| Enterprise | from 25,000/yr | on quote | on quote | runs on your own infrastructure, quarterly refresh, contractual service commitment | contractual |
Add-ons: generation of representative questions (€1,500 per hundred); scoring an additional retriever (€750); refreshing the reference set after the corpus changes (half price); a half-day prioritisation workshop (€1,800).
Why Goldset?
Validation
A ground-truth reference set is only worth something if its reliability is measured. We distinguish three claims, each established separately.
| Claim | Protocol | Result |
|---|---|---|
| Labels reproduce human judgment | Public benchmarks with human qrels (BEIR, mMARCO fr), with a candidate pool identical to the one used in production | Ranking derived from our labels, scored against human qrels: nDCG@10 = 98%, versus 92% for the best state-of-the-art retriever on the same pool |
| The property holds on a domain-specific corpus | Blind sample, independently annotated by two domain experts | Agreement with the expert: 99.98%, with two experts agreeing with each other at 98.2% |
| The reference set ranks systems correctly | Eight retrieval configurations ranked with human qrels, then with ours | Kendall's τ = 0.993: the two rankings coincide |
The third claim is the most important in practice. The real use of a reference set is not to rule on a single document, but to decide whether one configuration is better than another — and that is also the level at which this kind of judgment is most reliable. No annotation method, human ones included, guarantees the accuracy of any single label taken in isolation.
Comparison with alternatives
The question is not whether free solutions exist — they do — but what they actually produce. An open generator prompts a model to write a question from each chunk: the resulting pair is relevant by construction, but it contains only one relevant chunk per question, whereas several answer it in a real corpus. Any recall metric built on that is distorted: finding the single source chunk yields perfect recall, while any other relevant chunk retrieved counts as a miss.3
| Open generator | Human annotation | Goldset | |
|---|---|---|---|
| Relevant chunks per question | 1, by construction | every one in the pool | every one in the pool |
Recall (recall@k) | a first, rough estimate | yes | yes |
Fine-grained metrics (nDCG@k) | no: a single chunk, binary relevance | yes, if the annotation is graded | yes, relevance graded from 0 to 2 |
| Measured reliability | none | inter-annotator agreement | reliability index, validated against human judgments |
| Cost for 100 questions | ≈ 0, with no question validation | a few days of a domain expert's time, subject to availability | €2,500 |
| Reissued as the corpus evolves | in theory | in practice, never | yes, at half price |
Market landscape
The previous table contrasted categories of methods; this one names actual tools. Most belong to the same family as "open generator" — a set of questions generated from the corpus, with at most one relevant chunk per question — and a few automate relevance evaluation without building that question set themselves. None cover both steps with a measured reliability guarantee.
| Tool | Kind | Covers | Doesn't cover |
|---|---|---|---|
| RAGAS | open source | Generates a question set from a knowledge graph built over the corpus | One relevant chunk per question, by construction; published human validation limited to a single study |
| Giskard / RAGET | open source | Generates varied question types, then evaluates with a language model | Same recall limitation; no published human correlation |
| LlamaIndex / DeepEval | open source | Generates question/context pairs directly from the index | One relevant chunk per question, by construction |
| UMBRELA | open source | Reference relevance assessor, a published reproduction of an assessor protocol | A research building block: no candidate pool, no pipeline, no deliverable |
| RAGELO | open source | Ranks RAG systems against each other by comparing their answers | Does not build a corpus's qrels; it compares systems, not documents |
| ARES | open source (research) | Fine-tuned evaluators, with statistical confidence intervals | Engineering work to build in-house; not a delivered service |
| Bedrock / Vectara / Galileo / Patronus | managed, proprietary | Evaluates language-model-generated answers | Evaluates an answer, not a retrieval reference set; hosted outside the EU |
| Toloka / Scale AI | managed, human annotation | Manual annotation at scale | Costly and slow on a confidential, specialised corpus; never repeated |
Each of these tools covers part of the need; none covers all of it. What sets Goldset apart:
- Each question is evaluated against several candidate chunks from your corpus, not just one. That is what makes the measured recall meaningful.
- Each chunk gets a relevance grade and a reliability index.
- Its labels reproduce human judgment (nDCG@10 = 98% on public benchmarks), without taking up your experts' time.
- The deliverable is ready to use, with no integration.
- Processing is hosted in the European Union.
The difference lies in combining all five points, not in any one of them alone.
In short, for your leadership
To know whether an AI assistant finds the right documents, you need a reference set of correct answers. Building it by hand ties up a domain expert for several days they rarely have free: the project waits, or moves ahead unmeasured. Goldset delivers this reference set without taking up your experts' time, for €2,500 excl. tax per hundred questions, with data processed exclusively in the European Union and deleted after delivery.
Frequently asked questions
- How do I know your reference set is correct?
Our labels are validated on public benchmarks with human judgments. They have also been validated on a domain-specific corpus annotated by domain experts: performance there is the same, which shows the method generalises beyond benchmarks. A check on your own corpus remains possible, though not necessary: fifty annotations from one of your experts are enough for us to give you the agreement rate between them and us. And the deliverable is not binary: every chunk carries a relevance grade and a reliability index, and non-converging questions are isolated in a separate file.
- Isn't evaluation by a language model biased?
Taken naively, yes, and that is documented. That is precisely the problem our protocol addresses — and it is also what makes the service worth paying for: if the result could be obtained with one prompt and one model, there would be nothing to sell. The choice of models used is not adjusted case by case: it systematically targets the best quality available, spread across several independent families, which mechanically limits the risk that a judgment favours one particular system. What we expose are the measured effects: reliability index, flagged borderline cases and, if you wish, agreement with your own experts.
- How many questions are needed?
Fifty for a first signal. A hundred to two hundred to compare two configurations with an interpretable gap. Five hundred and above to segment by question type and pinpoint where failures occur — which is usually where the useful information is.
- We don't have a set of questions.
Start with your real questions, even a small number: usage logs, tickets, support requests. These are the best ones, because automatically generated questions are systematically better-formed than real users' questions. Failing that, we can generate them, explicitly covering several axes — task type, complexity, phrasing style.
- What happens when the corpus changes?
The reference set stays valid for the unchanged part. A refresh only reprocesses the questions affected by the modified chunks, at half price.
- What file format?
Two UTF-8 encoded CSV files.
questions.csvwith the columnsidandquestion;chunks.csvwithid,text, and optionallysource. If the files don't validate, you receive the details of the offending rows within the hour.
Notes
- The proportion of flagged questions is small in practice, but it is not zero, and we do not try to hide it: an ambiguous question, or one whose answer is absent from the corpus, should be identified as such rather than given an arbitrary label that would silently distort your metrics.
- The coverage of the candidate pool mechanically bounds the measurable recall: a relevant chunk missing from the pool is not labelled, so it cannot be counted. This is the exercise's structural limit; it is shown in every delivery.
- This is a recognised limitation of this family of methods, not an implementation flaw of any particular tool.
- Description based on each tool's public documentation at the time of writing; these projects move fast.