How to use benchmarks in a proposal without cherry-picking

Comparison question

Decide what the benchmark must prove before looking for a number

A benchmark is useful only in relation to a stated decision. Write the proposed sentence exactly as an evaluator would read it, identify the tender criterion it answers, and record the conclusion the reader is expected to draw. "The sector first-visit completion benchmark is 82 percent" might be offered as market context, evidence that a target is ambitious, support for a proposed operating model or an implied claim that the bidder performs better. Those are different uses. A source suitable for background may be too weak for a comparative differentiator.

Separate four objects at the start: the external benchmark result, the bidder's own result, the buyer's requirement and any future offer. A sector median does not measure the bidder. A top-quartile value does not become the buyer's target. An observed peer result does not show that the offered service can reach it. Keeping the objects apart prevents a modest contextual sentence from turning into an unsupported performance claim during executive editing.

The fictional Northbank Estate Response Ltd is bidding to maintain 240 public buildings. The quality question asks how the service will reduce repeat visits for urgent repairs. A draft says: "Industry-leading providers complete 82 percent of repairs on the first visit, and our model is designed to exceed that benchmark." The cited number comes from an annual facilities survey. Before deciding whether 82 percent belongs in the answer, Northbank records the intended inference: the figure is supposed to show that its 85 percent proposed target is credible. That is a demanding use because it connects an external observation to a future commitment.

Selection protocol

Make the search auditable before a favorable result can end it

Cherry-picking often begins in the search, not the spreadsheet. If the instruction is "find an industry benchmark around 80 percent," the desired answer has already selected the evidence. Write a short protocol without a preferred value. It should state the comparison concept, eligible publication dates, sectors, geographies, organization sizes, measurement designs, publisher types, languages searched, source repositories and the date on which the search closes. Define minimum documentation, such as an accessible method and cohort description.

Build a candidate register as the search runs. Keep a row for every plausible source, including results later rejected. Record title, publisher, sponsor, edition, observation period, metric label, apparent population, original URL, discovery route, access status and disposition. Exclusion needs a method reason: wrong service, no recoverable denominator, superseded edition, duplicate data, buyer-incompatible population or unavailable method. "Lower value" and "we already found a better source" are not acceptable reasons.

The protocol can be proportionate. A minor contextual sentence may need a two-hour search of named official and professional sources. A central claim affecting a contractual target may require an analyst, service owner and independent reviewer. What matters is that another reviewer can tell whether the search gave plausible contrary evidence a fair chance. The OECD higher-education benchmarking work offers a useful discipline: candidate indicators were gathered and mapped to a conceptual framework before baseline indicators were selected for relevance, comparability and common methodology.

Northbank finds three plausible publications. One reports 82 percent for a favorable subgroup, one reports a 69 percent median under a broader work-order definition, and one reports a distribution but no directly matching repair class. All three stay in the register. The team cannot discard the second because it complicates the story. It must either explain a pre-existing eligibility difference or show that the benchmark conclusion depends on source choice.

Primary publication

Read past the benchmark headline to the publication that created it

A benchmark copied through a consultancy post or search result loses its identity. Recover the original report, exact edition, release date, observation period, table, method appendix and any questionnaire definition. Record whether the publisher produced the data, summarized member submissions, modeled several sources or repeated another study. Keep the original title and version in the citation. A current webpage can point to a report without replacing the report's method.

Identify the roles around the publication. Funding alone does not invalidate a benchmark, and an official logo does not make every use appropriate. Record sponsor, study designer, data collector, analyst, publisher and any participating vendor. Read the stated purpose. A survey built to help members diagnose operations may tolerate self-selected participation that would be unsuitable for estimating a sector norm. A marketing study may define a comparison group around customers using the sponsor's product. The method, not the institution's prestige, decides what can be inferred.

Check revision history. A publisher may correct a table, reweight the sample, redefine first-visit completion or replace a preliminary edition. Save the version reviewed and note later corrections. UK statistical guidance requires producers to explain source selection, methods, representativeness, comparability, bias and limitations. A proposal team is not an official statistics producer, but those questions are a strong test for whether a borrowed number can survive scrutiny.

Population and participation

Separate the market the report names from the organizations it actually observed

Write four populations separately. The target population is the group the report wants to describe. The frame is the list from which participation was possible. The invitation set is who was asked. The respondent set is who supplied usable data. A report titled "Global Facilities Benchmark" may describe only 68 large member organizations that chose to answer an online questionnaire. Calling the respondent result a global industry average skips every selection step.

Capture the inclusion rule, exclusions, response rate, missing items, imputation, weighting and concentration. In business surveys, one respondent can represent thousands of sites or a large share of activity. An organization-weighted mean answers a different question from a work-order-weighted mean. If participant size, sector or performance affects the decision to respond, the observed group may differ systematically from the target population. The US Census Bureau treats coverage, nonresponse, processing and measurement as separate sources of nonsampling error and cautions that ordinary response rates are generally inappropriate for self-selected samples.

Do not repair a weak cohort by choosing a flattering subgroup after seeing its value. A subgroup is defensible when the comparison dimension was in the protocol, the publisher defined it coherently, its sample remains adequate, and the subgroup resembles the buyer on factors that matter to the measure. "Public sector" may still combine schools, hospitals, housing and administrative offices. "Enterprise" may describe revenue rather than estate complexity. Name the actual boundaries.

Northbank's 82 percent comes from 21 respondents in the large commercial estate subgroup. Their work orders cover mainly staffed offices with on-site stores. The full survey has 68 respondents; the report gives no invitation denominator and does not estimate nonresponse bias. The value may accurately summarize those 21 submissions. It cannot, on that evidence, be presented as the performance of industry-leading providers or the general facilities market.

• Target population: the group the publisher intends to describe.
• Sampling frame: the units that had a chance to enter the study.
• Invitation set: the units approached under the collection design.
• Respondent set: the units that supplied usable observations.
• Published cohort: the subset used for the displayed comparison.

Metric and statistic

Match the denominator and statistic, not the label

Benchmark labels compress decisions. "First-visit completion" can mean jobs closed during the first attendance, faults made safe, permanent repairs completed, requests needing no return within seven days, or work finished without waiting for parts. Recover the unit of analysis, eligibility rule, start and end event, success condition, denominator, exclusions and follow-up window. If the buyer counts urgent repair requests while the report counts technician visits, one request with three visits changes the two measures differently.

Identify the published statistic. Mean, median, weighted mean, percentile and best quartile are not interchangeable descriptions of what is typical. A top-quartile threshold may mean the 75th percentile of organizations, the average within the highest quarter or a vendor-defined performance band. Ask whether organizations, sites, jobs or revenue received equal weight. Inspect the distribution and sample size where available. A mean can be pulled by a few large values; a percentile from a small subgroup can move when one respondent changes.

Check time and normalization. The survey may combine calendar and fiscal years, use monthly snapshots, annualize partial data or divide by floor area, full-time equivalent or completed work order. Compare editions only after identifying method changes. The United Nations and Eurostat quality frameworks link comparability to common definitions, units, classifications and methods. A proposal should not call two figures comparable when the underlying change could arise from measurement rather than performance.

Northbank learns that the report excludes planned maintenance, jobs awaiting specialist parts and visits where access was unavailable. It counts a job as complete when the asset is safe and usable, even if cosmetic work remains. The buyer counts every urgent request and treats a return for any unfinished task within 14 days as a repeat visit. The shared phrase "first visit" therefore hides a narrower numerator and denominator in the benchmark.

Applicability

Explain the differences that could move the result

Create a side-by-side context profile without collapsing it into a score. Compare service content, asset mix, age and condition, geography, access, hours, urgency, demand variability, workforce model, parts strategy, subcontracting, regulation, customer vulnerability, observation period and measurement rule. For each difference, state the plausible direction of effect, the evidence for that direction and whether the difference is material to the intended sentence. Unknown is a valid finding.

Resemblance is not transitive. The benchmark may match Northbank on organization size, another source may match the buyer on public buildings, and a third may match the repair definition. Those partial similarities cannot be averaged into one fully comparable cohort. One decisive mismatch can control the decision. A result that excludes parts waits cannot directly justify a target in a contract where parts waits remain supplier risk, even if geography and estate size look similar.

Distinguish adjustment from invention. If the source publishes strata that match the buyer and the method supports recombination, an analyst may calculate a transparent alternative and preserve the formula. Do not apply an unsupported uplift because public buildings "should be easier," or convert a commercial office rate to a public-estate target through judgment alone. GAO's data-reliability framework asks whether data are accurate, complete and applicable to the intended purpose. Applicability is where an otherwise sound benchmark can fail this proposal.

Northbank records several material differences. The buyer estate includes schools used outside normal hours, heritage buildings, remote depots and secure sites. The benchmark cohort mainly uses staffed offices, on-site stores and weekday access. Emergency demand and parts responsibility differ. These facts do not prove that Northbank will perform below 82 percent. They show that the survey cannot establish the credibility of an 85 percent contractual target without a separate capacity and delivery analysis.

Sensitivity

See whether a reasonable source choice changes the story

Rerun the proposed conclusion under other choices allowed by the original protocol. Use the prior edition, the total sample, the closest pre-defined subgroup, the median instead of the mean where both are published, and every other eligible source. Do not manufacture a pooled average from methods that cannot be combined. The purpose is not to find one final number through voting. It is to learn whether the claim survives ordinary analytical discretion.

Record a short decision table: choice changed, reason it is plausible, resulting value or direction, and effect on wording. The OECD and Joint Research Centre handbook treats indicator selection, missing-data treatment, normalization, weighting and aggregation as choices that can alter a composite result, and recommends uncertainty and sensitivity analysis. A single operational benchmark may be simpler than a composite index, but the same discipline applies when cohort, edition or statistic selection drives the message.

If every eligible comparison supports the same bounded conclusion, the claim gains stability. If the range is wide but all values are relevant, report the range and causes where the tender permits. If one source supports "above average" and another supports "below average," do not choose the first and call the second inapplicable after the fact. Either resolve the methodological difference using the pre-set rules or withdraw the ranking.

For Northbank, the 82 percent subgroup mean falls to 74 percent in the full sample. The second eligible study reports a 69 percent median with a broader denominator. The prior edition reports 77 percent, but the metric changed. None of these values is directly transferable to the buyer. The sensitivity test shows that "industry-leading providers complete 82 percent" is driven by a selected cohort and statistic. The sentence is removed.

Decision and wording

Keep the method and limitation beside the comparison

Choose a disposition that describes use, not quality in the abstract. Direct comparison is available only when the source method and buyer context align on every dimension material to the inference. Comparison with material limits allows a named, narrow contrast. Directional context means the publication helps frame a question but cannot rank the bidder, buyer or target. Recalculation is allowed only from sufficient source data under a defensible rule. Method unavailable, cohort mismatch, selection unresolved and benchmark not used keep uncertainty visible.

Write the sentence from the record. Name publisher, edition, observation period, cohort, measure and statistic. Place the decisive limitation in the same sentence or immediately after it. Avoid "industry average" unless the study supports that population-level description. Avoid "best in class," "top quartile" and "proven target" unless their definitions and relevance are explicit. The citation should take the evaluator to the original table and method, not merely the report homepage.

Northbank classifies the 82 percent result as directional_context_only. Its response says: "A 2026 voluntary facilities survey reported a mean first-visit completion rate of 82 percent for 21 large commercial-estate respondents, under a method that excluded parts-waiting jobs and used a seven-day return window. Because the council definition retains those jobs and tests returns for 14 days, we have not used the survey value as a comparator or as support for our proposed target." The answer then explains Northbank's own parts, triage and measurement plan. The external number no longer carries work it cannot do.

The service and commercial authorities decide the 85 percent offer separately, using capacity, risk, price and buyer measurement rules. The benchmark record reopens if the tender definition changes, a source correction appears, a new eligible publication enters the search window, or the proposal changes the inference. Final review checks every occurrence. A caveat in the methods section cannot cure an unqualified claim in the executive summary.