White Paper

Chatbot Benchmark Methodology

Every number on this site has an entry below: what it counts, how it is measured, where it comes from, and what it leaves out.

Last reviewed 2026-09-22

Where HoverBot stands today

HoverBot is in a design-partner pilot. No production customer cohort has yet closed a full measurement window, so the outcome ranges below are targets: the ranges the product is built and tuned to reach, informed by runs on HoverBot's own demo tenants and by public industry benchmarks. Each entry will move to a measured provenance, with the sample it was measured on, as pilots complete. Until then, treat every range as a design goal, not a promise.

Provenance labels

Every figure carries one of these labels, here and wherever the figure appears.

Measured, production
Observed on live customer tenants over a full measurement window, using the definition on this page.
Measured, demo tenants
Observed on HoverBot-operated demo tenants. Traffic is real but small and not from paying customers.
Modeled
Derived from a stated formula and input assumptions rather than observed on a deployment.
Target
The range HoverBot is built and tuned to reach. It has not yet been confirmed on a production customer cohort.
Capability
A property of the underlying components that HoverBot exposes, not an outcome measured on deployments.

Published figures

Ticket deflection: 25-45%

Target
Definition
Conversations that end without a human handoff or a ticket being created, divided by all conversations the chatbot handled in the window. A conversation the visitor abandons after one bot reply does not count as deflected.
How it is measured
Counted from conversation and escalation events over a 30-day window after the tuning period, per tenant. Handoffs are recorded by the escalation skill; ticket creation is recorded by the escalation skill.
Limitations
Depends on how repetitive the ticket mix is and how aggressively the handoff threshold is set. A tenant can raise deflection by refusing to escalate, which is why accuracy is published alongside it.
Last reviewed
2026-09-22

Qualified lead conversion: 15-30%

Target
Definition
Conversations in which the visitor completes a lead-capture flow with every required field, divided by conversations in which the flow was offered.
How it is measured
Counted from lead-capture skill events over a 30-day window, per tenant. A lead counts once, on the first completed submission.
Limitations
The denominator is conversations that reached a qualification flow, not all site visitors. Do not compare it with site-wide conversion rates.
Last reviewed
2026-09-22

Response accuracy: 80-90%

Target
Definition
Bot responses judged correct and grounded in the tenant's knowledge base by a human reviewer, divided by responses sampled on topics the knowledge base documents. Responses on undocumented topics are scored separately as handoff decisions.
How it is measured
Weekly human review of a random sample of conversations per tenant, scored against the source document the retriever cited. It is not resolution rate, intent classification accuracy, or a model self-score.
Limitations
Sample sizes on pilot tenants are small. The figure says nothing about topics the knowledge base does not cover.
Last reviewed
2026-09-22

Implementation time: 7-30 days

Target
Definition
Calendar days from the kickoff session to the customer's approval to serve production traffic.
How it is measured
Taken from project records for each deployment. Widget installation is a single script tag and is not what drives the window; knowledge base preparation, guardrail configuration and handoff routing are.
Limitations
Approval cycles on the customer side are usually the largest component and are outside HoverBot's control.
Last reviewed
2026-09-22

Supported languages: 50+

Capability
Definition
Languages in which the underlying language model can read a visitor message and reply in kind without a separate model or translation step.
How it is measured
Taken from the model provider's published language coverage. HoverBot has not run a per-language accuracy test, and does not detect language with a separate classifier; the model replies in the language it is written to.
Limitations
Answer quality in a language follows how much of the knowledge base exists in that language. Coverage is not the same as accuracy.
Last reviewed
2026-09-22

Time to positive ROI: 30-90 days

Modeled
Definition
Days until cumulative support cost avoided exceeds cumulative HoverBot cost, using the ROI calculator's formula.
How it is measured
Computed from the calculator inputs: monthly ticket volume, cost per ticket, and the deflection target above. It is not observed on a deployment.
Limitations
Inherits every assumption of the deflection target. Change the deflection input and the payback window moves with it.
Last reviewed
2026-09-22

How a figure becomes measured

  1. A tenant serves production traffic for at least 30 days after its tuning period ends.
  2. The metric is computed from that tenant's conversation, escalation and lead events using the definition above, with nothing excluded except test conversations tagged as such.
  3. Response accuracy is scored by a human reviewer against the cited source document, on a random sample of at least 100 responses per tenant.
  4. The entry on this page changes provenance, states the number of tenants and conversations in the sample, and records the review date.
  5. Published ranges are updated quarterly. A range is never widened to include a favourable outlier.

What is excluded

  • Sandbox and test workspaces, and conversations tagged as tests.
  • Deployments with incomplete event telemetry for the window.
  • Figures from the illustrative scenarios in the resources section. Those are modeled from demo-tenant data and are labelled as such on every page.

Related pages

The technical overview describes the system these figures are measured on, and the trust center lists the subprocessors and data handling behind it.