SupraAgent benchmark: twelve iterations, first-source routing and long-context limits

Second round: six more iterations for the first source, long sources and visible limits

Update, 11 November 2026, round two. A second set of six iterations, again with specialist subagents, worked on what a reader experiences first: the first listed source, facts near the end of very long sources, a plain statement of what was not read, and local latency. The same 96 bilingual module questions and 1,037 endpoint sections were replayed; the long-source study grew from 96 to 144 controlled cases up to 1,024,000 characters. Everything below is a local diagnostic. It is not answer correctness, a user study or evidence of being the best agent.

Hit@1 and MRR are new: position bias makes the first listed source matter most (Craswell et al., WSDM 2008). The gain comes from BM25F title weighting (Robertson, Zaragoza and Taylor, CIKM 2004), weight 2.5 from a sweep plateau, with no question ranking its module source lower than before. The runtime gain comes from a bounded per-document token-statistics cache (20,000 entries, 64 MiB of text, identical scores cold and warm; Robertson and Zaragoza, 2009). Hash vectors are unchanged at 74/96; their p95 moved by +0.45 % without a code change and is not attributed to this round. Timing is one workstation without model calls or production IO, read against the 0.1-second perceptual threshold of Card, Robertson and Mackinlay, CHI 1991, not as a service commitment.

1. Module routing: the first listed source counts. BM25, 96 questions, first repeat.

Long sources: facts at the end are no longer lost, and the answer says what was skipped

The complete scan is still bounded at 200,000 characters. Beyond it the packer now segments at most sixteen bounded windows of 8,192 characters on either side of whole-word matches of the question terms, rarest term first, discards every window-edge sentence and carries bracket scope across the gaps. Each bound that is hit is a named omission in the provider prompt. Facts at the beginning, middle and end are retained in all 72 positive cases in English and German, as motivated by Lost in the Middle (Liu et al., TACL 2024). The 36 oversized-negation controls never expose a detached predicate, no injected instruction is projected and absent facts produce no false positive, at the prompt and at the answer. Integration revealed that the grounding audit still checked cited numbers against the first 200,000 characters only and rejected every tail fact with a fallback sentence; it now reads the same question-anchored windows, which is why the answer column matches the prompt column.

The deterministic answer now opens with one localized sentence whenever a source was only partially scanned, following guidelines G2 and G11 of Amershi et al., CHI 2019; the notice contains no citation marker and is checked as a literal string, not as a user study. The packer refactor is byte-identical to the archived packer on 360 random corpora and the whole grid, and makes the 200,000-character scan 2.9 times faster. The answer path at 1,024,000 characters is slower than before because packer, attribution checker and grounding audit each scan the tail windows; it returns the fact instead of a fallback. Correctness and citation support remain separate questions (Gao et al., EMNLP 2023); lengths are characters, not tokens, and no model comprehension is measured (Bai et al., ACL 2024).

2. Long sources: facts retained per length, beginning, middle and end, German and English.

3. Making limits visible: prompt notice, answer notice and fact in the answer.

4. Local runtime in milliseconds, same workstation, no model call or production IO.

What the second round changed, in order: fresh baseline and new first-source, cache and answer-level metrics; packer performance with byte-identical output; the query-anchored tail scan; BM25F title weighting; the bounded token-statistics cache; answer-level disclosure plus the grounding-audit fix, with the full gate, 1,133 focused tests and the complete replay rerun on the final code. A tested shortcut, filtering the German preposition "vor", was rejected because the project keeps temporal words searchable on purpose. Reports, archived sources, raw answers and figures are in the round-two evidence bundle (GitHub access required); every cited DOI was verified through Crossref.

Six improvement iterations: accepted results

Update, 11 November 2026. Six iterations with three specialist subagents improved the local benchmark and its evidence checks. The accepted BM25 candidate retrieves a source from the requested module in its first four results for 96/96 questions, up from 93/96. This measures module routing; independent answer correctness and user satisfaction remain unmeasured.

Each full replay contains 576 executions: 96 questions × two retrieval methods × three repeats, using the same 1,037 endpoint sections. The final candidate has zero failed runs and no per-trial routing or citation loss. BM25 reaches 288/288 module hits across repeats; all 927 observed numeric citation markers resolve to admitted source IDs. That does not establish semantic citation support. The observed p95 reduction is 8.94%; local runtime includes retrieval, admission and deterministic synthesis, and excludes corpus loading, model generation, network and production IO. Repeats and translation pairs are dependent; no significance or production-latency claim is made. Hash-vector routing remains 74/96.

Long sources: measured retention and explicit limits

A new 96-case diagnostic crosses 8,000, 32,000, 128,000 and 256,000 source characters with evidence at the beginning, middle and end, in German and English. Four scenarios test facts, oversized negation, an instruction attack and absent evidence. The paired study records 576 prompt preparations at the actual synthesis and privacy boundary. These are Unicode characters, not model tokens; no model comprehension is measured.

Retention remains 44/48 before and after the disclosure change. All four losses occur at the end of 256,000-character sources beyond the 200,000-character scan bound. The candidate exposes incomplete excerpts and controlled omission reasons to the model: scan-limit visibility improves from 0/24 to 24/24 overflow cases. Whether a live model communicates those limits usefully to users remains untested. Twenty-four oversized-negation controls expose no detached positive predicate; the tested attack instruction is never projected, and absent-fact controls produce no positive match. These bounded controls do not establish general attack resistance.

This change shows no long-source speed improvement: paired prompt-preparation p50/p95 changes from 16.042/36.081 ms to 21.895/36.439 ms. Source scanning remains bounded at 200,000 characters and 4,096 source units.

What changed in the six iterations

1. Reproduced the baseline and corrected missing-evidence, failure, timing and repeat-stability reporting.

2. Tested Unicode matching and language filtering to fix three German support routing misses.

3. Rejected a slower token-counting optimization; retained bounded query-weight and length-normalization work.

4. Added controlled long-source position tests and negative controls.

5. Exposed incomplete-source and omission metadata at the provider prompt boundary.

6. Rejected an integrated candidate that lost citations on one English workflow answer. Conservative German-cue filtering restored English scoring; the final complete-grid comparison passed. Pinned document links are verified against their source bytes.

260 focused tests passed. A further 135 fixture executions preserve the earlier context/fact diagnostics. The six-iteration evidence bundle retains source hashes, accepted and rejected candidates, outputs and reproduction steps (GitHub access required). These are measured local code candidates; production deployment is not verified.

The design follows peer-reviewed work: Lost in the Middle, TACL 2024 motivates varying evidence position; LongBench, ACL 2024 motivates broader context evaluation; ALCE, EMNLP 2023 separates answer and citation quality; Guidelines for Human-AI Interaction, CHI 2019 informs capability-limit disclosure. The papers do not certify SupraAgent. A world's-best claim would additionally require independent competitor comparisons, human UX evaluation, fresh holdouts and verified workflow outcomes.

Earlier release record and charts

The following studies and charts retain the earlier release values for comparison; the accepted results above supersede its module-routing and runtime measurements.

Updated 11 November 2026. The expanded SupraAgent benchmark separates retrieved context, observable answer properties and local component runtime. It adds 96 German and English module questions about actions, activities, processes and support.

The new result: BM25 retrieves at least one source from the requested module in its top four for 93 of 96 questions; deterministic hash vectors do so for 74 of 96. This measures module routing, rather than answer correctness or usefulness. Report and definitions (GitHub access required).

Three studies with distinct scopes

The new runs use real retrieval, admission and synthesis components with a local fixture index and deterministic extraction. They make no model-provider calls and execute no business actions. Repeats and translation pairs are not independent trials. The earlier native answers are shown as recorded evidence and were not regenerated here.

Context performance: measuring retrieval

Three graph hops recover every expected source and reference fact in these small authored fixtures. This is an integration diagnostic. Literal fact presence does not establish entailment; hash vectors are not a modern neural embedding model. Synthetic benchmark.invalid addresses are test identifiers. Complete fixture run · Fixture definitions.

This release does not measure effective long-context capacity. That requires controlled context lengths and systematic placement of relevant evidence at the beginning, middle and end, as motivated by Lost in the Middle.

96 questions for modules and support

Coverage includes People, Teaching, Attema, Blogery, Store Connect, Deals, IoT Manage, Shortcuts, iAccelerator, Help Center, Templaterr and SupraAgent. Each module has four question types in two languages: actions with inputs and prerequisites, activities and status, process steps with verifiable outcomes, and support for permission errors and missing information.

Examples ask how to assign a learner to a Teaching group, what to check before an IoT Manage device update, how to separate reads, changes, approval and outcome checks when publishing in Blogery, and how to diagnose and escalate a People permission error without bypassing access controls. These are diagnostic questions, not evidence of completed actions.

The corpus contains 1,037 complete endpoint documentation sections. Both methods see the same sources; insertion order is reversed on alternate repeats. Generated input and policy hints do not replace complete handler schemas or current tenant permissions. All questions and assessment rules · Answers, source links and timings.

Runtime includes local index-wrapper creation, retrieval, admission and deterministic synthesis, with 288 measurements per method. p50 and p95 use nearest-rank percentiles. Corpus loading, network, model generation and production IO are excluded. These values are not a service latency commitment.

All three BM25 misses are German support questions. Hash vectors score 0/8 module hits for Help Center in this set. These identify concrete development targets, rather than a general language or support-quality ranking. Language and question-type rows overlap and must not be added together.

Answer performance: what source links establish

Every observed numeric source marker in the module replay resolves to an admitted source ID: BM25 885/885 and hash vectors 894/894 across all repeats. One BM25 answer has no such marker. Source-ID resolution is not semantic citation support. Answer correctness, translation quality, support usefulness and process success remain explicitly unmeasured without independent assessment.

The earlier run contains eight reused public development questions with two repeats each. The entire final answer is scored against a short reference. Verbose correct answers can therefore receive low F1 scores, while word overlap does not establish truth. Two source/reference ambiguities remain documented. The global recorded provider transport median is 3,738.776 ms, excluding complete orchestration, and is not comparable to local module replay runtime. Original historical record, complete_creator_accuracy_benchmark section.

Four charts, four bounded measurements

1. Context fixtures: source recall over five annotated cases and literal fact presence over four cases. Does not measure semantic accuracy.

2. Module hit@4: BM25 93/96, hash vectors 74/96. At least one source from the target module; does not establish answer correctness.

3. Local runtime: BM25 p50 22.372 ms, p95 35.196 ms; hash vectors p50 92.306 ms, p95 121.854 ms. 288 runs per method, without model calls or production IO.

4. Historical native answers: whole-answer token F1 4.51%, 6.02%, 7.19%. Eight reused questions, 16 outputs per method; not semantic accuracy.

Open each figure at full resolution. The tables provide the same values as readable text.

Implementation evidence and peer-reviewed research

Implementation evidence links the retrieval adapters and arbiter, synthesizer, source attribution and new module replay. Inputs, outputs, SHA-256 receipts and checks are in the GitHub evidence bundle; repository access is restricted.

• Lewis et al. (NeurIPS 2020), Retrieval-Augmented Generation: Motivates combining retrieval and generation. SupraAgent reports measurements for these stages separately.

• Liu et al. (TACL 2024), Lost in the Middle: Shows position sensitivity in long contexts. Source recall therefore does not establish an effective long-context window.

• Gao et al. (EMNLP 2023), ALCE: Separates answer quality from citation quality. A resolving source ID does not establish that the source supports a claim.

• Yang et al. (EMNLP 2018), HotpotQA: Provides multi-hop questions and supporting-fact annotations. The reused public development questions are not a sealed customer-domain test.

• Yao et al. (ICLR 2025), tau-bench: Evaluates policy-constrained tool use against the resulting database state. Actual process performance requires equivalent outcome checks for SupraAgent.

The HotpotQA dataset on Hugging Face complements the paper reference. These publications support the methodology; they do not peer-review or certify SupraAgent. Implementation evidence consists of code, run receipts and tests. Paper results are not presented as SupraAgent benchmark scores.

Evidence still needed for a product-level quality claim

Required evidence includes independently assessed answers and citation support, fresh tenant-domain holdout questions, controlled long-context tests and isolated workflow runs with verified before/after state. Until then, this benchmark supports specific regression and retrieval diagnostics. It does not establish general accuracy, successful live workflows or superiority to other agents. Reproduction, limitations and full sources.

Auf Deutsch lesen