Making a Tagged PDF Answerable: What Remediation and Metadata Do — and Do Not — Change

1 Why intervene

Part 2 left a specific question open. Eight documents that pass accessibility checkers at 98–100 yielded a task universe of which the best retriever completed 42% of task–phrasing pairs. The paper's §8 turned the diagnosis into eighteen authoring guidelines, but a guideline derived from a failure profile is a hypothesis, not a lever, until a document is changed and the score moves. Part 3 is that test: the same eight PDFs, edited by the same person who tagged them, along the guidelines, with the benchmark side frozen — tasks, gold and the four validated phrasings are reused byte-for-byte.

The failure analysis that preceded the intervention (on the 5,175 pilot runs) also changed what we expected to move. Only 22% of tasks were unanswerable by every system and phrasing; 45 of those 97 had full evidence in front of the model and failed at the answer stage. Half of the retrieval-class failures were context-window losses — the evidence was retrieved but ranked below the shown prefix — not reach failures. And phrasing was the largest single factor (0.55 → 0.26 across the four phrasings of the same task). So the pre-registered prediction (H4b) was that gains, if any, would concentrate in retrieval-class failures on the documents whose punch lists were worked, while the answer stage and lexically distant phrasings would move little.

2 The intervention

2.1 Edits made to the documents

Remediation was done in the tagging tool under a content-integrity rule: every gold value, list item and note text remains in the document unchanged; headings are not renamed; allowed edits are the guidelines' own — tags, scopes, sectioning, captions, alt text, ActualText, note and link binding. The compiled-graph diff (Table 1) is the authoritative record of what changed.

The edits cluster into four kinds. Explicit note binding (D3, D5, D6): footnotes re-tagged as FENote with reference pointers that the updated dump exports as refer_tag; 58 bindings in the tax guide alone. Captions and sectioning (D1, D2): table blocks wrapped in Sect with a Caption, and — this matters below — in D2 the caption replaced the heading (headings 28→22). Key–value tables as lists (D3, D8): two-column label/value tables converted to L/LI/Lbl/LBody. Text-layer hygiene (D1, D8): kerning artefacts ("F un d") fixed; composite cells given ActualText. D4 and D7 show no tag-tree change and act as controls.

2.2 Pipeline changes forced by the edits

Three of the edits exposed gaps in the consumer, which were fixed so that the intervention is measured against a retriever that understands the structures the guidelines ask for. (a) FENote was unmapped (34 notes in D3 silently became untyped); it now maps to NOTE, and refer_tag compiles to REFERENCES edges (referrer→note, direction-normalised). Expansion follows them both ways, and the NOTE_QUALIFY motif binds by explicit reference first, marker match second. (b) Captions: the compiler now emits CAPTION_FOR (caption→captioned table/figure/list) and, when a container opens with a caption instead of a heading, caption-as-title scope; captions join the structural seed channel and the index's context projection; a caption seed expands into its table. (c) Gold re-anchoring: node ids change on re-tagging, so a re-anchorer maps each gold reference by text (exact → whitespace-squashed → thousands-separator-insensitive → substring → alt text → role/page/ordinal), disambiguated by header/label/heading context, keeping both the original and the new text-layer string as accepted evidence. All 436 tasks re-anchored with no unresolved reference.

3 Re-assessment protocol

Two layers, because they answer different questions. The retrieval layer is deterministic: for every paired run, evidence recall and relation coverage are recomputed from the stored retrieval traces on the full retrieved list (reach) and on the 1,400-word shown prefix. No language model is involved, so the paired delta is exactly the effect of the document edits plus the consumer fixes above. The answer layer repeats Part 2's scoring — answerer over the shown evidence, judge over proposition checklists, strict PASS contract — but the pilot had run with an 8,000-character window (one chunk shown) while the current protocol shows 1,400 words (two chunks). A first pass against the pilot scores produced a +6.4-point "gain" for chunking on byte-identical evidence; the original substrate was therefore re-answered and re-judged at 1,400 words so that before and after share one protocol. That re-baseline is the comparison reported here; the pilot-protocol comparison is kept only as a measure of answerer/judge drift.

Statistics: exact McNemar on discordant pairs per system; a task-clustered bootstrap CI on the mean ΔPASS; per-document McNemar for the document table. Pre-registered hypotheses are in docs/dtaa-phase3-plan.md.

4 Results

4.1 Retrieval layer (deterministic)

The intervention is neutral at the retrieval layer for the structure-preserving retriever and for the reconstructed substrate; the structure-blind chunker gains where the text layer was cleaned (D1: kerning). The only per-document movement of size was D2, where turning headings into captions removed the tables from their section scope: Egres reach fell nine points on that document until captions were compiled and consumed as titles, after which it is one point up. Every motif-level Δ lies within ±3 points.

4.2 Answer layer (like-for-like 1,400-word budget)

Table 4b is the study's least expected result. Re-answering the original documents under the 1,400-word window instead of the pilot's 8,000-character one raised Egres from 0.42 to 0.54 and chunking from 0.30 to 0.36; the remediation then added +0.2 and +0.6 points respectively. Half of Egres' retrieval-class failures in the pilot were evidence that had been retrieved but not shown; widening the window is what converted them.

Reading Tables 4–8 against Table 2: whatever moves at the answer layer moved without the evidence set changing, so the chunk-hybrid column is the floor for answerer/judge variance under this protocol, and the Egres column should be read net of it (H4c: Egres − chunk ≈ -0.3 points). The retrieved-but-not-shown share of Egres' class-R failures is 135/596 before and 123/591 after — the budget, not the document, decides those.

4.3 The task universe after remediation (H4d)

Explicit binding is the one guideline whose effect is visible at the universe level: the tax guide's deterministically bindable qualification tasks go from 2 to 18 because the repeated */*** markers that defeated the page-unique rule are now resolved by the document itself. The ambiguity lint did not move (D6: 131 repeated header–value pairs; D8: 10) — captions and summaries were added to D2, where nothing was flagged, not to the documents where the lint fired; and converting D8's tables to lists removed three LABEL_VALUE tasks without disambiguating the flagged pairs.

5 What the intervention reveals

5.1 Structure that a retriever already had cannot be "improved" for it

Egres reads the tag tree. Adding Sect wrappers, captions and list tags to a document whose headings, tables and lists were already tagged gives it nothing new to walk; reach stays at 0.846. The edits change how the document describes its parts, not how they are connected — and description is consumed at the answer stage and in the index, neither of which was reading it (§5.4).

5.2 A guideline can backfire against a naive consumer

Replacing a heading with a caption is the correct PDF/UA form for a titled table block, and D2 lost nine points of reach for it. The lesson generalises: a structure standard's preferred encoding is only as good as the least capable consumer, and the guideline should say "add a caption; keep the heading" until consumers treat captions as titles. The consumer fix is small (CAPTION_FOR edges, caption seeds, caption scope) and should be part of any structure-aware retriever.

5.3 Explicit binding is the deterministic lever

The only edit that changed what the document can declare is note binding. Its effect is invisible on the frozen tasks — they were sampled from the universe the old document could generate, so they were bindable already — and visible on the universe the new document generates (2→18). The evaluation lesson is that an intervention study must re-enumerate, not only re-score: frozen tasks measure regression, the universe measures capability.

5.4 The answer stage never saw the structure

The answerer receives each evidence item as [id] (TABLE_CELL p2) 16.2: no row or column header, no caption, no heading path, no bound note. The wrong-neighbour reads that make up 135 of Egres' 200 answer-stage failures are the direct consequence, and no document edit can reach them. Likewise the index embeds ActualText | text | Alt only, so expansion text (E), table Summary and abbreviation lists — the guidelines' vehicles for phrasing robustness — never affect a seed. The remediation was aimed at the two failure classes the pipeline was structurally unable to pass on to the model.

6 Round 2: metadata that reaches the model

Round 2 acts on §5.4. On the tagging-tool side, every TD now carries explicit Headers pointers (compiled at top priority, origin SOURCE_ATTRIBUTE: D4 2,154/2,154 header edges declared, D1 387/387 — against 261 declared + 53 positional in the original D1), and an abbreviation list plus a block of reliable professional knowledge (the credit-rating hierarchy AAA … CAA/CCC as used by Moody's, S&P and Fitch) is attached to each document's metadata — no PDF structure involved. On the pipeline side, three arms on the improved substrate share documents, frozen tasks, questions, retrieval traces and an 1,800-word budget and differ only in what is written next to each evidence item: A bare item text; B + structural context from compiled edges (heading path, caption, row/column headers, list label, bound notes); C + abbreviation expansions and knowledge glossed onto the items that contain the term, and embedded into the index projection. The chunk baseline has no edges and receives no glosses, so it is the control for both factors.

At the retrieval layer the declared headers change nothing (Egres reach 0.846 in both) — the compiler was already deriving the same relations from Scope — so every effect below is an answer-layer effect.

Table-cell context is the lever. Writing headers, captions, heading paths and bound notes next to the value moves Egres by +3.2 points (p = 7e-10) while the chunk baseline moves -0.2: 46% of Egres' answer-stage (class A) failures at arm A pass at arm B, notes +11 and section sets +10 points, the drug-formulary tables of D4 +15, and the lexically distant Q2 phrasing +3. This is the effect the round-1 remediation was aiming at and could not produce, because the structure never reached the model.

Professional knowledge works where it applies, and costs a little where it does not. Pooled over all runs the knowledge arm is flat (Egres +0.3). On the 757 Egres runs whose task involves a glossed term it is +2.4 points (p = 0.0051): the credit-quality tasks of D1 go from 0.80 to 0.97 once "BAA" is accompanied by "medium grade, lowest investment-grade tier", D5's exposure tasks from 0.29 to 0.44 (net/gross/market-cap definitions), D8 from 0.35 to 0.45. On the remaining runs it is -1.3 (p = 0.066): gloss lines spend budget. The chunk baseline is exactly 0 in both subsets (identical evidence, cached answers). Knowledge should therefore be attached per item, only where the term occurs — which is what the metadata-driven design does — not appended globally.

The budget, again. Raising the shared window from 1,400 to 1,800 words moves the chunk baseline +11.6 points (a third chunk fits) and Egres +1.6. Under the same budget Egres with context (0.59) leads chunking (0.48) by twelve points on the improved documents.

7 Practical guidance

What the two rounds support, stated as instructions. The evidence behind each item is the table it cites; items without a measured effect are marked as such.

7.1 For taggers: making a PDF answerable, not merely compliant

T1 — Give every data cell explicit header pointers. Set Headers on each TD (all row and column headers that apply, including group headers), not only Scope on the TH. Retrieval reach does not change (the compiler derives the same relations from Scope), but the declared pointers are what let a consumer print "row ‘Chile’ · column ‘Fund’" beside a value with certainty, and they resolve repeated values unambiguously (D2: 69 ambiguous gold references → 4). Round 2 rests on them.

T2 — Bind notes explicitly. Tag footnotes as FENote (or Note) with reference pointers from the marker to the note; never rely on a bare * being unique on the page. This is the one edit that expanded what the document can declare (D3: 2 → 18 bindable qualification tasks, Table 9) and it is what lets the consumer show "note: …" beside the claim (NOTE_QUALIFY +11 points in round 2).

T3 — Caption every table and figure, and keep the heading. A Caption as first child of the table block is right; a caption instead of a heading removed the table from its section scope for a heading-only consumer (D2 −9 points of reach, §5.2). Until every consumer treats captions as titles, add — do not replace.

T4 — Make the caption say what the table is. "Country distribution (%), fund vs. index, as of 31 Mar 2025" beats "Table 3". Captions are indexed and shown as context; a caption that repeats the document title adds nothing (Table 1's D2 captions did, and the gain came from the scope, not the words).

T5 — Put units and periods in headers, one value per cell. "$1,192.50 plus 12%" in one cell, or a unit that lives only in a footnote, defeats numeric matching and comparison. Composite cells were among the round-1 answer-stage residuals; splitting them is a source edit no pipeline can replicate.

T6 — Keep the text layer machine-clean. Fix kerning artefacts ("F un d"), keep ActualText consistent with the visible string (the "7,566" vs "7566" case broke string matching for every system), and avoid ActualText that changes the number's form. This was the only document edit that moved the structure-blind baseline (D1 +4 reach).

T7 — Disambiguate repeated header–value pairs. When the same header pair maps to different values in one section (D6: 131 cases, D8: 10), add a caption or a qualifying header ("2Q24", "Class Y"); the answerability lint lists them. No consumer fixed these and none can.

T8 — Attach an abbreviation list as document metadata. Every "BAA", "DIN/PIN", "MAGI", "PVITINST" the document uses, with its expansion — none of the eight files carried an E attribute. Round 2 shows expansions and short domain definitions attached per term are worth +2.4 points on the tasks that use them (Table 12); the same text placed nowhere is worth nothing.

T9 — Attach reliable professional knowledge, scoped to the terms that occur. Standard hierarchies (rating tiers, dosage forms, share-class codes) belong in metadata, not in the page. They helped exactly where the term appears (D1 credit quality 0.80 → 0.97) and cost budget where it does not (−1.3 on unrelated tasks); the consumer must apply them per item.

T10 — Key–value tables as lists only when the pairs are self-contained. Converting two-column label/value tables to L/LI/Lbl/LBody is legitimate, but it removed three tasks in D8 and cost 3 points there (Table 5): a list body loses the column header that told the reader what kind of value it is. If the table had meaningful column headers, keep it a table with T1.

T11 — Section headings that match the reading. An H nested under the wrong parent ("Key employee" under "Highly compensated employee") produces gold the reader disagrees with and answers the judge rejects. Levels should follow the visual hierarchy, and every Sect should open with its own heading or caption.

T12 — Informative alt text, or none. Figure tasks are answerable only through alt text (chunk baseline 0.00 on FIGURE_ALT); "Brandmark of …" answers nothing. State what the figure conveys, with its numbers.

T13 — Do not re-tag what already works. Wrapping already-tagged headings, tables and lists in more containers changed nothing for a structure-aware retriever (§5.1). Spend the effort on T1, T2, T5–T9.

7.2 For pipeline builders

P1 — Serialize structure into the evidence block. Heading path, caption, row/column headers, list label and bound notes next to each item: +3.2 points for Egres, none for chunks (Table 11). The cheapest change in the study and the largest.

P2 — Apply glosses per item, never globally. Expansions and knowledge only where the term occurs; a global preamble spends the window on unrelated tasks (−1.3 where the term is absent).

P3 — Budget per evidence bundle. A cell travels with its headers and caption, a claim with its note; trim bundles, not items. Report "retrieved but not shown" separately: it was half of the retrieval failures, and the 1,400 → 1,800-word step alone was worth +11.6 points to the chunk baseline.

P4 — Read captions as titles and follow reference edges. CAPTION_FOR and REFERENCES are small additions to a heading-only walker and they are what T2/T3 assume.

P5 — Treat declared relations as provenance. Keep the origin of every header edge (declared / scope-derived / positional); it tells the reader how much to trust a pairing.

7.3 For evaluators

E1 — Re-enumerate, not only re-score. Frozen tasks measure regression; the universe measures capability (T2's effect was invisible on frozen tasks).

E2 — Hold the protocol fixed and keep a control. Half the first before/after was a budget change; the structure-blind baseline on identical evidence is the noise floor.

E3 — Report A_t with its companions. Any-phrasing answerability, cross-system union and a graded completeness score bound what remains recoverable; strict PASS alone is dominated by long series.

8 Limitations

One remediator, who also produced the diagnosis; eight documents; two of them effectively unchanged. Re-anchoring used context-based disambiguation for repeated table values (D2: 69, D3: 23 references) — an error there affects relation coverage for that task only, never evidence recall. The answer layer's noise floor is large: on identical evidence the pilot-protocol comparison moved chunking by +6.4 points, so answer-layer deltas below the chunk column's own movement are not evidence of anything. The retrieval layer is the study's reliable instrument. Round 2 ran once per arm with a single seed; the knowledge subset is defined post hoc by term occurrence in gold/intents (D7's 335 "EDU" runs dilute it — excluding them the targeted Δ is larger).

9 Conclusion

The question behind Part 3 was whether a tagged PDF can be made answerable by its author, or whether answerability is a property of the pipeline that reads it. The answer is that it is a property of the pair, and the two halves are not symmetric. Tagging more thoroughly does not help a consumer that already reads the tag tree; it can even hurt when a standard's preferred encoding outruns the consumer. What the author can do is declare relations the document could not previously state — which note qualifies which claim, which headers govern which cell — keep the text layer honest, and attach, as metadata rather than as page content, the expansions and domain knowledge a reader is assumed to bring. These are modest, mechanical edits, and none of them changes what a sighted reader sees.

Their value is realised only downstream. The structure-aware retriever gained nothing from the edits until the pipeline began writing the declared context beside each piece of evidence; once it did, the answer-stage failures that no document edit had been able to touch began to clear, and injected knowledge proved useful precisely where the document used the term. The structure-blind baseline, which has nothing to carry, stayed where the evidence window put it. The practical guidance in §7 follows from this division of labour: a short list of authoring rules for taggers whose payoff is contingent on consumers that carry structure through, and a matching list for the builders of those consumers.

What remains open is scale and generality — eight documents, one remediator, one family of retrievers — and the residual the study could not move: ambiguity in the source itself. Repeated header–value pairs, composite cells and shared markers are fixed in the document or not at all, and the answerability lint that finds them is, in the end, the most direct tool this line of work offers an author. Part 2 measured the distance between compliant and answerable; Part 3 shows where that distance lies and that a good part of it can be closed from both ends.

Artifacts: dtaa-eval/phase3-arms.md, dtaa-eval/phase3-knowledge.md, dtaa-eval/abbreviations.json, dtaa-eval/phase3-summary.md, dtaa-eval/phase3-retrieval.md, dtaa-runs/corpus-pilot-improved/, dtaa-runs/corpus-perfect-b1400/, dtaa/reanchor.py, dtaa/context.py, docs/dtaa-phase3-plan.md, docs/dtaa-phase3-results.md.