Story2Proposal

01Overview: long-document quality depends on where the constraints live

The hard part of paper writing is keeping narrative, experimental evidence, and figures aligned throughout. A pipeline that generates freely and checks afterward cannot do this; writing the constraints into a contract shared across the whole run can.

Story2Proposal comes from the AgentAlpha team. It turns a research story (narrative description, experimental evidence, background) into a complete paper draft, with four agents — Architect, Writer, Refiner, Renderer — working in relay under one persistent, shared visual contract, while evaluation agents keep giving feedback and rewriting the contract during generation. Experiments draw on the Jericho research corpus, scored by ten independent experts on eight dimensions.

02Why solve this: generate first, check later, and structure inevitably drifts

Errors from unconstrained generation accumulate across sections; by the time post-hoc checking runs, the drift has already happened.

A paper is the long-form document with the strictest cross-section consistency demands. Contributions claimed in the introduction need detail in the method section, item-by-item verification in the experiments, and related work that does not contradict the method. Direct prompting has no intermediate control, single-agent long-form generation lacks division of labor, and outline methods supply only high-level structure — none of the three holds these constraints. The fallout falls into four classes: section drift, claims that do not match experiments, citations detached from the argument, and methods contradicting the experimental setup.

• PROBLEM 01Structure driftLater sections drift away from the original problem statement. With no shared state, each section answers only to itself.
• PROBLEM 02Visual artifacts lostFigures and tables exist only as text tokens in the prompt; repeated rewrites drop or duplicate them, and the references break along with them.
• PROBLEM 03Checking comes too lateChecking runs only after generation ends, when errors have already spread across sections; repair then costs more than prevention.

03Method breakdown: one contract, four generation agents, one set of evaluation agents

Story2Proposal writes structural and visual obligations explicitly into the contract state, and every agent works against that same state.

Step 1: The contract has three layers

A global visual registry records each figure's or table's type, semantic description, standard label, and expected reference location. Section-level obligations specify which figures must appear in each section. Validation rules require unique labels, valid cross-references, and figure descriptions aligned with the surrounding text.

Step 2: Architect builds the skeleton

Architect splits the research story into ordered sections, writes the argument outline, key claims, and supporting evidence for each, identifies candidate figures and registers them in the contract, and maps every figure to the section where it must appear.

Step 3: Writer drafts against the contract

Writer drafts section by section; visual markers and reference identifiers must match the contract registry. Figures may cite only registered items and cannot invent new markers.

Step 4: Refiner aligns globally

Refiner compresses repeated explanations, unifies terminology across sections, and confirms every referenced figure is properly explained in the surrounding text. On finding an inconsistency, it triggers a contract update through evaluation feedback.

Step 5: Renderer runs deterministic checks

Renderer outputs stable LaTeX, resolves every visual reference, standardizes labels, and checks cross-references against the contract. Every registered figure must appear exactly once, with consistent references.

Step 6: Evaluation agents revise the contract as writing proceeds

Each evaluation agent owns one dimension: reasoning verification, data fidelity, visual consistency. They report issues, confidence levels, and fixes on intermediate outputs, aggregated into contract updates. A typical case: a figure reference found without explanation adds a rule requiring that reference to carry a paragraph of explanation. This is the generate-evaluate-adapt loop.

04Experimental setup and data

Human reviewers are the standard here, because automatic metrics cannot capture document-level reasoning coherence.

The four base models are GPT, Claude, Gemini, and Qwen, run with the same pipeline configuration, generating a full draft from the same research story, with identical prompts and inputs across all compared systems. Each condition gets ten independent experts scoring the complete draft on eight dimensions — structural completeness, writing clarity, methodological rigor, experimental substance, citation discipline, reproducibility, format stability, and visual communication — aggregated into a single expert score.

Two experiments. The first compares against DirectChat: single-stage prompted generation with no contract and no multi-agent setup. The second compares against the structured generation baseline Fars on four benchmark papers: Escrowed Batch Reveal, Symbolic Execution, Hazard-Signature Tombstones, and Poisoning LLM-Induced Rules.

05Results: large leads across all four base models

1. FINDING 01

   The gain comes from the architecture, not the base model

   The margin across the four base models falls between +1.884 and +2.323, all positive. DirectChat ranges only from 3.864 to 4.078 across the four, while Story2Proposal lifts it to 5.962–6.257, and the gap between base models actually narrows. The reason: structural responsibility is stripped off the model and carried by the contract and the pipeline, leaving the model to handle local reasoning and content generation.

2. FINDING 02

   Still ahead of the strong structured baseline, with the margin narrowing to +0.508

   Fars already uses structured templates and averages 5.197. Story2Proposal reaches 5.705, higher on all four benchmark papers. The largest gap is on Hazard-Signature Tombstones at +0.891; the smallest on Symbolic Execution at +0.248. Templates fix the basic structure; the remaining gap comes from continuous checking and figure-text alignment, which is exactly what the contract does.

3. FINDING 03

   The lead holds for every single reviewer

   Story2Proposal scores above DirectChat in every one of the ten reviewers' columns, under every base model. E8, for example: 4.23 to 7.60 under GPT, 3.92 to 7.81 under Claude. E3's DirectChat baseline was already high, around 6, so the gain is the smallest there: 5.97 to 6.98 under GPT, +1.01. The lower the baseline, the bigger the contract's payoff.

4. FINDING 04

   The clearest gains are in structural completeness and format stability

   Expert feedback notes that contract-generated drafts have more consistent section organization, fewer layout errors, and clearer experimental descriptions. Against Fars, figures in the experiments section are consistently referenced and placed correctly, with stronger figure-text alignment. The mechanism is direct: figures live in a registry with identifiers and placement obligations, and agents cite registered items instead of regenerating markers.

06Closer look: mechanisms, boundaries, and limitations

The contract works because errors are stopped before they propagate; the contract's ceiling is set by the evaluation agents' judgment and the completeness of the input story.

Mechanism 1: each stage updates and validates the contract before moving to the next.This mirrors grammar-constrained decoding catching problems mid-generation, just at the document level. Structural errors do not accumulate across sections.

Mechanism 2: responsibilities are externalized.DirectChat asks one model to handle reasoning, formatting, section structure, and figure references at once, stacking task complexity. Story2Proposal hands the latter three to the contract and dedicated agents, leaving the model to handle local content. Swap the base model and the structural behavior stays the same.

Ablation.This version runs no ablation study. The separate contributions of the contract, the evaluation loop, and the four-agent division of labor are not measured apart; the gain is the effect of the architecture as a whole.

Limitation 1:The margin over Fars is clearly smaller than the margin over DirectChat. The largest gains come from replacing unconstrained generation.

Limitation 2:When evaluation agents miss subtle reasoning flaws, the contract never triggers a fix. A draft can be structurally compliant yet weakly argued. Richer evaluation signals and more specialized critique modules are needed.

Limitation 3:The contract constrains structure and visual obligations; it does not generate missing scientific evidence. Given an incomplete research story, the draft will be internally consistent but have holes in its content. Retrieval or evidence-verification steps are needed to complete the input before generation.

The paper also states that the draft text and figures were generated mainly by the Story2Proposal agents themselves, with humans responsible only for system design and experiment organization.

07Open source and resources

The paper and full HTML are linked in the buttons at the top: version arXiv:2603.27065v1, submitted March 28, 2026, categorized cs.CL. The paper gives no code repository or dataset release, and no access link for the Jericho research corpus. Raw experiment scores appear column by column per reviewer in Table 2 and Table 3 and can be checked directly.

08What this means for the data and evaluation industry

Quality must be constrained during production; inspection after delivery can only find problems, not remove them.

• TASK DATATask data needs machine-checkable structural obligationsThe contract encodes 'every figure appears exactly once and every reference resolves' as machine-checkable rules. Long-horizon task data works the same way: each task should carry its required artifact list and reference relations, checked item by item at annotation time.
• VERIFIERVerifiers need layers, with deterministic checks lastRenderer's checks are deterministic; the evaluation agents' judgment is probabilistic. The former guards the floor, the latter carries the semantics. Merge the two into one scorer and the floor gives way.
• EVALUATIONSplit evaluation into dimensions and keep per-reviewer detailThe eight-dimension by ten-reviewer matrix pins every improvement to a specific dimension and a specific person. A single total score hides disagreement; the contrast between E3 and E8 is visible only in the detail.
• LIMITStructural compliance is not scientific correctnessThe paper itself concedes the contract does not govern reasoning quality. Verifiers must cover evidence and reasoning separately, which still requires domain experts — and that is where high-quality data earns its value.

The judgment for DataMining is clear: the bottleneck of generative pipelines has shifted from model capability to the engineering of constraints and verification. Whoever supplies task data with structural obligations, layered verifiers, and dimension-level evaluation sets the reliability ceiling for the next generation of long-horizon agents.

09Cite

@article{qian2026story2proposal,
  title   = {Story2Proposal: A Scaffold for Structured Scientific Paper Writing},
  author  = {Qian, Zhuoyang and Shi, Wei and Lin, Xu and Ling, Li and Luo, Meng and others},
  journal = {arXiv preprint arXiv:2603.27065},
  year    = {2026}
}