Idea2Story

01Overview: moving paper reading from online inference to offline graph building

Idea2Story's argument is one sentence: literature understanding is an asset that can be computed in advance, not something to recompute on every task.

Existing automated research agents retrieve, read, and summarize large sets of heavily overlapping papers at runtime, then explore methods and experiment designs through open-ended generation and trial and error. Idea2Story splits this into two stages: offline, top-venue papers and reviews are distilled into a methodology knowledge graph; online, a user's vague idea is aligned to research paradigms in the graph, research patterns are retrieved and combined, and the result is revised through a review loop.

02Why this problem matters: the three costs of reading literature at runtime

The compute-at-runtime paradigm re-reads existing knowledge over and over, losing on cost, ceiling, and reliability at the same time.

• COSTHigh cost, long runtimeThe paper notes that a full run often takes hours, up to 15 hours from idea to experiments; in Agent Laboratory, literature review and experiment planning already take a large share of inference time.
• CONTEXTThe context window is a hard ceilingLiterature is long, heterogeneous, and redundant. Models must sustain coherent reasoning within a limited window; OpenHands and SWE-Agent perform modestly on real codebases and completion rates run low on MLE-bench and SciCode, with the root cause pointing to insufficient reasoning over hierarchical dependencies and structural constraints.
• RELIABILITYHallucination and drift stack upAgent Laboratory's automated review clearly overestimates paper quality; Kosmos fabricates opaque metrics and conflates statistical significance with scientific value; long-horizon execution also shows goal drift.

The ideas themselves are also a problem. Research by Si et al. shows LLM ideas look novel at first screening but fall behind human ideas once implemented; the more you generate, the more similar the outputs, and effective diversity drops. Idea2Story's answer: rather than letting the model improvise, have it retrieve and combine over a method structure already validated through peer review.

03Method breakdown: two stages, five moves

The offline stage turns papers into method units and a graph; the online stage retrieves, combines, and reviews; the two connect only through the knowledge graph.

Step 1: Build the paper pool

The corpus draws on accepted papers and their peer reviews from top venues over the most recent three years: about 5,000 NeurIPS and 8,000 ICLR papers. Full text and review materials are kept for each. Before ingestion, everything is anonymized — author and reviewer names, affiliations, emails, and identity references removed — and reviews pass a safety filter that strips aggressive and toxic language.

Step 2: Extract method units

The extractor splits each paper into introduction, method, and experiments, taking one class of signal from each: the introduction gives motivation and problem definition, the method gives modeling assumptions, learning objectives, architecture, and optimization strategy, the experiments show how the mechanism lands and is evaluated. The definition of a method unit is restrained: it is a self-contained statement of how a problem is posed or solved, stripped of implementation and experimental configuration; dataset choice, hyperparameter tuning, and engineering tricks are excluded by default unless they change the problem definition, model structure, or learning objective. Most papers yield one or a few units, normalized into indivisible meta-methods plus combination-level patterns describing how multiple units within one paper fit together.

Step 3: Cluster into research patterns

Each paper is represented by the embeddings of its method units; nonlinear dimensionality reduction preserves local semantic neighborhoods, then DBSCAN clusters by density. Each cluster corresponds to a recurring research pattern.

Step 4: Build the methodology knowledge graph

The graph is directed: nodes are normalized method units or meta-methods, and semantically similar units merge under one meta-method. Edges come from real papers: two method units used together in the same paper get a directed edge. Edges are therefore empirical evidence of 'actually used together in an accepted paper,' not hand-designed hypothetical relations. The graph encodes two layers at once: reusable method elements and composition constraints derived from co-occurrence statistics.

Step 5: Online retrieval and review-driven revision

Given a natural-language idea, the system computes a fixed-weight multi-view score for every research pattern. Idea level: semantic similarity to similar ideas in the corpus, aggregated over the ideas linked to the pattern. Domain level: relevance of the idea to the research area, multiplied by an effectiveness signal from the graph. Paper level: similarity between the idea and papers instantiating the pattern, multiplied by a quality weight from review metadata. Patterns are then ranked by total score, descending.

Candidate patterns then enter a review loop: an LLM acts as a reviewer, scoring technical rigor, novelty against existing literature, and problem-method alignment, and proposing revisions. If novelty is lacking, it recombines compatible method units or swaps in another implementation from the same pattern family; if feasibility or framing is the issue, it revises the problem definition or method structure. Only revisions that raise the review score are kept; otherwise it rolls back to the previous version, until the reviewer approves or no improvement remains.

04Experimental setup and data

This is a small-scale qualitative evaluation: a 13K-paper corpus, three ideas from an external collaborator as test inputs, against direct generation from the same model.

The corpus is the ~13K papers and reviews described above. Generation uses GLM-4.7 as the language model backbone. The baseline is direct LLM generation: the same model writes a complete research story without pattern modeling or retrieval. Test inputs are three research ideas from an external collaborator. To reduce evaluation bias, research stories from both sides go to Gemini 3 Pro, which took no part in generation and is not told how each story was produced, comparing novelty, method substance, and overall research quality.

05Results: the power to reframe problems comes from structured patterns

The paper offers case-level evidence: method units separate cleanly, the graph has readable structure, and the same vague input yields a deeper problem reframing.

The test input in all cases is “I want to build an e-commerce agent that can better understand user intent.” Excerpted from Table 1 of the paper.

1. FINDING 01

   A paper can be compressed into four slots: problem, solution, story, application

   The case comes from Learning Dynamics of LLM Finetuning. The extraction stores the base problem (understanding how specific training samples affect model predictions during finetuning), the solution pattern (a framework that step by step analyzes how influence accumulates among latent responses), the story (unifying training behavior under learning dynamics), and applications (improving alignment, diagnosing hallucinations) separately, without binding to any dataset or optimization configuration. Because extraction takes only method-level signals and drops implementation detail, the units can be compared, reused, and combined across papers.

2. FINDING 02

   The graph is hub-and-spoke; pattern nodes act as cross-domain bridges

   Figure 4 shows: a few high-frequency domains connect large numbers of papers and research patterns, reflecting the uneven distribution of research activity and marking the centers of method reuse. Many patterns connect several domains at once, while paper nodes usually belong to a single domain. Because patterns are an abstraction above papers, they can bridge weakly related domains, and retrieval need not depend on similarity within one domain or one paper.

3. FINDING 03

   On the same vague input, Idea2Story rewrites the problem; the baseline just adds components

   Idea2Story's IntentDiff reframes intent classification as structural evolution, grounded in generative refinement, structural priors, and evolving representations; the direct LLM's EcoIntent stays inside the traditional classification framework, stacking context modeling and hierarchical objectives into a stronger system. The difference comes from pattern retrieval handing the model a validated high-level problem shape instead of asking it to assemble one from scratch.

4. FINDING 04

   The external reviewer prefers Idea2Story on every evaluated case

   Gemini 3 Pro, blind to how each story was produced, compares novelty, method substance, and overall quality, and favors Idea2Story consistently across all evaluated cases. Directly generated stories stay at high-level abstraction with few concrete method commitments and rely on fairly standard techniques; Idea2Story's problem framing is clearer, its method structure more specific, and its novelty signal stronger. The evaluation covers only 3 cases — that is the boundary of this conclusion.

06Closer look: boundaries and limitations

This work shows the direction is feasible; it does not show the magnitude of the gains.

First, the evaluation is small: 3 ideas, 1 domain case, 1 LLM judge, no human review, and no quantitative comparison of success rate, cost, or runtime. The cost reduction and context-bottleneck relief emphasized in the introduction are argued mechanistically, with no measured numbers. Second, the scoring weights and the concrete parameters of the dimensionality reduction and DBSCAN appear only as formal definitions in the text; the values are not published. Third, the review loop's judge is still an LLM, and the paper's own related-work section concedes that LLM reviewers overweight technical correctness, underrate novelty, and raise scores unreasonably after author rebuttals. The rollback mechanism keeps only revisions that raise the review score, which constrains drift but caps judgment quality at the judge itself. Fourth, the current system stops at the research-pattern layer; experiment-driven validation, dataset selection, and full paper generation are listed as future work: feeding experimental results back into the research story so that method design and empirical validation feed each other.

07Open source and resources

The code is public; the paper lists no data release for the corpus or the graph.

• Code：github.com/AgentAlphaAGI/Idea2Paper, by the AgentAlpha Team.
• Papers：arXiv:2601.20833，Full HTML。
• Data: The papers and reviews used in the graph come from public top venues; the paper provides no download for the cleaned method-unit dataset.

08What this means for the data and evaluation industry

Idea2Story proves one thing: high-quality expert judgment, once structured into units and relations, becomes infrastructure the model draws on at every call.

• TASK DATAThe value of task data lies in the abstraction layer, not the raw textMethod units deliberately drop datasets, hyperparameters, and implementation detail while keeping problem and solution. Task data works the same way: a sample's value is the transferable structure it carries, so we annotate along problem, solution, and acceptance criteria rather than piling up raw text.
• VERIFIERReview signals are the scarcest raw material for verifiersThe paper uses peer review as the signal for quality weights and relation correction, which says review opinions sit closer to verifiers than the papers themselves. Verifiers must learn from real expert adjudications and keep the disagreements and the reasons, not just a score.
• EVALUATIONAn LLM judge cannot stand aloneThe paper's own related work lists the systematic blind spots of LLM reviewers: underrating novelty and raising scores after rebuttals. Evaluation needs expert-annotated calibration sets and separation between judge and generator. Using Gemini 3 Pro as the external judge is the right direction; 3 cases are far too few to conclude anything.
• GRAPHEdges from empirical evidence beat hand-designed edgesEdges in the graph come only from joint use inside real accepted papers — auditable data. We hold our data assets to the same standard: every relation must trace back to a concrete source, be re-checkable, and be updatable.

The paper also supports one industry judgment: the bottleneck for automated research is not models writing better, but reusable structured knowledge and trustworthy evaluation. Teams that distill expert judgment into assets ahead of time will lead on both cost and stability; teams that recompute at runtime will keep paying for re-reading.

09Cite

@article{xu2026idea2story,
  title   = {Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives},
  author  = {Xu, Tengyue and Qian, Zhuoyang and Liu, Gaoge and Ling, Li and Zhang, Zhentao and others},
  journal = {arXiv preprint arXiv:2601.20833},
  year    = {2026}
}