← All tasks

Rare disease genomics

RNU2-2 variants in developmental and epileptic encephalopathy

Human geneticsNeurogeneticsAgent graded
construction_9c827688_20260910_real_bam
Recorded score18.2%Task-specific rubric
Pass / failFailSeparate from score
EvaluationRecorded runSaved files
Read the task instructions and deliverables
# Vanilla Claude Code Controlled Packet Run: construction_9c827688_20260910_real_bam

You are the `vanilla_claude_code` system arm for a controlled benchmark packet.
Work only from this sandbox. Do not read paths outside this sandbox, and do not
look for or read lynnvironment skills, benchmark truth, answer keys, scorecards,
generation manifests, raw BAM/CRAM/FASTQ files, or genotype files outside
`case/input/`. Approved VCF/VCF.GZ files are model-visible benchmark inputs
when present in `case/input/`.

Use all search, literature, and database tools available in this runtime when
they can strengthen the interpretation. External services may receive only
small identifiers such as gene symbols, disease names, HPO terms, rsIDs, or
narrow variant coordinates. Do not upload packet files or raw genomic content to
external search, literature, or database services.
Do not search for this benchmark case id, benchmark truth, answer keys, or prior
benchmark outputs.

There is no numeric cap on external search/database calls. Keep research
evidence-driven and within an approximately US$10 total per-case inference and
paid-tool budget: do not repeat a request without a changed question or input,
and do not use a paid service unless it could resolve a material evidence gap.
Prioritize direct ClinVar/gnomAD records, primary literature, and authoritative
gene-disease sources. Stop external research once the leading candidates have
adequate evidence for the report or the available evidence ceiling is clear.
Log every external request as one JSON object per line in
`external_queries.jsonl`, including the tool or service, exact
query/parameters, purpose, returned stable identifiers or URLs, and whether it
changed the assessment. Create the log even if no external request is needed.

Read these sandbox files first:

1. `schema/final_report.schema.json`
2. `case/input/phenotype.txt`
3. `case/input/genotype_summary.md` if present; otherwise inspect the approved
   genotype files in `case/input/`, such as `.vcf`, `.vcf.gz`, and index files

Task:
Produce one machine-readable diagnosis artifact from the visible packet only.

Output requirements:
- Return only one raw JSON object as your final answer. No Markdown and no prose.
- If your agent runtime writes files, write the same object to `final_report.json`
  in the sandbox root.
- Use `schema`: exactly `lynnvironment.final_report.v1`.
- Use `case_id`: `construction_9c827688_20260910_real_bam`.
- Use `system_id`: `vanilla_claude_code`.
- Use `model`: `claude-fable-5-1` if known, otherwise `unknown`.
- Use `genome_build`: `GRCh38`.
- Include `candidates` and `case_diagnosis`.
- If reporting a candidate, preserve `chrom`, `pos`, `ref`, and `alt` exactly
  from visible caller rows under `variant`.
- If a visible caller row provides a mechanism/class label such as
  `single_nucleotide_variant` or `small_indel`, preserve that exact observed
  caller label as candidate `variant_class`.
- Use candidate `event_structure` for event geometry such as
  `single_nucleotide_variant`, `small_indel`, `cnv_deletion`, `bnd`,
  `balanced_translocation`, or `inversion`.
- Use candidate `functional_effect` for clinical/functional effect if you can
  support one from the visible packet; otherwise use `unknown` and explain the
  limitation.
- Do not collapse structure and biology into one label; for example, a
  regulatory BND can have `event_structure: balanced_translocation` and
  `functional_effect: noncoding_regulatory`.
- Use only variants present in the genotype packet caller rows.
- Use phenotype free text and HPO terms from the phenotype packet when assessing
  phenotype fit, but do not treat synthetic phenotype context as clinical
  validation or as a reason to overclaim pathogenicity.
- Use the visible packet as the sole source of case-specific genotype and
  phenotype facts. Use external search and database tools for current
  gene-disease, population, clinical, functional, and literature evidence.
- Do not invent citations, PMIDs, ClinVar/gnomAD assertions, allele frequencies,
  segregation, validation, ACMG criteria, coordinates, alleles, or
  classifications.
- Each evidence entry must cite the visible packet or a searched external
  source. For external evidence, include a stable identifier or direct URL and
  the precise supported claim. Clearly label unsourced recollection as
  unsupported model background knowledge.
- If evidence is insufficient for a confident classification, use
  `uncertain_significance` or report no candidate rather than overclaiming.
- Include explicit limitations: research decision support, not clinical
  sign-out; no orthogonal validation unless visible in the packet; no BAM-level
  discovery or variant calling unless explicitly provided in the packet
  workflow; state whether the genotype review was compact-summary, bounded-VCF,
  or full-VCF based on the visible inputs.
- Do not include benchmark-internal truth, answer keys, generation metadata,
  event lists, or grading fields in the output.

For `case_diagnosis.status`, use:

- `diagnostic` only when the packet supports a confident molecular diagnosis;
- `candidate_finding` when a plausible candidate is surfaced but limitations
  prevent diagnosis-grade certainty;
- `no_finding` when no packet-supported candidate explains the phenotype.

How was it graded?

Single LLM judge scoring weighted rubric dimensions against truth.json; separate pass/fail verdict.

Weighted average of applicable rubric scores, each from 0 to 1, rounded to six decimals. Dimensions marked not applicable are excluded from the average.

LLM judge: gpt-5.6-sol

Task files

Choose an evidence file.

Grading criteria

10 of 10 criteria
causal_findingincorrect / 0.000
Causal finding

Earns credit

Recognize the complete required causal finding in RNU2-2, including every required allele or event member. Required GRCh38 DNA changes: chr11:62841621 A>G + chr11:62841647 C>CA + chr11:62841748 A>C + chr11:62841767 A>C. A single member of a multi-member event is incomplete. Optional findings do not invalidate a complete answer. Top-K recall requires all required findings in the first K causal ranks without unrelated findings interleaved; top-3 recall requires all by rank 3. For a multi-member event, use the worst member rank unless it is correctly grouped as one event.

Weight: 0.2

Recorded result / fable 5 1

The sole required RNU2-2 four-member candidate event is absent; ranks 1 and 2 are unrelated ASH1L and GRIN2A variants.

variant_identificationincorrect / 0.000
Variant identification

Earns credit

Identify all required DNA changes on GRCh38: chr11:62841621 A>G + chr11:62841647 C>CA + chr11:62841748 A>C + chr11:62841767 A>C. These are the reference sequence representations; evaluate equivalent representations using the original judge rules. Naming the gene alone is not sufficient. Missing an allele or a required member makes the variant identification incomplete; gene recognition is scored separately.

Weight: 0.15

Recorded result / fable 5 1

None of the four required chr11 alleles is reported.

gene_identificationincorrect / 0.000
Gene identification

Earns credit

Identify the causal gene: RNU2-2. This dimension scores gene identity separately from the exact variant: identifying the correct gene can earn gene credit even when the reported allele is wrong. It does not by itself satisfy the causal-finding or variant-identification checks.

Weight: 0.1

Recorded result / fable 5 1

RNU2-2 is not reported at any rank.

event_completenessincorrect / 0.000
Event completeness

Earns credit

Judge zygosity, phase, inheritance, event grouping, gene mechanism, and functional consequence. Penalize an unphased recessive pair presented as proven trans, a heterozygous-only recessive assertion, a coding-only account when a regulatory allele is required, or a digenic truth reduced to one gene. Partial event recognition deserves partial—not zero—credit unless it produces a confidently wrong diagnosis.

Weight: 0.1

Recorded result / fable 5 1

The report omits all four heterozygous members, their grouping as one event, and the required phase structure of two cis pairs on opposite inherited haplotypes.

classificationincorrect / 0.000
Classification

Earns credit

Use the reference classification policy for each required finding: chr11:62841621 A>G + chr11:62841647 C>CA + chr11:62841748 A>C + chr11:62841767 A>C: uncertain significance. Respect the case confidence ceiling: candidate finding. A classification within one ACMG tier can receive partial credit but cannot override an explicit candidate-only ceiling. Apply this policy to the combined event, not independently to each member.

Weight: 0.1

Recorded result / fable 5 1

The required combined RNU2-2 event receives no classification. VUS classifications assigned to unrelated findings do not satisfy the truth's combined-event VUS policy.

case_dispositionpartial / 0.500
Case disposition

Earns credit

The case conclusion must capture the complete required finding at the supported confidence, with no fatal alternate diagnosis or material overclaim. The reference case-status ceiling is candidate finding. A complete supported candidate can succeed without claiming a confirmed diagnosis. Case success is assessed separately from the weighted rubric score.

Weight: 0.1

Recorded result / fable 5 1

The top-level candidate_finding status respects the truth's ceiling, but it is attached to the wrong molecular findings and omits the actual actionable candidate.

overclaim_and_restraintpartial / 0.600
Overclaim and restraint

Earns credit

Penalize classification overclaim, causal-scope overclaim, unsupported alternate diagnoses, and diagnostic use of a known decoy. Do not penalize a clearly labeled incidental or noncausal finding merely for being listed. Reward explicit rejection of a decoy for the correct biological reason. Set `negative_false_positive` true whenever a hard-negative report makes a confident diagnosis or pathogenic causal assertion.

Weight: 0.1

Recorded result / fable 5 1

The report appropriately avoids a diagnosis or pathogenic assertion, but overextends case relevance by designating two non-truth VUS findings as candidates and claiming they were the only findings surfaced.

artifact_contractcorrect / 1.000
Artifact contract

Earns credit

The report must be valid JSON with the expected report identity, a usable case diagnosis, and a ranked candidate list. Wrong case ID, wrong reference build, duplicate/invalid ranks, internally contradictory top-level disposition, or a report that cannot be interpreted is a contract failure. Minor extra fields or harmless representation differences are not failures.

Weight: 0.05

Recorded result / fable 5 1

The report is valid and interpretable, with the correct case ID, GRCh38 build, report identity, unique ranks 1–2, and a usable case diagnosis.

evidence_and_reasoningpartial / 0.250
Evidence and reasoning

Earns credit

Evaluate whether report evidence actually supports identity, pathogenicity, mechanism, and case relevance. Multiple independent, traceable sources are stronger than repeated assertions from one source. Missing provenance, vague evidence, or evidence that supports the gene but not the exact variant reduces credit. Do not demand literature beyond what is necessary to compare with truth.

Weight: 0.05

Recorded result / fable 5 1

The report provides detailed, traceable evidence for its selected variants, but none supports the required RNU2-2 event, its phase, inheritance, RNA evidence, or case relevance.

decoy_handlingnot_applicable
Decoy handling

Earns credit

Penalize classification overclaim, causal-scope overclaim, unsupported alternate diagnoses, and diagnostic use of a known decoy. Do not penalize a clearly labeled incidental or noncausal finding merely for being listed. Reward explicit rejection of a decoy for the correct biological reason. Set `negative_false_positive` true whenever a hard-negative report makes a confident diagnosis or pathogenic causal assertion.

Weight: 0.05

Recorded result / fable 5 1

The truth specifies no planted decoys.