LabBench
Assess the cell-type information carried by non-promoter transcription signatures
Read the task instructions and deliverables
# Assess the cell-type information carried by non-promoter transcription signatures Researchers used newly made RNA to infer patterns of transcription-factor activity across cell samples. Review the clustering figure and source excerpts in documents/. Write cell-signature-review.md. Begin with one clear verdict on what the analysis establishes about the sampled cell groups. Defend it against the strongest competing interpretation using specific sample labels and figure features. Explain the shared and restricted regulator patterns, the strength of the study-to-study comparison, and the limits on applying the result to new samples. Distinguish what the records show from what remains untested. Cite files and locations. This is a retrospective review of completed work; no new experiment plan or raw-data analysis is required.
How was it graded?
Agent-judged binary criteria.
Fraction of criteria passed; every criterion has weight one.
LLM judge: gpt-6-astra
Task files
Choose an evidence file.
Grading criteria
Retain a bounded cell-associated signature interpretation
Earns credit
Commits to retaining evidence of cell-associated activity patterns in the sampled groups, supported by both same-group clustering across named studies and restricted versus shared motif patterns. Limits the claim to these displayed groups rather than claiming an established classifier for every cell type or dismissing all grouping as a study effect.
Does not earn credit
Rejects the cell-associated interpretation, claims universal or fully separate cell identification, gives no committed verdict, or offers the accepted verdict without the two kinds of evidence.
Accepted alternatives
May call this a proof of concept or a descriptive cell-group signature.
Weight: 1
Input evidence
Reference evidence
S007 · Lines 66, 70 and 74
Authors infer robust cell signatures from clustering and regulator patterns.
Recorded result / GPT 6 astra
The opening verdict retains cell-associated information within the sampled groups without claiming complete separation or validated prediction. Named cross-study clusters and shared versus restricted regulator patterns support that conclusion.
Use a same-cell comparison across studies
Earns credit
Names a displayed cell group containing samples from at least two distinct study labels, gives the actual labels, and uses their joint clustering to support the retained cell-associated interpretation. The final answer must retain a cell-associated interpretation limited to the displayed groups, supported by the comparisons required here, without claiming universal or fully separate cell identification or dismissing all grouping as a study effect. An explicit correction of an earlier error is acceptable.
Does not earn credit
Uses only two numbered rows from one study, compares different cell groups, omits the cell-associated inference, or denies that the named cross-study cluster supports it. Also fails if a comparison required here is wrong or missing, or the final answer omits or contradicts the bounded cell-associated interpretation, including by claiming universal or fully separate cell identification. Omitted unrelated motif-pattern evidence alone does not fail this criterion.
Accepted alternatives
For example, Allen2014 with Chen2015 in HCT116, or Gardini2014 with Laitem2015 in HeLa. The mixed-study AC16 pairs Luo2014-1/Danko2013-1 and Luo2014-2/Danko2013-2 are also valid cross-study evidence. This is one accomplishment, not an extra point for each pair.
Weight: 1
Input evidence
Reference evidence
S007 · Line 66
Authors rely on consistency across independently generated data.
S019 · Page 1, AC16 subtree
The source artwork shows the two mixed-study pairs.
Recorded result / GPT 6 astra
The report correctly identifies multiple studies within HeLa and HCT116, including Allen2014 and Chen2015 in HCT116. It uses their clustering to support the required bounded cell-associated interpretation.
Check different cell groups within a study label
Earns credit
Uses Danko2013-1/-2 in AC16 versus Danko2013-3 in MCF7 to explain why a study label alone cannot explain all of the displayed grouping; retains the cell-associated interpretation. The final answer must retain a cell-associated interpretation limited to the displayed groups, supported by the comparisons required here, without claiming universal or fully separate cell identification or dismissing all grouping as a study effect. An explicit correction of an earlier error is acceptable.
Does not earn credit
Calls the Danko2013 rows one cell group, fails to compare both groups, or uses the pattern to claim grouping is solely by study. Also fails if a comparison required here is wrong or missing, or the final answer omits or contradicts the bounded cell-associated interpretation, including by claiming universal or fully separate cell identification. Omitted unrelated motif-pattern evidence alone does not fail this criterion.
Weight: 1
Input evidence
Reference evidence
S019 · Page 1, Danko2013 rows
Human artwork places these rows in different cell-associated branches.
S007 · Line 66
The stated interpretation is cross-study cell identity.
Recorded result / GPT 6 astra
The report correctly places Danko2013-1/-2 in AC16 and Danko2013-3 in MCF7. It explains why this split weakens a study-only explanation while retaining the bounded cell-associated conclusion.
Keep the K562 evidence narrower than the multi-study groups
Earns credit
Contrasts the two Core2014 K562 rows with a named group containing multiple study labels, and concludes that K562 coherence alone is not an independent cross-study demonstration.
Does not earn credit
Treats the two Core2014 rows as two independent studies or makes the same cross-study claim for K562 without the contrast.
Accepted alternatives
HeLa, HCT116, MCF7, LnCAP, AC16 or T/B-Cells can serve as the contrast.
Weight: 1
Input evidence
Reference evidence
S019 · Page 1, K562 and other study-labeled branches
The artwork gives only Core2014 for K562.
Recorded result / GPT 6 astra
The report contrasts named multi-study groups with K562, whose only rows are Core2014-1/-2. Its explanation that cell identity and study source remain confounded for K562 correctly limits the independent study support.
Respect the combined hESC/HEK293T group
Earns credit
Explicitly notes the combined hESC/HEK293T label and its Chen2014, Saponaro2014 and Fong2014 branches, and limits the conclusion to grouping within this combined label rather than claiming the figure proves separate identification of hESC and HEK293T.
Does not earn credit
Misses the combined label, invents an exact cell assignment for each study row, or claims clean separation of hESC from HEK293T without a source mapping.
Accepted alternatives
Can say the precise identities require a matching sample map; no inferred map is required. May also explain that PITX2/OTX2/IKZF1/HMGA1 marks the combined display group without assigning those activities specifically to either hESC or HEK293T. This does not earn a separate point.
Weight: 1
Input evidence
Reference evidence
S019 · Page 1, Pink group and its five leaves
The human artwork preserves the combined cell label.
Recorded result / GPT 6 astra
The report identifies the combined hESC/HEK293T label and its Chen2014, Saponaro2014, and Fong2014 branches. It explicitly avoids assigning individual rows to either cell identity and does not claim separate identification.
Bound the NFKB interpretation to the shown immune group
Earns credit
Locates the NFKB1/2–REL/RELB bracketed columns in the T/B-Cells rows, explicitly compares the same column region with at least two named non-T/B display groups, and explains why the visible contrast supports a restricted signature in these samples. Distinguishes a missing visible activity mark from a demonstrated depletion call or universal biological absence.
Does not earn credit
Only repeats the NFKB-to-T/B annotation, leaves the comparison groups unnamed, compares different column regions, or interprets unmarked cells as proved depletion or universal absence.
Accepted alternatives
NF-kappaB is an acceptable spelling.
Weight: 1
Input evidence
Reference evidence
S007 · Line 70
The authors interpret NFKB as restricted to T/B cells in this analysis.
S019 · Page 1, NFKB bracket and aligned T/B-Cells, MCF7 and LnCAP rows
The original artwork preserves the column-aligned contrasts.
S008 · Line 42, encoding sentence
The method distinguishes absence from depletion.
Recorded result / GPT 6 astra
The report correctly maps NFKB1/2–REL/RELB to T/B-Cells and cautions against treating white cells as depletion or universal absence. However, it never compares those same columns with two named non-T/B groups, as required to demonstrate the restricted pattern.
Explain why widespread activity is weak identity evidence
Earns credit
Uses the JUN/FOS-associated left block in at least two named, separately branching cell groups to show that shared activity is compatible with distinct overall signatures. Explains that a factor with the same encoded state in two rows contributes zero to their Euclidean distance, so discrimination must come from differing features. Does not assume that every column or row in the entire bracketed block is identical.
Does not earn credit
Only labels the block shared or nondiscriminating, omits the named cross-group comparison or the distance consequence, or treats the entire bracketed block as an exactly constant matrix.
Accepted alternatives
Any two named groups with separate branches and visible activity in the JUN/FOS region are valid. The equal-state distance argument must still be given.
Weight: 1
Input evidence
Reference evidence
S007 · Line 74, first two sentences
The authors describe broadly shared Fos and Jun activity.
S008 · Line 42, encoding and clustering sentences
The zero contribution follows mathematically from the stated Euclidean method, not from measuring branch lengths.
S019 · Page 1, JUN/FOS block and HeLa/K562 branches
The source artwork supplies the named comparison.
Recorded result / GPT 6 astra
The report describes the JUN/FOS block as broadly shared, but does not compare it in two named, separately branching groups. It also omits the required explanation that equal encoded states contribute zero to Euclidean distance.
Interpret a multi-factor signature rather than a single decisive marker
Earns credit
Retains a cell-associated activity-pattern interpretation limited to the displayed groups, supported by the reciprocal and shared-block comparisons required here, and makes a reciprocal comparison of two named groups: identifies a restricted callout region relatively marked in the first group, checks that same region in the second, then identifies a different restricted region relatively marked in the second and checks it in the first. Uses the shared JUN/FOS block to explain why a multifeature pattern, rather than common activity alone, supports their distinction. Does not infer numerical states from white cells. Does not claim universal or fully separate cell identification, dismiss all grouping as a study effect, or treat these motif comparisons alone as evidence of cross-study robustness. An explicit correction of an earlier error is acceptable.
Does not earn credit
Lists two group-to-factor annotations without reciprocal same-column comparisons, treats all named factors as exclusive universal markers, omits the shared-block comparison, gets a required comparison wrong, or omits or contradicts the bounded cell-associated interpretation. Claims of universal or fully separate cell identification, an artifact-only verdict, numerical states inferred from white cells, or cross-study robustness inferred from these motif comparisons alone also fail. Omitted unrelated named cross-study clustering evidence alone does not fail this criterion.
Accepted alternatives
Any clearly supported group pair is acceptable, including HeLa and MCF7, if both same-column comparisons and the shared-block comparison are given and support the bounded cell-associated interpretation. Named cross-study clustering evidence is not required for this descriptive accomplishment; its omission still fails C-001 and cannot establish cross-study robustness.
Weight: 1
Input evidence
Reference evidence
S007 · Lines 70 and 74
The human interpretation combines restricted and broadly shared activity.
S019 · Page 1, HeLa and MCF7 rows at both restricted brackets and the shared left block
The original artwork preserves each reciprocal comparison.
Recorded result / GPT 6 astra
The bounded conclusion is correct, but the table only lists group-to-regulator annotations. It does not check each of two restricted regions in both named groups or compare their shared JUN/FOS activity. These required comparisons cannot be supplied by the general statement that signatures contain multiple features.
Interpret the clustering input as activity calls
Earns credit
Correctly gives the presence/absence/depletion encoding and Ward/Euclidean method, distinguishes inferred activity from protein abundance, and explains a concrete consequence of the encoding: for one feature, presence versus depletion differs by 2 and contributes 4 to squared Euclidean distance, whereas presence versus absence differs by 1 and contributes 1. States that these are consequences of the encoded states, not activity magnitudes read from row colors or distances measured from the drawn branches.
Does not earn credit
Merely copies the encoding and method, treats all unequal state pairs as equally distant, confuses the one-feature calculation with a full Ward merge cost, or interprets colors as protein concentrations.
Accepted alternatives
Equivalent equations or a one-feature table showing differences of 2 and 1 and squared contributions of 4 and 1 are acceptable. This is not a calculation of the full Ward merge cost.
Weight: 1
Input evidence
Reference evidence
S008 · Line 42, encoding and clustering sentences
The difference between 1 and -1 is 2; between 1 and 0 is 1. Squaring gives 4 and 1. These are direct consequences of the human method.
S007 · Line 66, first three sentences
The authors define the activity inference and data source.
Recorded result / GPT 6 astra
The report correctly states the 1/0/−1 encoding, Ward clustering, and Euclidean distance, and distinguishes inferred activity from TF abundance. It omits the required one-feature calculation: differences of 2 versus 1 and squared-distance contributions of 4 versus 1.
Separate grouping of known samples from prediction of new ones
Earns credit
Explains why a named cross-study success, such as the Luo2014/Danko2013 AC16 pairs, is descriptive evidence rather than a held-out accuracy estimate: both studies are included in constructing the displayed joint clustering, and neither is shown being assigned by a fixed rule derived without it. Distinguishes this missing evaluation for a new sample from the additional untested extension to a cell identity absent from the display, retaining the observed cell-associated result.
Does not earn credit
Only says there is no validation or accuracy, treats a mixed-study pair as a held-out test, fails to explain inclusion in the joint clustering, or treats missing generalization evidence as a refutation of the displayed pattern.
Accepted alternatives
Another correctly named mixed-study cell-group branch can replace AC16 if the answer explains joint inclusion and both limits on generalization.
Weight: 1
Input evidence
Reference evidence
S018 · Line 51
The authors call joint Ward clustering a proof of concept.
S007 · Line 66
Describes clustering of collected samples and cross-lab consistency.
S019 · Page 1, AC16 subtree
The original includes both studies in the same tree.
Recorded result / GPT 6 astra
The report names mixed-study successes and says prediction on new samples and unrepresented cell types remains untested. However, it does not explain that both named studies helped construct the joint clustering and that neither was assigned using a fixed rule built without it. The criterion explicitly requires this explanation beyond noting missing validation.
Use unfamiliar regulators as candidates, not proved causes
Earns credit
Uses a named restricted regulator set as a candidate contributor and compares the evidentiary strength of its group association across a multi-study group and the single-study K562 group. For example, contrasts the HeLa CXXC1/NR4A2/PPARA association across its study labels with GATA1/2/LHX6/STAT6/SMAD2 in Core2014-1/-2. Explains that the latter lacks an independent within-group study contrast, while even the stronger cross-study association remains an inferred activity association without a displayed regulator intervention or causal direction test. Does not rank biological causal effects from the figure.
Does not earn credit
Only appends association-is-not-causation to a factor list, omits the comparison of study support, treats two Core2014 rows as independent studies, or claims cross-study consistency establishes causation.
Accepted alternatives
Another named multi-study group with a correctly mapped restricted regulator set can replace HeLa. The comparison with single-study K562 and the separate causal limit remain required.
Weight: 1
Input evidence
Reference evidence
S007 · Line 70, final sentence; line 66, cross-lab consistency sentence
The authors nominate possible contributors and use cross-lab agreement as support, without proving causal effects.
S019 · Page 1, HeLa and K562 study labels and regulator brackets
The artwork shows unequal study support for the associations.
Recorded result / GPT 6 astra
The report lists HeLa and K562 regulator sets and correctly warns that inferred activity does not prove causation. It also discusses K562's single-study limitation elsewhere. But it never compares the study support for these named regulator associations, which is required beyond a factor list and a general causal caveat.
Avoid turning cross-study agreement into a claim of no study effects
Earns credit
Uses both agreement across named study labels in a cell group and remaining within-group structure, such as paired Gardini2014 or Liu2014 rows, to explain that a cell-associated signal can coexist with residual study or condition differences. Does not conclude that the cell signal is solely a study artifact. The final answer must retain a cell-associated interpretation limited to the displayed groups, supported by the comparisons required here, without claiming universal or fully separate cell identification or dismissing all grouping as a study effect. An explicit correction of an earlier error is acceptable.
Does not earn credit
Claims study effects have been eliminated, attributes every subbranch to a proved batch effect, or rejects the cell-associated signal despite the named cross-study evidence. Also fails if a comparison required here is wrong or missing, or the final answer omits or contradicts the bounded cell-associated interpretation, including by claiming universal or fully separate cell identification. Omitted unrelated motif-pattern evidence alone does not fail this criterion.
Accepted alternatives
Can state that branch structure does not by itself separate protocol effects from treatment or biological variation.
Weight: 1
Input evidence
Reference evidence
S007 · Line 66
Authors claim robust signatures despite slight protocol variation, not that all variation disappears.
S019 · Page 1, HeLa and MCF7 subtrees
The original contains same-study leaf pairs within broader groups.
Recorded result / GPT 6 astra
The report combines named cross-study grouping with close same-study pairs, including Gardini2014 and Liu2014. It retains the bounded cell-associated interpretation while explaining that the tree neither eliminates batch effects nor identifies the conditions behind the numbered samples.
Keep the reported sample counts separate from plot membership
Earns credit
Uses the visible figure to distinguish 39 labeled rows, eight displayed group labels and 351 factor columns from the source sampling statement about eight cell types and 18 labs. Explains that a row is not a lab and that the combined hESC/HEK293T label prevents treating eight display groups as eight independently resolved single cell labels.
Does not earn credit
Equates 39 rows with 39 labs, 351 with samples, or eight groups with eight independently resolved single cell labels; merely repeats counts without explaining the distinction.
Accepted alternatives
Exact lab membership may remain unresolved; no requirement to invent an explanation for the manuscript count.
Weight: 1
Input evidence
Reference evidence
S019 · Page 1, All row and group labels
The artwork confirms the same row membership.
S007 · Line 66
The manuscript uses cell types and labs as its units.
Recorded result / GPT 6 astra
The report correctly states eight source cell types, 18 labs, and 351 factors, and recognizes composite labels. It omits the 39 labeled rows and eight displayed group count and does not explain that a plotted row is not a lab. The required distinction among these units is therefore incomplete.
Keep the evidence about non-promoter sites separate from a promoter comparison
Earns credit
Connects the non-promoter site selection stated in study-excerpt.tex to the scope of Figure 5: the displayed grouping assesses signatures from those sites, but does not by itself show that they separate cell groups better than promoter-derived signatures because no matching promoter result is supplied. Does not treat that missing comparison as evidence that promoters and non-promoters perform equally.
Does not earn credit
Claims the displayed figure directly proves better cell separation than promoters, treats the shared JUN/FOS block as a promoter control, infers equal performance from the missing comparison, or gives a general caveat without connecting the stated site selection to the absent matched comparison.
Accepted alternatives
May say that promoter superiority cannot be independently assessed from this packet, while retaining the evidence about the displayed non-promoter signatures. No proposed experiment or reconstruction of the missing result is needed.
Weight: 1
Input evidence
Reference evidence
S007 · Line 66, final sentence; line 74, final two sentences
The authors make promoter-versus-non-promoter claims as a separate comparison. Those claims remain the historical reference; the supplied figure alone cannot independently verify them.
Recorded result / GPT 6 astra
The required promoter-comparison assessment is absent. The report does not explain that the non-promoter grouping cannot establish superiority over promoter signatures because no matching promoter result is supplied.
Limit regulator coverage to factors with a usable motif model
Earns credit
Connects the study excerpt’s restriction to TFs with a high quality PWM to the 351-factor activity display, and explains that the reported patterns cover that selected set of factors, not a complete inventory of every possible regulator of cell identity. Explains that a regulator missing from the assessed set cannot be declared inactive from this figure.
Does not earn credit
Treats the 351 displayed factors as every possible regulator, calls a factor outside the assessed set inactive, or mentions incomplete coverage without tying it to the stated high-quality motif-model selection and the displayed factor set.
Accepted alternatives
May describe a PWM as a DNA sequence motif model or use the source term without expanding it. No outside estimate of the total number of human transcription factors is required.
Weight: 1
Input evidence
Reference evidence
S018 · Line 47, pipeline from PSSM database and motif scans to activity predictions
The human analysis selects available sequence models before predicting factor activity; it does not measure all possible regulators.
S019 · Page 1, Transcription Factor Activity (N=351) heading
The original artwork fixes the size of the displayed factor set. The earlier database size in S018 is not substituted for this figure’s count.
Recorded result / GPT 6 astra
The report states the high-quality PWM requirement and the 351-factor display. It does not explain that this selected set is not a complete inventory of regulators or that a regulator outside the assessed set cannot be declared inactive. Mentioning motif coverage as a transfer assumption does not supply those conclusions.