Literature workflows
Assess repeatable changes in A549 cell morphology
Read the task instructions and deliverables
A549 lung cancer cells were exposed to compounds for 48 hours and measured with Cell Painting, an imaging assay that records cell shape, texture, and staining. The 24 tables contain well profiles from eight plate layouts. Measurements were scaled using each plate's DMSO solvent controls; no further batch adjustment has been applied.
Assess how often compound treatments produce repeatable morphology at nominal doses of 0.04 and 10 μM (micromolar). Identify samples with convincing low-dose-specific repeatability, and explain how they fit with the overall dose trend. Support your selection with treatment comparisons and check how stable it is to reasonable analysis choices.
Use Metadata_broad_sample as sample identity and report those IDs. Compare samples with exactly three treatment wells at each of dose codes 1 and 6, the low and high categories. Use this same eligible sample universe for both doses. DMSO rows received solvent without a compound. Columns beginning Cells_, Cytoplasm_, or Nuclei_ are dimensionless measurements. For this exercise, profile repeatability means the median Spearman correlation across features between distinct replicate pairs at a given dose; Spearman correlation compares ranked values, using average ranks for ties. Use all supplied measurement features.
Write reasoning.md as 4–8 bullet points giving key observations, the inference each supports, and how controls and possible confounders were handled. Include the two dose-specific reproducibility rates and explain the background comparison used to obtain them.
Return EXACTLY, as the contents of answer.json:
{"low_dose_specific_samples": ["<Metadata_broad_sample>", "..."]}How was it graded?
Deterministic checks compare the submitted answer fields with the reference values and tolerances. An LLM judge assesses the remaining reasoning criteria and cannot override the deterministic results.
One point per passed criterion, divided by the total number of criteria. Unresolved judgments produce a score range. Critical failures are reported separately and do not add weight. No overall pass threshold is specified.
LLM judge: Not recorded in this packet
Task files
Choose an evidence file.
Grading criteria
Identify samples with repeatable morphology changes specific to low dose
Earns credit
Identify samples with convincing repeatability at 0.04 µM that is not retained at 10 µM. Submit low_dose_specific_samples as a JSON list of strings. After alias normalization and removal of optional members, set overlap must be at least 80%. Overlap means shared members divided by all distinct members in either set (Jaccard similarity); both omissions and extra members reduce it. Reference set (13 members): BRD-A02710418-003-11-8, BRD-A29322418-237-04-0, BRD-A29623586-001-01-3, BRD-A31811760-001-06-7, BRD-K16444452-001-09-1, BRD-K16542329-001-13-5, BRD-K28667793-001-28-1, BRD-K28971625-003-01-4, BRD-K33852358-001-09-9, BRD-K34415467-003-13-0, BRD-K75024268-001-01-0, BRD-K92731339-227-04-9, BRD-K99107520-001-24-1.
Does not earn credit
No credit for a missing or malformed answer, a field that is not a list of strings, or set overlap below the required threshold. Optional members do not count as errors.
Accepted alternatives
Order does not affect the set comparison. Repeated members are counted once by the scorer. Letter case is ignored. Leading and trailing whitespace is ignored. Accepted aliases: meptazinol → BRD-A02710418-003-11-8; potassium-canrenoate → BRD-A29322418-237-04-0; anisodamine → BRD-A29623586-001-01-3; miltefosine → BRD-A31811760-001-06-7; ibudilast → BRD-K16444452-001-09-1; zaprinast → BRD-K16542329-001-13-5; pyrazinamide → BRD-K28667793-001-28-1; ZD-7114 → BRD-K28971625-003-01-4; arofylline → BRD-K33852358-001-09-9; trimethobenzamide → BRD-K34415467-003-13-0; lesinurad → BRD-K75024268-001-01-0; perindopril → BRD-K92731339-227-04-9; felbamate → BRD-K99107520-001-24-1. Optional members, ignored in the comparison: BRD-A17655518-001-24-4, BRD-A18611368-001-01-5, BRD-A19736161-001-06-7, BRD-A19795905-001-08-0, BRD-A22305049-003-13-6, BRD-A24514565-001-09-7, BRD-A25234499-001-18-3, BRD-A26423207-001-01-0, BRD-A26711594-003-10-6, BRD-A32269193-001-01-5, BRD-A33711280-003-03-1, BRD-K09471561-001-18-2, BRD-K09711437-001-04-3, BRD-K11196887-001-21-2, BRD-K11717138-001-30-3, BRD-K16195444-003-26-0, BRD-K16295392-003-01-3, BRD-K16762525-001-01-6, BRD-K19939891-001-05-4, BRD-K21908111-001-01-8, BRD-K24219278-001-01-6, BRD-K24576554-001-04-8, BRD-K25204779-001-01-1, BRD-K26712974-001-01-5, BRD-K27938825-001-02-4, BRD-K28143534-003-25-9, BRD-K29653726-001-13-4, BRD-K31385749-003-01-3, BRD-K33141550-003-05-7, BRD-K35245662-001-01-2, BRD-K36627727-001-05-4, BRD-K80535353-001-01-4, BRD-K87278688-001-01-0, BRD-K89704198-001-12-3.
Input evidence
Metadata_broad_sample, Metadata_broad_sample_type, Metadata_dose_recode, Metadata_Well, Metadata_Plate_Map_Name, Metadata_Plate; all 1,745 measurement columns; every eligible sample at both doses · The full 447-sample universe supplies replicate and unrelated-treatment profiles for detecting low-dose-specific repeatability. Seven packet-only variants give 13 core and 34 optional boundary samples.
Reference evidence
S04 · reference/human_result.txt: S5 paragraphs 1–4; S25 paragraphs 1–4; answer_key.json: _fields.low_dose_specific_samples; reference/baseline_audit.json
The authors judge reproducibility relative to dose-, position-, and size-matched backgrounds and report per-compound evidence. The private key applies that reasoning to this exact upstream subset, not the full-study spherized data.
S11 · reference/originals/tableS2_compound_percent_replicating.tsv: assay=Cell Painting, dose=0.04 uM or 10 uM; compound and p_value columns
The human record reports separate compound-level results at the two doses. Exact packet exceptions differ from this full-study result, as documented in notes.
Recorded result / GPT 6 astra
Failed. low_dose_specific_samples: matched 3 of 13 reference members. Missing: BRD-A29322418-237-04-0, BRD-A29623586-001-01-3, BRD-K16444452-001-09-1, BRD-K16542329-001-13-5, BRD-K28667793-001-28-1, BRD-K28971625-003-01-4, BRD-K34415467-003-13-0, BRD-K75024268-001-01-0, BRD-K92731339-227-04-9, BRD-K99107520-001-24-1. Set overlap: 3/13 = 23.08%; required at least 80%.
Support the selected samples with low- and high-dose evidence
Earns credit
C-001 passes. For at least two reported samples from the key’s stable set, gives the correct sample IDs, both dose-specific replicate scores, and both matched-background thresholds or tail probabilities (the share of unrelated groups with a score at least as high). Shows credible low-dose repeatability and loss of that evidence at high dose. The background uses unrelated samples, accounts for dose and repeated well position, matches the three-replicate group size, and excludes the sample being tested. Connects these observations to the final low-dose-specific selection, rather than treating a fall in raw correlation as sufficient.
Does not earn credit
C-001 fails or the final selection is missing; either comparison uses the wrong sample or dose; background groups repeat the tested sample, pool its replicates as unrelated samples, or ignore the relevant dose, well position, or group size; only raw score changes or names are given.
Accepted alternatives
Any two samples from the stable reference set with reproducible calculations and the same low-dose-specific inference are accepted. Rounded scores and thresholds, or equivalent empirical tail probabilities, are sufficient.
Prerequisites
C-001
Input evidence
BRD-K16542329-001-13-5 (zaprinast), BRD-K34415467-003-13-0 (trimethobenzamide), and other reported samples; dose codes 1 and 6; sample/dose/well/plate metadata and measurement features · Replicates and distinct samples at the matching dose and well allow separate background comparisons for both doses. Specific paired comparisons distinguish a low-dose-only finding from a raw decline.
Reference evidence
S04 · reference/human_result.txt: S25 paragraphs 1–3 and S5 paragraphs 1–4; reference/treatment_statistics.csv: the selected sample IDs at each dose
The authors require replicate similarity beyond a matched background. A low-dose-specific finding must apply that test separately to both conditions. The packet-specific values are derived, not quoted as historical full-study values.
Recorded result / GPT 6 astra
This criterion explicitly requires C-001 to pass. That prerequisite fails, so the paired evidence for three selected samples cannot establish support for the required final selection. Lines 4–6 give both dose scores, but the stated backgrounds also omit full well-position matching.
Distinguish stable selections from uncertain boundary cases
Earns credit
C-001 passes. Uses a background comparison that accounts for dose, well position, and replicate count. Demonstrates a reasonable sensitivity check with reproducible observations: at least one reported sample from the key’s stable set retains the low-dose-specific finding, while at least one real borderline sample changes classification or has evidence close enough to a decision boundary to explain uncertainty. Identifies what changed in the analysis and connects stability to confidence in the selected set. Does not interpret high-dose non-detection as proof of no biological effect.
Does not earn credit
C-001 fails or is missing; the final background comparison omits a needed match; only says results are robust without a checked example; changes sample identities or confounds the comparison; gives a generic suggestion to repeat the assay; or claims failure to detect repeatability proves absence of a biological response.
Accepted alternatives
A different reasonable cutoff, repeated null sampling, or another source-supported matched-background check is acceptable. No specific cutoff, seed, historical wording, or six-variant exercise is required from the solver.
Prerequisites
C-001
Input evidence
All eligible sample/dose groups and their matching non-replicate pools; compare BRD-K16542329-001-13-5 (zaprinast) with BRD-A17655518-001-24-4 (ibuprofen), or other supported stable and boundary examples · The source rows support recomputing a background threshold after changing a tail cutoff or resampling unrelated groups. They contain strong and borderline cases rather than a preselected hit list.
Reference evidence
S04 · reference/human_result.txt: S5 paragraph 3; S25 paragraph 4; S26 paragraph 2; reference/baseline_audit.json: variants and dont_care
The authors check whether conclusions survive changes to backgrounds and sampling, and note that a restricted pool can affect estimates. The packet audit applies this judgment to stability of selected samples.
Recorded result / GPT 6 astra
This criterion explicitly requires C-001 to pass. The submitted set fails that prerequisite. The cutoff and centering checks are concrete, and the submission correctly avoids claiming that high-dose non-detection proves inactivity. However, those checks do not repair the failed selection or the missing full well-position comparison.
Use vehicle controls to identify a shared background
Earns credit
Supports the final background design with a quantitative comparison of DMSO profiles across different physical plates: correlations at the same well position exceed those at different positions. Explains that this similarity exists without compound treatment, so repeated placement can add a shared technical signal and must be represented in the background.
Does not earn credit
The final analysis rejects needed positional matching; the comparison uses compound-treated wells as if they were DMSO, includes self-pairs or only pairs within a single plate, reverses the control contrast, or gives a position keyword without observations and inference.
Accepted alternatives
Means or medians are both accepted. A clearly stated, valid subset of cross-plate DMSO pairs may be used if the submitted values can be reproduced and support the same inference.
Input evidence
Metadata_broad_sample=DMSO and Metadata_broad_sample_type=control; all measurement columns; compare pairs on different physical plates, split by equal versus unequal Metadata_Well · Same-position DMSO mean Spearman correlation is 0.3088178089 over 6,624 pairs; different-position mean is -0.0106970652 over 152,352 pairs. The corresponding medians are 0.3633442761 and -0.0194433161.
Reference evidence
S04 · reference/human_result.txt: S24 paragraph 3; S5 paragraphs 3–4; S25 paragraph 2
The authors identify technical artifacts in DMSO profiles, warn that repeated well placement can inflate similarity, and match the background by position.
S09 · Figure S5, upper-right Cell Painting non-spherized panel
The authors show position-dependent correlations in non-spherized profiles. This supports the qualitative control interpretation; packet-specific Spearman values are independently computed.
Recorded result / GPT 6 astra
The required quantitative comparison of cross-plate DMSO correlations at the same versus different well positions is missing. The submission gives DMSO triplet threshold ranges across layouts, but those ranges do not demonstrate the positional contrast. The human evidence shows that repeated position creates similarity even without compound treatment.
Interpret the dose trend after background matching
Earns credit
Uses the same 447 eligible samples at both doses, gives correctly computed dose-specific rates or counts under a stated matched-background analysis, and shows that repeatability rises at the high dose while substantial compound-associated signal remains beyond background. The inference concerns treatments exceeding dose-, position-, and group-size-matched backgrounds, not merely rising raw correlations or a changed group of samples. At the authors’ 95% rule the packet gives 102/447 at low dose and 249/447 at high dose (about 22.82% and 55.70%).
Does not earn credit
The final background design ignores a necessary comparison, the dose trend or remaining-signal conclusion is wrong or missing, raw correlations or different groups of samples replace the comparison, or the paper’s full-study rates replace packet calculations.
Accepted alternatives
Other reproducible rates from a defensible matched-background sensitivity analysis are accepted if both dose counts use the same group of samples and support the correct direction and remaining-signal inference. The exact 95% rule is not required.
Input evidence
All eligible treatment groups at dose codes 1 and 6; Metadata_broad_sample and all measurement columns · The same 447 samples produce 102 passing groups at low dose and 249 at high dose, an increase of about 32.89 percentage points.
Reference evidence
S04 · reference/human_result.txt: S5 paragraphs 2–3; S25 paragraphs 2–3
The authors conclude that percent replicating rises with dose after matching the background. Their full-cohort spherized 57–83% values are not the values for this packet.
S10 · Figure S7b, Cell Painting nonspherized row, 0.04 and 10 μM panels
The author figure also shows rising reproducibility in the non-spherized condition. Its Pearson, full-cohort percentages differ from the explicitly defined subset/Spearman quantities in this task.
Recorded result / GPT 6 astra
The submission correctly uses the same 447 samples and reports higher repeatability at high dose. However, its rates use pooled, layout-matched, or column-matched backgrounds. Matching a column does not account for the full repeated well position. Thus the rates do not satisfy the required background comparison. This fails independently of C-001; the issue is the background design, not a requirement to use exactly the historical cutoff.
Judge individual treatments against their own backgrounds
Earns credit
Uses the correct final matched-background design and remaining-signal conclusion, and compares two real treatments at the same dose: one fails its own matched threshold even though its raw replicate correlation is higher than that of another treatment that passes. Gives sample identities, scores, thresholds, and correct results, and explains why a high raw correlation alone is not enough while some compound-associated repeatability remains after matching.
Does not earn credit
The final background design or remaining-signal conclusion is wrong or missing; scores or thresholds come from wrong sample/dose groups; only assay names or historical conclusions are given; no valid higher-score failure versus lower-score pass comparison is shown; or the response claims all similarity is technical or all positive correlations are reproducible.
Accepted alternatives
Any same-dose pair with a reproducible reversal under the stated matched-background method is accepted. Sensibly rounded correlations are sufficient. The earlier 95% example is balicatib at low dose (0.756239 below 0.779190), compared with tivantinib (0.353566 above 0.261022).
Input evidence
Dose code 1: BRD-K27204852-001-01-9 (balicatib) at P24 and BRD-K33379087-001-07-5 (tivantinib) at H18; all other eligible samples at those same wells and dose · Balicatib has score 0.756239 and threshold 0.779190 and fails. Tivantinib has score 0.353566 and threshold 0.261022 and passes. Their ordering by raw correlation therefore does not order their evidence above background.
Reference evidence
S04 · reference/human_result.txt: S25 paragraphs 1–3; S5 paragraphs 3–4
The authors judge each treatment against a matched non-replicate distribution, because background similarity can be raised by position. They find real replicate signal despite technical effects. Named subset values are recomputed in reference/treatment_statistics.csv.
Recorded result / GPT 6 astra
The submission compares higher-scoring trimethobenzamide with lower-scoring selected samples, but its failure-versus-pass comparison uses column-only backgrounds. This fails the criterion's explicit requirement for a correct final matched-background design. Under the full-position reference comparison, trimethobenzamide's low-dose score of 0.745316 exceeds its 0.666917 threshold and passes, so the submitted example does not establish the required reversal.