evals: expose raw question / problem_statement in dataset metadata

GAIADataset and SWEBenchDataset both build a formatted prompt (`rec.problem`)
and discard the raw input text. The hybrid paradigm runner needs the raw
text so it can apply the GAIA / SWE-bench answer-format instructions from
`agents.hybrid._prompts.format_prompt(task)` — otherwise paradigms see a
double-wrapped prompt and the GAIA "FINAL ANSWER: <x>" rule is missing.

Additive change — adds `metadata["question"]` for GAIA and
`metadata["problem_statement"]` for SWE-bench. No existing consumer
breaks; the runner now picks up the raw text correctly.

Verified end-to-end with `python -m openjarvis.agents.hybrid.runner
--cell advisors-gaia-qwen9b-opus-3`: 3/3 tasks, acc=0.667, $0.04 — matches
the hybrid harness's cell exactly.
This commit is contained in:
Andrew Park
2026-05-13 15:27:48 -07:00
parent a03e659bc9
commit e21e1bbad6
2 changed files with 2 additions and 0 deletions
+1
View File
@@ -159,6 +159,7 @@ class GAIADataset(DatasetProvider):
metadata = {
"task_id": task_id,
"question": question,
"level": level,
"file_name": file_name,
"file_path": str(file_path) if file_path else None,
@@ -142,6 +142,7 @@ class SWEBenchDataset(DatasetProvider):
metadata: dict[str, Any] = {
"instance_id": instance_id,
"problem_statement": problem_statement,
"repo": repo,
"base_commit": base_commit,
"hints_text": hints_text,