mirror of
https://github.com/open-jarvis/OpenJarvis.git
synced 2026-07-27 21:05:34 +00:00
evals: expose raw question / problem_statement in dataset metadata
GAIADataset and SWEBenchDataset both build a formatted prompt (`rec.problem`) and discard the raw input text. The hybrid paradigm runner needs the raw text so it can apply the GAIA / SWE-bench answer-format instructions from `agents.hybrid._prompts.format_prompt(task)` — otherwise paradigms see a double-wrapped prompt and the GAIA "FINAL ANSWER: <x>" rule is missing. Additive change — adds `metadata["question"]` for GAIA and `metadata["problem_statement"]` for SWE-bench. No existing consumer breaks; the runner now picks up the raw text correctly. Verified end-to-end with `python -m openjarvis.agents.hybrid.runner --cell advisors-gaia-qwen9b-opus-3`: 3/3 tasks, acc=0.667, $0.04 — matches the hybrid harness's cell exactly.
This commit is contained in:
@@ -159,6 +159,7 @@ class GAIADataset(DatasetProvider):
|
||||
|
||||
metadata = {
|
||||
"task_id": task_id,
|
||||
"question": question,
|
||||
"level": level,
|
||||
"file_name": file_name,
|
||||
"file_path": str(file_path) if file_path else None,
|
||||
|
||||
@@ -142,6 +142,7 @@ class SWEBenchDataset(DatasetProvider):
|
||||
|
||||
metadata: dict[str, Any] = {
|
||||
"instance_id": instance_id,
|
||||
"problem_statement": problem_statement,
|
||||
"repo": repo,
|
||||
"base_commit": base_commit,
|
||||
"hints_text": hints_text,
|
||||
|
||||
Reference in New Issue
Block a user