fix(evals): add tool_choice=auto + fix traces thread safety (#163)

Two fixes for eval accuracy and stability:

1. TauBench agent: add tool_choice="auto" to match tau2's native
   LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
   below leaderboard because the models don't receive explicit
   tool-calling guidance.

2. SystemBuilder: apply self._traces flag to config.traces.enabled.
   Previously builder.traces(False) was a no-op — traces stayed
   enabled, creating SQLite connections in the main thread that
   crashed when accessed from ThreadPoolExecutor worker threads
   in GAIA evals ("SQLite objects created in a thread can only be
   used in that same thread").

Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Avanika Narayan
2026-03-31 19:29:55 -07:00
committed by GitHub
co-authored by Jon Saad-Falcon Claude Opus 4.6
parent 43b3a59033
commit 39a8082af3
2 changed files with 7 additions and 1 deletions
@@ -177,6 +177,7 @@ class JarvisHalfDuplexAgent:
"temperature": self._temperature,
"max_tokens": self._max_tokens,
"tools": openai_tools,
"tool_choice": "auto",
}
# Disable thinking mode for local models (Qwen3.5 etc.)
# to avoid <think> tags interfering with tool call parsing
+6 -1
View File
@@ -503,10 +503,15 @@ class SystemBuilder:
# Resolve model
model = self._resolve_model(config, engine)
# Compute telemetry_enabled once
# Compute telemetry_enabled and traces_enabled once
telemetry_enabled = (
self._telemetry if self._telemetry is not None else config.telemetry.enabled
)
traces_enabled = (
self._traces if self._traces is not None else config.traces.enabled
)
# Apply traces flag to config so downstream code respects it
config.traces.enabled = traces_enabled
gpu_monitor = None
energy_monitor = None
if telemetry_enabled and config.telemetry.gpu_metrics: