Files
OpenJarvis/tests/evals/datasets
4523715bff fix(ci): gate live HuggingFace Hub download tests behind hub marker (#507)
The two eval-dataset suites that download real corpora from the
HuggingFace Hub at runtime were running in the default CI lane. When the
Hub was unreachable or rate-limited they failed and reddened `main` even
though no code changed — confirmed by #506 (docs-only) failing on merge
while its own PR run passed an hour earlier. They also dominated CI
wall-time (~33 min of downloads + retry backoff on the failing run).

Add a `hub` pytest marker, apply it to both suites via module-level
`pytestmark`, and exclude it from the default CI lane
(`-m "not live and not cloud and not hub"`). The tests stay runnable on
demand with `pytest -m hub`.

The ADP provider swallows per-config download errors and returns 0
records on a network failure, so a Hub outage surfaced there as
`assert 1 <= 0` (not an exception) — it could not be made non-flaky by
exception handling alone, only by gating.

Coverage holds: removing these from CI drops total from 60.92% to
~60.73% (paranoid worst case 60.18%), still above the 60% gate.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 16:31:50 -07:00
..
2026-03-12 17:29:39 +00:00
2026-03-12 17:29:39 +00:00
2026-03-12 17:29:39 +00:00
2026-03-12 17:29:39 +00:00
2026-03-12 17:29:39 +00:00
2026-03-12 17:29:39 +00:00