admin/Claw3D

Fork 0

mirror of https://github.com/iamlukethedev/Claw3D.git synced 2026-07-30 11:12:32 +00:00

Files

T

4e552967d9 [PRIORITY][FEAT] merge runtime profiles, office systems, doctor, and vera lane convergence (#109 )

* fix: include kanbanImmersive in immersiveOverlayActive calculation

When Kanban board is open, HUD elements (camera preset buttons, edit toolbar, overlays) should be suppressed. The kanbanImmersive flag was defined but not included in the immersiveOverlayActive condition, causing HUD elements to remain visible.

This fix adds kanbanImmersive to the immersiveOverlayActive calculation so HUD elements are properly hidden when the Kanban board is open.

Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>

* Fix: Hide mini status bar when Kanban immersive overlay is open

Wraps the bottom-left mini status bar (showing agent stats, vibe score, and
control hints) with !immersiveOverlayActive check to match the behavior of
other HUD elements like camera controls and toolbar.

This ensures the status bar is properly hidden when the Kanban board or any
other immersive overlay is active, maintaining a clean immersive experience.

Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>

* chore: drop unrelated package-lock line from branch

Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>

* universal-backend-plan

* backend-neutral runtime seam

* package.json update

* feat: add Hermes gateway adapter as alternative to OpenClaw

Adds a WebSocket adapter that lets Claw3D connect to a Hermes AI agent
runtime without any changes to the frontend. The adapter implements the
full Claw3D gateway protocol and bridges it to the Hermes HTTP API.

Changes:
- server/hermes-gateway-adapter.js: WebSocket bridge implementing the
  Claw3D gateway protocol against the Hermes HTTP API. Supports all
  core methods (agents, sessions, chat streaming, cron, config, files,
  approvals) and multi-agent orchestration via spawn_agent/delegate_task
  tools. Persists conversation history to ~/.hermes/clawd3d-history.json.
- scripts/clawd3d-start.sh: All-in-one startup script that launches
  Hermes, the adapter, and the Next.js dev server with auto port
  conflict resolution. Alias as `claw3d` for convenience.
- src/features/office/hooks/useCronAgents.ts: Hook that polls the
  gateway for cron-scheduled agents and surfaces them in the 3D office.
- package.json: adds `hermes-adapter` npm script
- .env.example: documents Hermes config vars
- docs/hermes-gateway.md: setup guide and protocol reference

Usage:
  npm run hermes-adapter   # start adapter (connect to http://localhost:8642)
  npm run dev              # start Claw3D, point browser at localhost:3000
  # or: bash scripts/clawd3d-start.sh  (starts everything automatically)

Both OpenClaw and Hermes are supported simultaneously — the gateway URL
in NEXT_PUBLIC_GATEWAY_URL determines which backend Claw3D connects to.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat: add read_agent_context tool for cross-agent coordination

Agents can now read each other's conversation history via the
read_agent_context tool, enabling the orchestrator to check what
a sub-agent has done before re-delegating work.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat: wire Hermes office UX and role-aware runtime updates

* feature update - demomode & hermes adapter

* fix lint blockers

* lintfix #2

* fix: stabilize retro office camera preset callbacks

* Initial plan

* fix: stabilize retro office overview preset hooks

Agent-Logs-Url: https://github.com/gsknnft/Claw3D/sessions/9cc71555-591e-44cf-aec4-25affbdcb405

Co-authored-by: gsknnft <123185582+gsknnft@users.noreply.github.com>

* feat: add truthful backend selection, Hermes adapter hardening, and demo gateway mode

* fix: address bugbot review and finalize backend selection

* fixed - onboarding and hermes calls

* office systems roadmap

* feat specs in docs

* specs ready

* feat: continue custom runtime seam and gateway alignment

* custom lane wired

* feat: add custom runtime provider path and office runtime alignment

* office_sys prep

* tighten multi-floor runtime spec

* multi-floor v1 implementation

* moved floor nav

* runtime architecture specs

* claw3doctor specs

* feat: add first pass claw3doctor diagnostics

* docs: align roadmap with runtime profiles and office systems priorities

* feat: expand claw3doctor provider diagnostics and json output

* feat: improve claw3doctor formatting and multi-runtime diagnostics

* feat: formalize runtime profile resolution

* feat: expand claw3doctor profile health diagnostics

* feat: polish claw3doctor output and tunnel remediation

* feat: add claw3doctor profile scoping and failure classification

* docs: define claw3doctor v1 boundary and v2 backlog

* test: fix stale claw3doctor branch expectations

* test: fix stale expectations on claw3doctor branch

* fix(claw3doctor): scope provider-specific checks to --profile / --all-profiles flags

PR #101 finding:
- Medium: --profile <adapter> and --all-profiles only scoped the profile health
  probe loop; OpenClaw/Hermes/Demo/Custom check blocks still ran based on the
  selected adapter type in runtimeContext, making CLI output misleading.

Fix: introduce adapterInScope(adapterType, defaultBehavior) helper in main().
  - --profile <adapter>  -> only that adapter's checks run
  - --all-profiles       -> all adapter checks run
  - no flag              -> falls back to existing shouldRun* predicate (unchanged)

Also:
- Export parseDoctorArgs from claw3doctor-core.mjs (removed duplicate in script)
- Add test suites: parseDoctorArgs flag parsing (6 cases) and
  adapterInScope scoping semantics (4 cases) — 10 new tests, all green

* fix(office): persist per-floor selectedAgentId on focusLocalAgent + wire officeFloors runtime state

PR #96 findings:
- Medium #1: focusLocalAgent now writes selectedAgentId back to floorRosterCache
  so handleSelectFloor restores the agent the user last picked on each floor
  rather than snapping back to the hydration-time suggestion.
- Medium #2: Add useEffect that calls settingsCoordinator.schedulePatch with
  officeFloors[activeFloorId] patch on every status/gatewayUrl change, writing
  status, gatewayUrl, lastKnownGoodAt, lastErrorCode, and lastErrorMessage so
  the persisted floor runtime state actually tracks live connection transitions.

* fix unkept changes

* fix audit findings - cross-floor misattribution & officefloors silent drops

* pushed changes

* multi-agentic runtime & chat bubble fix

* partial parity with office-sys-next

* claw3doctor parity

* partial parity with v_lane

* merged feat/office-systems-next -> merge_sys

* parity across PR branches

* rm *.orig postmerge

* full parity across unmerged PRs & main

* fix lukes findings

* fix findings - bigger chatbox

* real local upload path

* fixed file upload, MIME integgration

* minor fix

* fix lukess findings

* deleted *.orig

* three bugs fixed - gatewayclient, coord, claw3doctor

* address findings

* fix lukes findings

* fix findings #2

* fix: ignore temporary skill-agent names during identity recovery

* fix: preserve stable identity names during temp-name recovery

* fix(gateway): correct disconnect race and token-blanking on adapter switch

- disconnect() now checks actual connection status rather than selectedAdapterType,
  which may already reflect the target adapter when the effect fires. Prevents stale
  WebSocket clients from persisting after switching to local/claw3d/custom backends.
- setSelectedAdapterType() falls back to loadedGatewaySettings.current.profiles token
  when the in-memory adapterProfiles entry has an empty token (sanitized API form).
  Prevents saved tokens from being cleared when switching between backends.

Closes Luke findings: High (stale gateway on floor switch), Medium (token blanking).

Authored-By: GSKNNFT

* feat(gateway): loopback bypass, control-ui remap, operator.read scope, 75ms connect

Cherry-picked clean additions from pr/gsknnft-2 (fix/reduce-gateway-connect-delay):

- proxy-url.ts: resolveStudioProxyGatewayUrl() now accepts optional upstreamGatewayUrl;
  loopback hosts (localhost/127.0.0.1/::1) bypass the Studio proxy and connect directly
- gateway-proxy.js: remap unauthenticated openclaw-control-ui connections to webchat-ui
  client ID so OpenClaw doesn't reject them as unknown clients
- GatewayBrowserClient.ts + nodeGatewayClient.ts: add operator.read scope to both
  browser and Node gateway clients for expanded access control
- GatewayBrowserClient.ts: reduce socket open→connect delay from 750ms to 75ms

GatewayClient.ts rewrites from that PR were intentionally excluded — they would
regress our disconnect-race fix, token-blanking fix, broken-regex fix, local/claw3d
adapter support, adapterProfiles type export, and private envelope path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(floor-nav): show only available floors per active adapter; block hang on unconfigured runtime

- floors.ts: add paperclip to FloorProvider; add listAvailableFloorsForAdapter() —
  lobby always visible, runtime floors only shown when their provider matches the
  active adapter, so demo-only users only see Lobby
- OfficeFloorNav: accept activeAdapterType prop, filter floor list via
  listAvailableFloorsForAdapter(); fall back displayActiveFloorId to lobby if current
  floor is no longer in the available set
- OfficeScreen: pass selectedAdapterType to OfficeFloorNav; add guard in
  handleSelectFloor — bail back to lobby immediately when a runtime floor has no
  gateway URL configured, preventing the connect-hang limbo state

Authored-By: GSKNNFT

* hardening: drop unsafe-eval in production CSP; add TRUSTED_PROXY IP resolution

next.config.ts:
- unsafe-eval removed from production script-src (Next.js dev/HMR needs it, but
  production build does not; React and Three.js make no use of eval)
- connect-src intentionally kept broad with note: gateway URLs are user-configured
  at runtime, cannot be enumerated at build time

server/access-gate.js:
- Add resolveClientIp() helper: when TRUSTED_PROXY=1 env var is set, prefer the
  first value of X-Forwarded-For for rate-limiter keying (correct behavior behind
  nginx/Caddy/Vercel edge). Without the flag, remoteAddress is used (safe default
  for direct exposure — prevents X-Forwarded-For spoofing by untrusted clients).

Authored-By: GSKNNFT

* fix(security): remove upstream tokens from browser API; propagate abort to custom runtime

HIGH — /api/studio: strip gatewayPrivate and localGatewayDefaultsPrivate from GET and
PUT responses. Upstream tokens must not cross the browser API boundary. The Studio
proxy (server/gateway-proxy.js) already injects the server-side token into connect
frames when the browser sends an empty token, so the browser never needed raw tokens.
GatewayClient.ts and OfficeScreen.tsx updated to work from sanitized public settings only.

MEDIUM — /api/runtime/custom route: pass request.signal to the upstream fetch() call.
Client abort (e.g. hitting Stop) now cancels the upstream runtime request instead of
leaving it running after the browser fetch resolves.

Authored By: GSKNNFT

* fix(security): preserve stored token through empty-token UI state; handle non-JSON health responses

GatewayClient.ts — autosave effects no longer overwrite persisted gateway tokens with
empty strings. When the in-memory token is empty (proxy handles auth server-side),
the patch omits the token field (undefined) so mergeGatewaySettings/mergeGatewayProfiles
treats it as "leave unchanged". adapterProfiles updater also preserves the existing
stored token when the new token is empty, preventing floor-switch from erasing tokens.

runtime/custom/http.ts — requestCustomRuntime() checks the response Content-Type before
calling response.json(). Non-JSON responses (e.g. plain-text /health "OK") are returned
as-is instead of throwing a JSON parse error, making the custom/local/claw3d health
probe path reliable for runtimes that return plain text.

Authored By: GSKNNFT

* fix(bug): avoid overwriting stored tokens with an empty UI value

GatewayClient.ts:944 → token: "" || undefined = undefined → omitted from patch
mergeGatewayConnectionState: patch.token === undefined → patchedToken = undefined → nextToken = undefined || current?.token ?? "" = "abc123" ✓

scenarionn- user explicitly clears token (empty string patch):

mergeGatewayConnectionState: patchedToken = "" → nextToken = "" || current?.token ?? "" = falls back to existing stored token

Authored By: GSKNNFT

* fix ongoing findings issue

* test: update gateway connection persistence expectations

Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>

* fix: tighten ts.net hostname matching in doctor

Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>

* chore(release): prepare v0.1.4

Pin in-repo app version to 0.1.4 to match the planned GitHub release
tag, and document the convergence release in CHANGELOG.md with verified
0.1.3 and 0.1.4 entries covering runtime profiles, multi-floor offices,
remote messaging/handoffs, file uploads, security hardening, and the
claw3doctor diagnostics CLI.

Made-with: Cursor

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>
Co-authored-by: Elias Pfeffer <eliaspfeffer@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Greg Clark <greg.clark@gmail.com>
Co-authored-by: iamlukethedev <lucas.guilherme@smartwayslfl.com>

2026-04-23 18:29:20 -05:00

8.4 KiB

Raw Permalink Blame History

Multi-Agent Beta

This document explains the current multi-agent beta in Claw3D: what it does, how the two connection modes work, and how to connect a second office.

What This Beta Does

Claw3D can render a second office inside the same 3D scene so you can visualize agents from another machine.

Today the beta supports:

showing a second office in the same world;
displaying remote agents as read-only presence;
optionally sending a plain-text message to a remote agent;
keeping the remote side isolated from your local files and office controls.

This is a beta feature. It is designed for visibility and lightweight cross-office messaging, not full shared-state collaboration.

Mental Model

There are always two roles:

Local office: the Claw3D instance you are currently using;
Remote office: another Claw3D instance or another OpenClaw gateway you want to visualize.

The remote office can be connected in one of two ways:

Remote Claw3D presence endpoint.
Remote OpenClaw gateway.

Connection Modes

1. Remote Claw3D Presence Endpoint

Use this when the other machine is also running Claw3D.

How it works:

your local Claw3D server polls the remote Claw3D presence endpoint;
it also tries to load the remote office layout snapshot;
the local 3D scene renders the remote office as a read-only clone inside the same world.

Typical URL:

https://other-office.example.com/api/office/presence

This mode is best when you want the remote side to feel like another full Claw3D office.

2. Remote OpenClaw Gateway

Use this when the other machine only runs OpenClaw and does not run Claw3D.

How it works:

the browser connects directly to the remote gateway;
Claw3D derives a read-only presence snapshot from gateway data such as agents.list, status, and sessions.preview;
because there is no remote Claw3D layout endpoint, the second office uses a fallback office visualization.

Typical URL:

ws://remote-host:18789

or:

wss://remote-host.example.com

If you paste an http:// or https:// URL into gateway mode, Claw3D normalizes it to ws:// or wss:// before connecting.

This mode is best when you want remote agent visibility without requiring a second Claw3D deployment.

What You Can See

When the beta is enabled, you can:

see a second office in the same environment;
see remote agents appear in that office;
see remote agents move and change basic activity state;
click a remote agent and open a text-only messaging panel.

What You Cannot See

The remote office is intentionally limited.

You cannot:

inspect the remote machine filesystem;
browse the remote agent chat history in full;
control the remote office furniture or builder state;
take over the remote instance as if it were local.

The goal is cross-office visualization, not remote workstation access.

Remote Messaging

Remote messaging is currently a lightweight relay with two send modes.

What it does:

lets you send a plain-text note to a remote agent;
lets you choose direct or interval delivery in the remote chat panel;
is available from the remote agent chat panel;
is designed to avoid exposing remote files or tool output in the Claw3D UI.

direct is for one-off pings.

interval is for an ongoing coordination thread where you expect short periodic updates or checkpoints.

Current limitations:

remote replies are not mirrored back into the panel yet;
the panel currently shows your sent message plus delivery/system feedback;
this is not a shared transcript viewer.

Runtime Message And Handoff Layer

Under the hood, Claw3D now uses a shared runtime contract for:

agents.message
agents.handoff

OpenClaw, Hermes, Demo, and direct custom/local/claw3d runtime profiles can all target the same message/handoff seam. Provider-native adapters such as Anthropic or Claude Code are still a follow-up slice.

How To Connect

Prerequisites

Before enabling the second office, make sure:

your local Claw3D is already working with your local OpenClaw gateway;
you know which remote mode you want to use;
the remote machine is reachable from your machine or browser;
any required token, origin allowlist, or private-network access is already configured.

Setup Steps

Start your local Claw3D instance.
Open the office UI.
Open the office settings panel.
Turn on Show second office.
Choose the correct Source type.
Fill the matching connection fields.

Setup For `Remote Claw3D presence endpoint`

Use:

Source type: Remote Claw3D presence endpoint.
Presence URL: the remote /api/office/presence URL.
Optional token: only if that remote Claw3D endpoint is protected.

Example:

https://other-office.example.com/api/office/presence

Expected behavior:

the second office appears inside the world;
remote agents show up when the remote office has active presence;
if the remote layout snapshot is unavailable, Claw3D falls back to a default/fallback office rendering for the remote side.

Setup For `Remote OpenClaw gateway`

Use:

Source type: Remote OpenClaw gateway.
Gateway URL: the remote gateway WebSocket URL.
Shared gateway token: optional when the gateway already allows your Control UI origin and connection model.

Examples:

ws://remote-host:18789

wss://remote-host.example.com

Expected behavior:

the second office appears inside the world;
remote agents are derived from gateway presence data;
the office shell is a fallback visualization, not a true remote layout clone from another Claw3D instance.

Recommended Network Patterns

Same private network

Use a reachable private IP or local hostname for the remote Claw3D endpoint or OpenClaw gateway.

Tailscale

Tailscale is a good fit for this beta because it lets both sides connect over a private network without exposing services publicly.

Common patterns:

remote Claw3D endpoint over https://<machine>.ts.net/api/office/presence;
remote OpenClaw gateway over wss://<machine>.ts.net if you are proxying the gateway through HTTPS/WSS;
direct gateway over ws://<machine>:18789 when both devices can reach the service privately.

Disable Behavior

If you turn Show second office off:

the extra office should disappear from the 3D scene;
the path/outdoor connection should disappear;
remote office presence and layout hooks should stop driving the scene.

This lets you return to a single-office view.

Troubleshooting

No remote agents appear

Check:

the remote URL is correct;
the remote machine is actually reachable;
the remote service is running;
the selected Source type matches the service you are pointing at.

Presence endpoint works but the remote layout does not

That usually means the other machine has Claw3D presence available but not a layout snapshot yet. The beta should still render a fallback remote office.

Gateway mode connects but messaging fails

In gateway mode, the browser connects directly to the remote gateway. That means the remote gateway may still reject the connection based on origin policy or other gateway-side security rules.

If that happens, check:

the remote gateway URL;
whether the remote gateway allows your Control UI origin;
whether the remote gateway expects a token or device-auth flow you have not configured.

You can reach an HTTPS page but gateway mode still fails

Opening a web page in the browser does not automatically mean the OpenClaw gateway WebSocket is reachable.