On this page

On this page

Gateway

Health checks

Short guide to verify Gateway and channel health without guessing. It covers the CLI health checks, the HTTP check endpoints, the dedicated health command, and uptime monitoring.

Quick checks

  • openclaw status - local summary: gateway reachability/mode, update hint, linked channel auth age, sessions + recent activity.
  • openclaw status --all - full local diagnosis (read-only, color, safe to paste for debugging).
  • openclaw status --deep - asks the running gateway for a live check (health with probe:true), including per-account channel checks when supported.
  • openclaw status --usage - show model provider usage/quota snapshots.
  • openclaw health - asks the running gateway for its health snapshot (WS-only; no direct channel sockets from the CLI).
  • openclaw health --verbose (alias --debug) - forces a live health check and prints gateway connection details.
  • openclaw health --json - machine-readable health snapshot output.
  • Send /status as a standalone chat command in any channel to get a status reply without invoking the agent.
  • Logs: run openclaw logs --follow (or openclaw --profile <profile> logs --follow) and filter for web-heartbeat, web-reconnect, web-auto-reply, web-inbound.

For Discord and other chat providers, session rows are not socket liveness. openclaw sessions, Gateway sessions.list, and the agent sessions_list tool read stored conversation state. A provider can reconnect and show healthy channel status before any new session row is materialized. Use the channel status and health commands above for live connectivity checks.

Per-agent session counts and recent activity include only that agent's sessions, even when agents share a session store. Status counts each physical store once in its aggregate. The top-level health session summary represents the default agent, or the first configured agent when there is no default; it is not a fleet total. A running Gateway serves clean health and status session summaries from its resident session-row projection. Store hydration and exact dirty-row refreshes retain the existing read-only store fallback. These summaries wait for current selection metadata. Display preparation and background model catalog discovery can continue independently, so a large fleet's catalog warmup does not hold session counts and recent-activity summaries open. Startup prepares selection and database facts for the roster; full display rows for up to 100 recent unarchived sessions warm in the background while the remaining startup work continues. Connect does not wait for that page, and each client still selects and authorizes its current visible rows. Other display rows and optional transcript previews are prepared when a page or session recipient first requests them. Those rows stay resident for subsequent reads and refresh from committed changes. Unrequested rows do not repeat display enrichment when the startup model catalog becomes available. Cold pages share bounded enrichment batches. Each requested row becomes available when its own preparation finishes, with event-loop yields between slices; pages do not repeatedly select the roster while their accepted rows are still preparing.

Deep diagnostics

openclaw health --json reports modelRuntime.degraded: true when a large fleet's model preparation exceeds the startup budget. pendingAgents names the agents still preparing and stage identifies the current acquisition phase. The Gateway remains running and completed agents remain usable. Background preparation clears the degraded status when all runtimes are ready.

If a newer model-runtime refresh replaces preparation in progress, startup follows the admitted replacement through preparation and publication. Cancelling the retired acquisition does not stop the Gateway; genuine replacement failures still propagate. A config reload that retires the plugins behind that preparation follows the same bounded retry. If replacements outlast those retries, the agent's refusal asks for a Gateway restart; it does not require database repair.

Transcript projection repair runs after readiness in yielding worker-backed work. Pending GitHub publication recovery runs in the background and does not delay readiness or agent dispatch while Git or GitHub responds. Placement authority recovery still completes before dispatch opens; shutdown joins publication work already in progress, and failures retain their recorded effects for retry. An agent deferred by database inspection becomes usable after its session, credentials, and model preparation finish; rebuilding its transcript projections does not hold agent admission or another agent's preparation. Shutdown still joins accepted repair work, and repair failures are logged without withdrawing admission.

When agents.defaults.authInheritance.agentId names an owner whose PostgreSQL database is still preparing, admitted agents start with their own credentials. Inherited credentials become available automatically when the owner's admission completes. Failed owner inspections remain visible and do not use this pending fallback. The Gateway stays running during preparation. After initial recovery, admitted agents can chat while a different agent's store is still preparing. Each pending agent remains gated until its saved inputs have been replayed into the ordered session queue. Its inherited credentials and fresh-chat admission become available through the existing preparation completion signal.

Health and status collection groups fast session-store reads into short work slices, keeping busy background preparation from delaying every individual read. Slow reads finish their transaction before yielding to other Gateway work.

The health RPC includes live coreReady status, independent of cached health data and optional channel availability. The macOS app waits for this flag before reporting its Gateway ready. Older Gateways omit the flag; the app retains its successful-health-response compatibility behavior for those versions.

  • Creds on disk: ls -l ~/.openclaw/credentials/whatsapp/<accountId>/creds.json (mtime should be recent).
  • Session store: ls -l ~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite. Count and recent recipients are surfaced via status.
  • Relink flow: openclaw channels logout && openclaw channels login --verbose when status codes 409-515 or loggedOut appear in logs. The QR login flow auto-restarts once for status 515 after pairing.
  • Diagnostics are enabled by default (diagnostics.enabled: false disables them). Memory events record RSS/heap byte counts and threshold/growth pressure. Liveness warnings record event-loop delay/utilization, CPU-core ratio, and active/waiting/queued session counts when the process is running but saturated. Oversized-payload events record what was rejected/truncated/chunked plus sizes and limits, never message text, attachment contents, webhook bodies, raw request/response bodies, tokens, cookies, or secret values.
  • The same heartbeat drives the bounded stability recorder: openclaw gateway stability (or the diagnostics.stability Gateway RPC). Fatal Gateway exits, shutdown timeouts, and restart startup failures persist the latest snapshot under ~/.openclaw/logs/stability/. Inspect the newest bundle with openclaw gateway stability --bundle latest.
  • For bug reports, run openclaw gateway diagnostics export and attach the generated zip: a Markdown summary, the newest stability bundle, sanitized log metadata, sanitized Gateway status/health snapshots, and config shape. Chat text, webhook bodies, tool outputs, credentials, cookies, account/message identifiers, and secret values are omitted or redacted. See Diagnostics Export.

Health monitor config

  • channels.<provider>.healthMonitor.enabled: disable health-monitor restarts for a specific channel while leaving global monitoring enabled.
  • channels.<provider>.accounts.<accountId>.healthMonitor.enabled: multi-account override that wins over the channel-level setting.
  • These per-channel overrides apply to the channels that expose them today: Discord, Google Chat, iMessage, IRC, Microsoft Teams, Signal, Slack, Telegram, and WhatsApp.
  • A crashing channel is recovered by its own auto-restart backoff first (auto-restart attempt N/10 in the logs). The health monitor stays out of the way until that ladder ends with giving up after 10 restart attempts, then takes over as the last restart owner.
  • During slow startup, health-monitor recovery waits for the same channel-startup barrier as initial startup; expiry of the 60-second monitor grace does not launch providers early. Stopping a waiting account cancels that start.

Inbound ingress health

Channel connectivity and inbound admission are separate failure domains. A channel can hold a healthy transport connection — sending replies normally — while its durable ingress queue is unavailable, so not a single inbound message is admitted.

  • When a channel cannot open its durable ingress queue, its start fails and the gateway records the account as unable to receive. openclaw channels status reports Channel cannot admit inbound events; its durable ingress queue is unavailable. Outbound may still work.
  • Such an account is unhealthy regardless of transport state, and readiness reports it as failing. Previously it reported health: healthy and the health monitor never touched it.
  • Recovery stays automatic. The ingress verdict describes the account's last start attempt and is cleared by the next one, so the ordinary restart path is also how a transient queue-open failure recovers. Those restarts log as health-monitor: restarting (reason: ingress-unavailable) instead of the generic stuck.
  • If the restarts keep repeating, the cause is not transient. Check the logged ingress failure: a plugin denied the openChannelIngressQueue capability, for example, needs operator action rather than another restart.
  • Channels that never report ingress state are unaffected: absence means "no signal", never "broken". There is no traffic-staleness heuristic, so a genuinely quiet channel is never marked unhealthy for having received nothing.

HTTP checks

The Gateway exposes three unauthenticated GET/HEAD check pairs:

Endpoints Meaning Use
/health, /healthz The HTTP server is live. Process liveness and restart decisions.
/startup, /startupz Startup sidecars and agent database inspection/preparation have settled, and the Gateway is not draining. Channel health is not consulted. Startup-phase monitoring and update verification.
/ready, /readyz Startup sidecars and fresh-chat admission are ready, the Gateway is not draining, required agent databases have completed admission without failures, and configured channel accounts pass deep readiness checks. Traffic admission and operator monitoring.

All three check pairs share the Gateway HTTP transport; there is no separate always-on check listener. Connection refusal is expected before the initial bind and after transport closure. An in-process restart can close and reopen the listener while the same PID stays alive; requests racing closure can also reset.

While the listener is available, the normal Control UI document and static app assets remain available during core startup or draining. A fresh browser loads the real application shell and shows its connection state while waiting for the Gateway. Existing conversations keep their mounted route, transcript and draft. Authentication failures still use the normal sign-in flow.

When the app assets are unavailable, a self-contained 503 availability page retries automatically with bounded request timeouts and backoff. It does not require the app bundle or a WebSocket connection. APIs and authenticated dynamic routes keep their normal admission and authorization checks. Optional startup sidecars do not block the normal UI once core request preparation is ready.

An HTTP document is not a completed WebSocket handshake: the loaded app still owns connection recovery and reload-required states. If the document loads but the app cannot mount after three recovery navigations, its fallback offers manual reload and wait actions. A fresh browser with no cached app cannot receive a Gateway page while its listener is fully stopped. An existing ingress can provide a retryable HTML error page for that interval; keep API and WebSocket handling intact, and do not label an unknown connection failure an update.

Supervisors should combine HTTP checks with process and exit information, allow bounded startup/restart grace, and honor the process owner's shutdown budget. One refusal or reset during these transitions does not prove a wedged Gateway. Choose deployment-specific budgets and investigate persistent unavailability after they expire; a living PID does not justify indefinite grace. Use /readyz and normal admission for traffic; neither /healthz 200 nor /startupz 200 certifies readiness.

The HTTP listener opens before the initial session projection is complete. During that build, /healthz can return 200, while /startupz and /readyz remain pending. Other HTTP requests and WebSocket upgrades wait for the complete roster metadata before authentication and routing; they never see a partial roster. Discovery and remote-skill listeners run alongside projection preparation. The initial fleet-wide skills scan runs in tracked maintenance after readiness; on-demand skill reads still prepare the requested workspace. Readiness also waits for the reply and WebSocket connection modules to load, so immediate connections and first turns reuse the prepared runtime. Update verification still waits for startup completion before checking readiness. The later http server listening startup summary is not the actual bind timestamp.

These probe paths stay at the server root even when gateway.controlUi.basePath is set. A UI base path such as /health does not turn its child routes into probes: they still wait for initialization and retain their normal authentication.

/startupz returns 503 with status: "starting" while startup sidecars or agent database inspection/preparation are pending, 503 with status: "draining" during drain, and 200 with status: "started" otherwise. After sidecars settle, pending inspection for any agent, including an optional agent, reports pendingReason: "agent-database-inspection". This lets openclaw update candidate verification wait within its startup budget before checking readiness. A settled inspection failure does not keep startup pending; readiness and per-agent admission report the failure.

Startup admits one resume input for each eligible session marked at graceful shutdown or retaining a persisted unfinished run from the immediately preceding Gateway lifetime with no live owner. Older orphaned runs are settled without resuming them. Captured queued user messages then use ordinary admission. Fresh chat does not wait behind a per-agent recovery predecessor. Database and model/plugin preparation still gate their own consumers.

A default or system agent database with pending startup inspection/preparation or a confirmed admission failure keeps readiness false, with failing: ["agent-database:<id>"] and the exact admission reason and repair hint in agentDatabases. Pending startup inspection is reported there with code agent-database-inspection-pending. The HTTP listener, Control UI, and admitted agents remain available while readiness is false; pending optional agents do not hold readiness. Readiness does not wait for transcript maintenance or resumed agent work to finish. Agent-scoped sessions.list, plus sessions.resolve, sessions.describe, and models.list, wait up to 20 seconds for their selected agents to finish startup preparation, then recheck request authorization before reading. Requests targeting healthy agents proceed independently. Unscoped session lists continue returning admitted agents while other stores prepare. Writes still require completed admission. RPC requests refused for pending inspection return UNAVAILABLE with retryable: true and retryAfterMs: 250; callers can retry the same request after that delay. Inspection failures and ownership mismatches remain non-retryable. A refused optional agent can remain isolated while healthy agents serve requests. Gateway ready announcements use the same readiness decision.

Agents whose admission is closed by deletion appear separately in agentDeletions with code agent-deletion. Pending or completed deletion does not fail ready or coreReady, including after the agent is removed from configuration. Ordinary requests remain refused; pending cleanup follows the deletion journal.

If a newer model-runtime publication supersedes deferred agent preparation, admission waits for that publication once, then retries preparation against current configuration and secrets. Recovery uses the existing four-attempt startup publication budget without reopening or migrating the database again. A failed publication or exhausted budget keeps the agent unavailable and records its reason and repair hint in readiness.

Temporary SQLite worker-capacity refusals during an agent's database open share the existing four-attempt startup lock-retry budget. Releasing capacity can let that agent recover without restarting the Gateway. Persistent pressure or a non-retryable database failure still leaves it unavailable; opening retries do not replay migration or publication.

If Node.js is upgraded or removed while the Gateway is running, its retained runtime may stop launching child workers even though the Gateway process is still alive. A confirmed missing executable or runtime loader failure marks childRuntime.available as false in health/status and makes /readyz return 503 with failing: ["child-runtime"]. Worker requests explain which runtime failed and ask you to restart the Gateway. The diagnosis is logged once and further worker launches fail immediately; ordinary worker crashes do not mark the runtime unavailable. A health or readiness check that finds the executable missing publishes the same diagnosis, even when resident workers still serve reads.

For a service-managed Gateway, the lifecycle owner checks the effective service command's Node with a bounded --version probe. On macOS, the loaded launchd job must also match the service definition on disk. If the probe succeeds, the Gateway requests one graceful restart: new work is fenced, active work drains, and the supervisor starts a fresh process. An in-process restart cannot repair a missing runtime and is never used for this recovery.

The Gateway stays up in report-only mode if the service Node is also unusable, the command or recent restart history cannot be verified, or runtime recovery was the most recent shutdown within five minutes. Unmanaged foreground and embedded Gateways, external supervisors without a verifiable native service command, and OPENCLAW_NO_RESPAWN=1 also remain report-only. Repair the service runtime, then use openclaw gateway restart (or your process owner's restart).

The existing childRuntime.available: false health field and child-runtime readiness reason remain visible. The runtime log records restart-requested or report-only with the decision reason. Repeated worker failures do not request another restart. If a prepared supervisor handoff later fails, the old Gateway completes shutdown instead of attempting an in-process restart; inspect the logs and restart through the service owner.

A broken Telegram or other channel account can also make /readyz return 503 while /startupz remains started. Neither check replaces the other: startup completion alone does not certify agent or channel availability.

Detailed readiness responses also include coreReady, which excludes channel availability while retaining startup, fresh-chat admission, drain, database admission, child-runtime, and plugin-replacement checks. Updates can complete with channel warnings when the core is ready. HTTP status and ready still include channel health, including for ordinary service start and traffic admission.

Disabled, unconfigured, and ordinarily unlinked accounts do not fail channel readiness. An enabled, configured account with a recorded terminal disconnect, blocked lifecycle, or unavailable ingress still fails its health check even if it becomes unlinked. For example, a terminal WhatsApp logout makes /readyz return 503 with failing: ["whatsapp"] in detailed responses. Terminal disconnects and blocked accounts require operator action; the health monitor does not restart them.

Unexpected channel exits use automatic restart backoff. Intentional stops cancel a pending retry; failed restart waits leave a stopped account with restartPending: false and the failure recorded in lastError.

A previously connected channel can recover its transport without failing the whole Gateway. During the existing two-minute reconnect grace, /readyz stays 200 when all other readiness checks pass, and detailed responses include degraded: ["discord"] (or the affected channel IDs). Channel status retains connected: false, lifecycle: "recovering", healthState: "reconnect-grace", and the disconnect timestamp; transport logs record the reconnect attempts. The window starts at the first disconnect and retries do not renew it. Recovery clears degradation. An unrecovered channel fails readiness after the window, including while an agent run is active. Blocked or terminal failures bypass reconnect grace. Initial connection still uses the existing startup readiness rules; an account that has never connected is not a reconnecting account.

Remote unauthenticated startup responses contain only ok and status. Local-direct and authenticated callers also receive version, uptimeMs, and pendingReason while startup is pending. Readiness details follow the same local-or-authenticated gate because they can name failing subsystems.

Proxies listed in gateway.trustedProxies can request aggregate readiness and startup checks without forwarded client headers. Diagnostic details still require client attribution and authorization. Proxy attribution remains required for non-probe routes; untrusted proxy sources do not receive this exception.

Shared-state integrity failure

A terminal shared-state admission failure immediately makes /ready and /readyz return 503, including failures discovered by a SQLite worker after startup. Detailed responses include failing: ["state-database"] and stateDatabase.reason with the recorded refusal. This bypasses cached channel health; checks read the admission owner's recorded result without querying SQLite.

/healthz still reports HTTP liveness. Supervisors that need to detect a Gateway that is running but cannot admit work must monitor /readyz.

Plugin replacement recovery

During plugin replacement or recovery, /readyz returns 503. Detailed responses include failing: ["plugin-reload"] and a pluginReload object with the affected pluginIds, the current phase (reloading, recovering, or failed), and any recovery deadlineAtMs and actionable reason. These owner-reported facts bypass the channel readiness cache, so a failed replacement cannot appear as only a generic channel outage or stale healthy result.

The health monitor does not spend channel restart attempts while replacement holds channel admission paused. After successful rollback, the previous plugin configuration restarts its channels and ordinary readiness checks resume. If automatic recovery reaches its deadline, phase: "failed" retains the failure reason and next action. Admission pauses are released, allowing the monitor to restart callable channels; a plugin whose admitted work or cleanup still owns resources requires the reported repair or retry before it can restart. See Config hot reload for the recovery contract.

CPU pressure and event-loop delay

Detailed readiness can include the latest completed eventLoop diagnostic snapshot. The sampler owns observation windows; health reads do not reset a pending measurement. During active work, degradation, or diagnostic sample export, native event-loop delay sampling runs at 20 ms resolution and the Gateway publishes completed observations once per second. A healthy idle Gateway publishes every ten seconds and pauses the native histogram. Idle windows still track blocking callbacks and overdue sampling deadlines; delayP99Ms is zero while the histogram is paused. Work becoming active restores full sampling at the next idle deadline, within ten seconds. A newly interested exporter advances the pending sample to the current window's one-second boundary, or schedules it immediately if that boundary has already elapsed, without resetting the window. Long blocking callbacks remain part of the observation even when they delay the publisher. No snapshot is available until the first window completes. Its cpuCoreRatio measures user and system CPU time across the whole Gateway process, including worker and native threads, divided by elapsed wall time. The unit is core equivalents: 1 means one CPU core fully occupied over the interval, and parallel work can produce values above 1. It is not a percentage of the host's total CPU capacity.

The health RPC also reads the latest completed sample when returning a cached summary or publishing a newly collected one. Slow channel checks do not freeze its CPU and delay readings. If the sampler resets, health responses omit eventLoop until a new window completes instead of reviving a cached sample.

The optional cpuBreakdown separates independent native counters:

  • hostUtilization is the busy fraction of the host CPU time reported by the aggregate /proc/stat counters on Linux and os.cpus() elsewhere, from 0 to 1 across hostCpuCount online logical CPUs. Linux counts user, nice, system, idle, and hard IRQ time, matching os.cpus(); iowait, softirq, and steal are excluded, and guest time is not counted twice. This includes other processes and is not the Gateway's CPU quota or container allowance.
  • mainThreadCoreRatio measures the Gateway main thread with process.threadCpuUsage(), not event-loop utilization.
  • workerCoreRatio sums Worker.cpuUsage() counters for workers owned by the task pools and SQLite broker. It does not include subprocesses, remote workers, or workers created outside those owners.
  • otherThreadsCoreRatio is an estimated residual: process CPU minus the measured main and tracked worker CPU, clamped at zero. It includes untracked workers and native threads, not a measured worker category.

Thread values use the same core-equivalent unit as cpuCoreRatio. Worker reads are asynchronous and must complete within 100 ms of each sampling boundary; these are not atomic cross-thread measurements. The residual can vary with measurement skew. A timed-out request never delays the event-loop sample, and at most one native request is outstanding per tracked worker, even across monitor resets. Startup, worker creation/exit, host CPU topology changes, counter resets, and collection failures require fresh baselines before publishing the affected rates. Missing fields mean unavailable, not zero.

The main-thread and host counters are collected independently on Node and Bun. Tracked worker CPU and the residual are omitted on Bun: its worker API can report zero when native counter collection fails. Unsupported or failed native APIs on any platform leave the corresponding field absent.

The Control UI's Gateway CPU box reads the same sampler through system.info.eventLoop. Its headline shows process core-equivalents: 2.58 cores means about two and a half fully occupied logical CPUs. Build and tool subprocesses are excluded. The separate Host reading includes all host processes and uses 100% for all reported logical CPUs. In the detail overlay, process and thread percentages use 100% for one fully occupied core. Values show a dash until their first complete measurement, or while unavailable.

Event-loop delay and utilization describe the main thread separately. A cpu degradation reason reports process CPU pressure with delay co-evidence; it does not identify the thread consuming CPU or prove a main-thread hang. Inspect the delay measurements alongside CPU pressure. The eventLoop diagnostic does not change the readiness result by itself.

Instrumented synchronous work lasting at least 250 ms and unattributed event-loop gaps longer than one second produce a main-thread stall log with a code-owned task label where available. Native launches through the process and command runners and PTY helper have fixed task labels; these measure the synchronous launch, not the child's running time. During active event-loop sampling on Node, Promise continuations retain their inherited code-owned task labels; their synchronous execution is measured without charging time awaiting I/O. On Node 26, a bounded CPU flight recorder writes one additional bounded main-thread stall profile line with a stall interval's top inclusive frames, sample count, and truncation status. Several stalls can finish before reporting; only the longest interval is profiled. It samples every 25 ms and rotates five-second windows without stopping the sampler. At most two windows are retained; no profile files are written. Symbols and locations use the same redaction policy as on-demand diagnostic profiles. Inclusive counts overlap when a sampled stack contains several listed frames; they are not independent durations. The deepest stacks preserve the sampled leaf and nearest available OpenClaw caller, including across dependency frames. [...] marks omitted intermediate frames. If a continuation has no recorded OpenClaw frame, its captured task appears as [async task:...], distinct from a sampled call frame. Unnamed work remains unattributed. Lower-ranked entries are dropped first to keep each report within 2 KiB; reportTruncated=true identifies this log-size trimming.

On Node 24, the observer instead captures up to three interrupted JavaScript stacks, 250 ms apart, after 500 ms without a heartbeat. It opens no debugger listener and starts no CPU profiler. Each stack contains at most 16 frames, with bounded symbol/location lengths; at most three stack payloads can wait for the main thread. Reports use sampleMode=interrupt, truncated=true, and null self times: these are point observations, not sampled CPU durations. Source maps and user-defined stack formatting hooks are not invoked. Native calls may defer inspector interrupts until unrelated JavaScript resumes; captures dispatched more than 100 ms late are discarded. Such calls can still produce kernel-only evidence.

The recorder skips Bun and processes already started with conflicting profiling, tracing, or debugger options. Callback labels still work on those runtimes. Native blocking work can remain unsymbolized, and an unusually long stall can exhaust the sample cap; truncated=true identifies incomplete evidence. Recorder failures are logged once and disable recording for that monitor's lifetime.

Linux kernel observations remain available when CPU recording is unsupported, conflicting, or fails. These reports use cpuProfile=unavailable instead of CPU sample counts and intervals; they retain the stall bounds and kernel evidence.

samples=0 means neither retained profile contains a timestamped sample in that stall interval. It does not prove the entire process stopped or identify the blocking operation. For example, Linux libuv blocks the sampling signal on the calling thread around fork(), including allocator fork hooks. The synchronous task labels remain available during such sampling gaps.

On Linux, the recorder also watches a shared main-thread heartbeat from a worker. After one second without a heartbeat it asynchronously reads the main thread's /proc/self/task/<pid> state, wait channel, syscall number, kernel stack (when permitted), I/O counters, and scheduler counters. It takes at most three snapshots, one second apart, while the stall is still ongoing. The kernel field on the same profile line contains only snapshots inside the reported stall interval. No syscall arguments, memory addresses, or user data are recorded. Other platforms skip procfs sampling; no configuration is needed.

state=D with folio_wait or filemap_fault in wchan/stack points to page or file-backed waits; compact_zone or try_to_compact_pages points to compaction. Writeback/fsync symbols and the architecture-specific syscall number distinguish write waits; futex symbols identify a lock or Atomics.wait. state=R means runnable, including currently running: compare sched (CPU runtime and run-queue wait, in nanoseconds) across snapshots before concluding scheduling starvation. These observations narrow the cause; they do not prove why a page or lock was needed.

Snapshot at is milliseconds from the stall start. Counter arrays follow procfs order: io is rchar, wchar, syscr, syscw, read_bytes, write_bytes, cancelled_write_bytes; ctxt is voluntary/nonvoluntary switches from process status; rssKb is VmRSS. psi contains system-wide I/O, memory, and CPU pressure, each as some/full cumulative microseconds. Permission errors appear as ?EACCES or ?EPERM; unavailable counter arrays are empty. Snapshots are best-effort, non-atomic reads: delayed observer scheduling or I/O may miss a stall entirely. Counter and stack fields may be omitted to keep kernel evidence within 1,400 bytes and the complete profile within 2 KiB; state, wait channel, and syscall remain.

Uptime monitoring

External uptime monitoring services should use the dedicated /health endpoint, not /v1/chat/completions.

  • DO use: GET /health - instant response, no session created, no LLM call, returns {"ok":true,"status":"live"}
  • DON'T use: /v1/chat/completions for health checks - each request creates a full agent session with skill snapshot, context assembly, and LLM calls

When no x-openclaw-session-key header or user field is provided, /v1/chat/completions generates a new random session for each request. Monitoring services that ping every 15 minutes create ~96 sessions/day, each consuming 4-22KB. Over time this causes session store bloat and can lead to context window overflow.

Monitoring service setup examples

  • BetterStack: Set health check URL to https://<your-gateway-host>:<port>/health
  • UptimeRobot: Add a new HTTP monitor with URL https://<your-gateway-host>:<port>/health
  • Generic: Any HTTP GET to /health returns 200 with {"ok":true,"status":"live"} while the gateway's HTTP server is live

When something fails

  • logged out or status 409-515 -> relink with openclaw channels logout then openclaw channels login.
  • Gateway unreachable -> start it: openclaw gateway --port 18789 (use --force if the port is busy).
  • No inbound messages -> confirm linked phone is online and the sender is allowed (channels.whatsapp.allowFrom); for group chats, check that allowlist + mention rules match (channels.whatsapp.groups, agents.entries.*.groupChat.mentionPatterns).

Dedicated "health" command

openclaw health asks the running gateway for its health snapshot (no direct channel sockets from the CLI). By default it returns a fresh cached gateway snapshot and the gateway refreshes that cache in the background; --verbose forces a live check instead. Connections and cached health reads share a one-minute background refresh cadence, so repeated diagnostic connections do not each rebuild the health snapshot. Explicit live checks and refreshes for missing or stale health still run immediately. The passive WebSocket health event sends a snapshot only when health changes. Clients advertising health-summary receive it without agent heartbeat configuration and session activity summaries; request the health RPC for those details. Other clients retain the full event payload. Refresh timestamps, check duration, session activity summaries, channel traffic timestamps, busy/run counters, run start times, auth age, and event-loop sample readings alone do not trigger an event. Event-loop degradation triggers an event only after the sampler observes it continuously for at least one minute; recovery and channel lifecycle, error, and other health status changes publish on the next refresh. Full snapshots still include current diagnostic samples, even when those samples do not independently trigger an event. Use tick for connection liveness and the health RPC for current diagnostic readings; hello snapshots retain their health field. Snapshots describe loaded and configured channels. Stored credentials alone do not activate a channel or add it to Gateway health; use channel setup to enable it. The command reports linked creds/auth age when available, per-channel check summaries, session-store summary, and check duration. Live checks use bounded account concurrency and a Gateway-owned deadline, so one slow account returns a structured timeout while completed sibling results remain available. The command exits non-zero if the gateway is unreachable or the Gateway call itself times out.

Cron heartbeat summaries reuse the running scheduler's last successful read until jobs or configuration change. Failed reads are retried on the next refresh. Before the scheduler loads, health reads the selected cron store directly.

Queue warnings

A successful health RPC reports top-level ok: true. That value means the Gateway produced the snapshot; it does not mean every delivery queue is clear. Check deliveryQueues.ingressPressure for durable inbound lanes that may be blocking later events. The field is omitted when no pressured lanes are found.

Ingress pressure uses conservative built-in diagnostic thresholds, not authoritative retry or claim policy for any plugin. A durable lane appears only when an active pending or claimed row has either reached at least eight attempts and has a recorded delivery error, or a claimed row has not refreshed its claim for 30 minutes. Ordinary retries 1-7 are absent. Claim-recovery increments without a recorded error are also absent, while live claims stay absent because their claim timestamp is refreshed. Rows without a durable lane key are omitted because they cannot prove that later events are blocked; runtime persists a derived lane after a real derived-lane retry.

Each result is grouped by channel account and reports pressured lane, pending, claimed, and blocked counts plus the oldest affected receive time. All active rows in a pressured lane contribute to those counts. The snapshot never includes lane IDs, event IDs, payloads, claim owners or tokens, recorded errors, or session and target identifiers.

Options:

  • --json: machine-readable JSON output
  • --timeout <ms>: override the default 10s Gateway connection timeout; it does not widen the Gateway's internal live-check deadline
  • --verbose: force a live check and print gateway connection details
  • --debug: alias for --verbose

The health snapshot includes: ok (boolean), ts (timestamp), durationMs (check time), per-channel status, agent availability, session-store summary, and optional delivery-queue warnings.

Was this useful?