Logging

OpenClaw has two main log surfaces:

• File logs (JSON lines) written by the Gateway.
• Console output in the terminal running the Gateway.

The Control UI Logs tab tails the gateway file log. This page explains where logs live, how to read them, and how to configure log levels and formats.

Where logs live

By default, the Gateway writes a rolling log file per day. The default profile keeps the historical path:

/tmp/openclaw/openclaw-YYYY-MM-DD.log

Named profiles use a profile-qualified filename in the same directory:

/tmp/openclaw/openclaw-<profile>-YYYY-MM-DD.log

The filename profile segment is lowercase and limited to letters, numbers, and dashes. Simple lowercase names stay readable, so the --dev shorthand writes openclaw-dev-YYYY-MM-DD.log. Case, underscores, and literal dashes use a reversible dash escape so distinct profile names never share a log file. Oversized values set directly through the environment use a bounded hash suffix to stay within filesystem filename limits. An explicit logging.file overrides these defaults.

The date uses the gateway host's local timezone. When /tmp/openclaw is unsafe or unavailable (and always on Windows), OpenClaw uses a user-scoped openclaw-<uid> directory under the OS temp dir instead. Dated log files are pruned after 24 hours.

Each file rotates when the next write would exceed logging.maxFileBytes (default: 100 MB). OpenClaw keeps up to five numbered archives beside the active file, such as openclaw-YYYY-MM-DD.1.log or openclaw-dev-YYYY-MM-DD.1.log, and keeps writing to a fresh active log instead of suppressing diagnostics.

You can override the path in ~/.openclaw/openclaw.json:

{  "logging": {    "file": "/path/to/openclaw.log"  }}

How to read logs

CLI: live tail (recommended)

Tail the gateway log file via RPC:

openclaw logs --followopenclaw --dev logs --followopenclaw --profile work logs --follow

The root profile selector resolves the same profile-specific file used by the Gateway, including CLI fallback reads when local RPC is unavailable.

Options:

Output modes:

• TTY sessions: pretty, colorized, structured log lines.
• Non-TTY sessions: plain text.

When you pass an explicit --url, the CLI does not auto-apply config or environment credentials; include --token yourself, or the call fails with gateway url override requires explicit credentials.

In JSON mode, the CLI emits type-tagged objects:

• meta: stream metadata (file, source, sourceKind, service, cursor, size)
• log: parsed log entry
• notice: truncation / rotation hints
• raw: unparsed log line
• error: gateway connection failures (written to stderr)

If the implicit local loopback Gateway asks for pairing, closes during connect, or times out before logs.tail answers, openclaw logs falls back to the configured Gateway file log automatically. Explicit --url targets do not use this fallback. openclaw logs --follow is stricter: on Linux it uses the active user-systemd Gateway journal by PID when available, and otherwise retries the live Gateway with backoff instead of following a potentially stale side-by-side file.

If the Gateway is unreachable, the CLI prints a short hint to run:

openclaw doctor

Control UI (web)

The Control UI's Logs tab tails the same file using logs.tail. See Control UI for how to open it.

The tab retains the latest 2,000 lines and renders a window around the current scroll position. Filter and Export cover the entire retained history; browser Find and native text selection cover the rendered window. Use Filter to find an older retained line, or Export to copy all matching lines.

Filter ignores case and matches canonically equivalent composed and decomposed Unicode text. This normalization applies only to comparisons; displayed and exported log lines retain their original text.

Scroll to the bottom, or focus the log stream and press End, to reach the newest line. Turn on Auto-follow to keep following new lines as they arrive.

Channel-only logs

To filter channel activity (WhatsApp/Telegram/etc), use:

openclaw channels logs --channel whatsapp

--channel defaults to all; --lines <n> (default 200) and --json are also available.

Log formats

File logs (JSONL)

Each line in the log file is a JSON object. The CLI and Control UI parse these entries to render structured output (time, level, subsystem, message).

File-log JSONL records also include machine-filterable top-level fields when available:

• hostname: gateway host name.
• message: flattened log message text for full-text search.
• agent_id: active agent id when the log call carries agent context.
• session_id: active session id/key when the log call carries session context.
• channel: active channel when the log call carries channel context.

OpenClaw preserves the original structured log arguments alongside these fields so existing parsers that read numbered tslog argument keys keep working.

Talk, realtime voice, and managed-room activity emits bounded lifecycle log records through this same file-log pipeline. These records include event type, mode, transport, provider, and size/timing measurements when available, but omit transcript text, audio payloads, turn ids, call ids, and provider item ids.

Console output

Console logs are TTY-aware and formatted for readability:

• Subsystem prefixes (e.g. gateway/channels/whatsapp)
• Level coloring (info/warn/error)
• Optional compact or JSON mode
• Structured fields on warn, error, and fatal records, appended as one compact key=value ... tail (nested values as JSON, capped at 2 KiB) so plain-text sinks such as journald keep the diagnostics; info and lower console lines stay message-only, and the file log always carries the full record. These fields go through the same redaction as the json console style before they are flattened, so a sensitive key such as apiToken is masked by name and the length cap can only clip text that is already masked

Console formatting is controlled by logging.consoleStyle.

Gateway startup notices for OPENCLAW_RAW_STREAM and OPENCLAW_RAW_STREAM_PATH identify configured flags without printing their values. Successful LSP startup, native web-search activation, and web-search provider auto-detection stay quiet. Video generation retains the first skipped-candidate warning and all returned attempts without repeating every skip at debug level.

SQLite worker diagnostics use stderr. After the final backend closes normally, the worker flushes queued file logs and gives pending console output up to five seconds to drain before acknowledging close. Source-loaded backends and compiled workers share the same file-log queue. Forced worker termination can still discard pending diagnostics.

Gateway WebSocket logs

openclaw gateway also has WebSocket protocol logging for RPC traffic:

• normal mode: only interesting results (errors, parse errors, slow calls)
• --verbose: all request/response traffic
• --ws-log auto|compact|full: pick the verbose rendering style
• --compact: alias for --ws-log compact

Examples:

openclaw gatewayopenclaw gateway --verbose --ws-log compactopenclaw gateway --verbose --ws-log full

Steering and input cancellation

Ordinary channel reply routing logs steering rejected; applying follow-up policy at warning level when steering falls back. The fixed reason code distinguishes an unavailable owner or injection path, a pending source operation, changed authority, terminal-reply state, and runtime rejection. Records include the channel, opaque session ID, and input or active run IDs when supplied. disposition distinguishes a follow-up policy decision, confirmed or rejected queue admission, and a known queue-cap rejection; it does not confirm delivery. These records omit message text, attachments, session keys, and raw exceptions.

When retained reply-delivery state prevents steering, the Gateway logs chat steering rejected; falling back to follow-up dispatch. Its structured fields distinguish the incoming input's runId from activeRunId and record the session, active source turn, recovery claim, and exact reason:

• terminal-pending or delivered-terminal: a final-reply receipt is present.
• unresolved-terminal-tool or delivery-ambiguous: a final-reply delivery is unresolved.
• already-delivered: the active source turn is recorded as completed.
• unknown-source-with-terminal-history: the active source is unknown and completed source turns are retained.
• stale-claim: the recovery claim does not authorize the active source turn.
• session-entry-unavailable: the Gateway could not read the current session state.

These checks also run during ordinary conversations; the warning does not mean a Gateway restart is in progress. Rejection falls back to follow-up dispatch; it does not itself cancel the input.

chat pending input aborted records an accepted input that was aborted before consumption, including inputs waiting in the follow-up queue. The message includes the cause and whether the saved input became cancelled or interrupted. Structured fields include its run, session, and agent IDs. Causes distinguish rpc, stop, timeout, restart, archive, delete, authority-revoked, and superseded; unclassified aborts use aborted. These records omit message text and attachments.

Configuring logging

All logging configuration lives under logging in ~/.openclaw/openclaw.json.

{  "logging": {    "level": "info",    "file": "/path/to/openclaw.log",    "consoleLevel": "info",    "consoleStyle": "pretty",    "redactPatterns": ["sk-.*"]  }}

Log levels

Levels: silent, fatal, error, warn, info, debug, trace.

• logging.level: file logs (JSONL) level (default: info).
• logging.consoleLevel: console verbosity level.

You can override both via the OPENCLAW_LOG_LEVEL environment variable (e.g. OPENCLAW_LOG_LEVEL=debug). The env var takes precedence over the config file, so you can raise verbosity for a single run without editing openclaw.json. You can also pass the global CLI option --log-level <level> (for example, openclaw --log-level debug gateway run), which overrides the environment variable for that command.

--verbose only affects console output and WS log verbosity; it does not change file log levels.

Expected shutdown cancellation

During Gateway shutdown, automatic provider catalog discovery can be canceled, and cron startup or session recovery can be refused admission. These expected lifecycle outcomes do not emit failure warnings or errors. Provider, storage, and cleanup failures still use their existing error reporting, including failures that wrap a cancellation error.

Provider request failures

Model transport failures emit Model request failed at warning level with the provider, model, error name, available status, code, type, cause name and code, and request and response IDs. The provider's error message and response body are not logged, because they can echo prompts, tool results, or masked credentials. Use the status, code, and type to tell an unknown model (for example 404 model_not_found) from a bad key (401 invalid_api_key). Generated response content and request payloads are not included. Log exports retain their existing redaction policy.

Embedded attempts also log Assistant attempt failed, including the saved error fields, before choosing recovery. Request validation and content-policy failures stop unchanged model retries and fallback. Context limits use the existing compaction recovery. Retired transcript input is terminal: changing models cannot restore its admission, and the original rejection remains visible in diagnostics.

Anthropic-compatible HTTP failures preserve the HTTP status separately from a bounded, redacted response body. JSON error bodies are parsed before diagnostic redaction and preview truncation, so a long proxy error does not lose its status or upstream rejection reason merely because the console preview is short. Oversized or malformed bodies can still be omitted by the diagnostic redactor.

Chat displays recognized request-limit facts, including the allowed and actual number of cache_control blocks, in both live failures and saved history. Raw proxy metadata stays in redacted diagnostics rather than the chat message.

Saved failed replies also distinguish rate limits, authentication failures, provider HTTP errors, and network interruptions. Worker inference preserves bounded, redacted error details for classification, including when a large partial response cannot fit in the transcript. Unrecognized errors still use generic chat copy; inspect the Gateway logs and stored error for diagnosis.

Ordinary OpenAI Responses failures log one provider-error warning with available code, type, and provider requestId metadata. Correlation with the originating model call uses clientRequestId when the provider supplies its own request ID. Missing fields and provider prose, request bodies, and response bodies are omitted. Same-model retry scheduling is logged at debug; exhausted retries produce an explicit error. Generic secret-egress request audit lines, including refusals, stay at debug. Structured secret.egress events still count every outcome, and certificate failures retain their actionable warning.

When reply dispatch consumes a thrown error after a partial reply or an adopted turn, its terminal message.dispatch.completed and message.processed diagnostics retain the failure outcome, identifiers, and timing. The extra free-form cause is omitted at the default info file level and less verbose levels, including from diagnostic subscribers and telemetry exports.

To collect that cause in the events' error field, enable diagnostics.enabled and set the effective file logging.level to debug or trace. The existing OPENCLAW_LOG_LEVEL and --log-level overrides apply; logging.consoleLevel or --verbose alone do not enable it. Chat notices stay generic.

The cause uses mandatory built-in credential redaction plus logging.redactPatterns, then is capped at 2,048 UTF-16 code units without splitting a surrogate pair. Redaction is not a guarantee that arbitrary private text is removed: debug causes can still contain provider or user content. Treat debug logs and telemetry as sensitive, and return to info after diagnosis.

Responses output-identity conflicts record the event type, output position, expected and observed item types, and whether a tool call completed, without recording item IDs or response content. OpenClaw uses the existing bounded session retry policy when the failing response produced no visible text or completed tool calls and its request enabled only client-executed function tools. Continuation keeps earlier tool results, so those completed actions are not replayed. Conflicts after output or with provider-hosted tools remain terminal; inspect earlier results before continuing. Previously emitted text remains terminal even if a later snapshot clears it. Automatic recovery requires a successful completed response without a refusal; conflicts before that terminal cannot be retried because the final outcome is unknown. Failed or incomplete terminal responses, including content filtering, cannot be overridden by identity recovery. The identity checks stay enforced on every attempt.

When OpenAI Responses cuts off a tool call at max_output_tokens, the embedded runner can continue once in the same run after earlier tools have settled. It appends a runtime transcript notice identifying the unfinished call when its ID is available and instructing the model to split the work into smaller tool calls. The incomplete call is not executed, completed tool results remain available, and earlier transcript entries are unchanged. A second truncation ends the run with the existing unfinished-tool-call warning. Transport error logs include the allowlisted incomplete reason without raw provider text; content-filtered and unrecognized incomplete responses retain their existing failure behavior.

A worker message-size failure is separate from a model context-window limit. Retry with a smaller response or continue on the Gateway. If the worker cannot preserve the model's continuation data, stop or reclaim it before retrying on the Gateway. Earlier tool actions may already have completed, so check their results before repeating them.

Targeted model transport diagnostics

OpenAI Responses transport failures retain structured status, code, type, and error/cause names. When the active request signal was aborted, abortReason identifies OpenClaw-owned causes with fixed labels: idle_timeout, no_progress, first_event_timeout, request_deadline, session_watchdog, restart, superseded, or caller_abort. Unclassified caller cancellation uses caller_abort. Provider error messages and arbitrary abort-reason text are never logged because they can contain prompt or tool content. Headers and request or response bodies are also omitted.

Prompt-cache history warnings identify the first changed message and bounded field paths, such as content[0].thinkingSignature or content[0].arguments. They report no message content, argument values, or signature bytes. Unknown extension keys appear as other; blocks beyond the retained diagnostic prefix appear as content[remaining]. A history warning identifies changed model context; it does not alone prove why a provider's cache-read tokens dropped.

When debugging provider calls, use targeted environment flags instead of raising all logs to debug:

OPENCLAW_DEBUG_MODEL_TRANSPORT=1 openclaw gatewayOPENCLAW_DEBUG_MODEL_PAYLOAD=tools OPENCLAW_DEBUG_SSE=events openclaw gateway

Available flags:

• OPENCLAW_DEBUG_MODEL_TRANSPORT=1: emit request start, fetch response, SDK headers, first streaming event, stream completion, and transport errors at info level.
• OPENCLAW_DEBUG_MODEL_PAYLOAD=summary: include a bounded request payload summary in model request logs.
• OPENCLAW_DEBUG_MODEL_PAYLOAD=tools: include all model-facing tool names in the payload summary.
• OPENCLAW_DEBUG_MODEL_PAYLOAD=full-redacted: include a redacted, capped JSON payload snapshot. Use only while debugging; secrets are redacted but prompts and message text may still be present.
• OPENCLAW_DEBUG_SSE=events: emit first-event and stream-completion timing.
• OPENCLAW_DEBUG_SSE=peek: also emit the first five redacted SSE event payloads, capped per event, on transports that support payload peeking. Responses transports remain metadata-only.
• OPENCLAW_DEBUG_CODE_MODE=1: emit code-mode model-surface diagnostics, including bounded activation facts, the final visible surface, and names of provider-native tools filtered because code mode owns the tool surface.

These flags log through normal OpenClaw logging, so openclaw logs --follow and the Control UI Logs tab show them. For backward compatibility, OPENCLAW_DEBUG_CODE_MODE also promotes general model-transport diagnostics to info; dedicated code-mode diagnostics are emitted only when that flag is enabled.

[model-fetch] start and response metadata (provider, API, model, status, latency, and request fields such as method, URL, timeout, proxy, and policy) uses debug by default. Responses with a non-2xx status remain at info, and transport failures remain warnings. HTTP 400 records also include a bounded, credential-redacted providerError from the provider's JSON error message when available. This distinguishes request validation failures from upstream credit failures that use the same status code. Diagnostics read at most 16 KiB with a one-second deadline; the SDK still receives the original response body. [model-fetch] and [responses] report caller-signaled AbortError as aborted at debug (or info with the targeted flags), rather than as provider failures. The requesting layer owns the final cancellation or timeout outcome; an SDK can also abort its fetch when its own deadline expires. Explicit TimeoutError reasons and failures without a caller abort remain warnings. Elapsed time includes local-service preparation and waiting for response headers, but excludes streaming the response body. The targeted debug flags above promote start and fast successful response metadata to info when troubleshooting.

Embedded model requests attach the raw sessionKey, runId, numbered requestId, and requestNumber to fetch and transport logs. When a failure also includes a provider requestId, the local identifier is labeled clientRequestId. String identifiers use lossless JSON escaping so control characters cannot forge log lines; their values are not redacted. The request sequence continues across same-run retries. Fetch response logs include the HTTP httpRequestId when supplied by the provider. Responses SSE and WebSocket stream_start / stream_done logs record the actual requested tier and max_output_tokens; completion also records the response ID, served tier, provider-reported output and reasoning token counts, stream duration, total request elapsed time, and terminal/incomplete reason. Unknown usage stays unknown, not zero. These metadata fields contain no prompt or response text; raw identifiers are retained locally and follow normal redaction at log egress.

[anthropic] replayed thinking dropped: N block(s) is a warning when Anthropic reports dropping invalidated thinking from replay. It includes the mismatch reasons and up to five affected message paths, not the thinking content. No debug flag is required.

[anthropic] server-side context edit: cleared N tool results (M input tokens) is an info-level line when Anthropic reports applying server-side tool-result clearing. It contains counts only, without tool arguments or result content, and requires no debug flag. See Session pruning for the routes and thresholds that enable clearing.

Trace correlation

File logs are JSONL. When a log call carries a valid diagnostic trace context, OpenClaw writes the trace fields as top-level JSON keys (traceId, spanId, parentSpanId, traceFlags) so external log processors can correlate the line with OTEL spans and provider traceparent propagation.

Gateway HTTP requests and Gateway WebSocket frames establish an internal request trace scope. Logs and diagnostic events emitted inside that async scope inherit the request trace when they do not pass an explicit trace context. Agent run and model-call traces become children of the active request trace, so local logs, diagnostic snapshots, OTEL spans, and trusted provider traceparent headers can be joined by traceId without logging raw request or model content.

Talk lifecycle log records also flow to diagnostics-otel log export when OpenTelemetry log export is enabled, using the same bounded attributes as file logs. Configure diagnostics.otel.logsExporter to choose OTLP, stdout JSONL, or both sinks.

Chat-send startup

The Gateway logs slow chat send when acknowledgement or startup takes at least one second. stage=request ends at acknowledgement; stage=startup ends when agent execution starts. replyInitialization covers inbound dispatch through session preparation. Later prompt, skills, tool catalog, model/auth, and runtime preparation belong to preparation, before the first model request. Fast sends retain numeric phase timings without formatting a log message; enabled diagnostic phase subscribers still receive their observations.

detail.* fields attribute those intervals using the existing reply and agent preparation spans, without enabling timeline file logging. They include session binding, cold transcript hydration, writer admission, snapshot reads, and commit, alongside workspace, Git baseline, hooks, and embedded preparation when used. reply.session.writer includes both queue waiting and reply.session.initialize; their difference indicates the writer wait. Parent and child spans overlap and must not be summed. Unfinished spans stop at the startup boundary; later model and tool latency does not extend them. Structured chat.send.detail.* phase events use the same durations when diagnostics are enabled.

Before acknowledgement, detail.persist.stage.* separates pending-input resolution, storage preparation, reads, authority preparation, message preparation, and durable commit. detail.persist.transcript.* covers target selection, owner authority, append, and transcript publication; detail.persist.turn.* further separates the append's preparation, authority, message hook, commit, and completion waits. detail.session.actor.command.queue measures the physical writer FIFO wait; execute includes the command's commit and publication intervals. Commit timing includes worker delivery, transaction admission, and durable settlement; it is not an isolated fsync measurement. detail.persist.resolveInput identifies input/media resolution, and detail.session.inputActor.acquire times actor acquisition. These nested spans use the same request-owned diagnostics without logging input bytes or enabling timeline files.

detail.attempt.setup.* checkpoints separate initialization, workspace preparation, and finalization. Within workspace, policy and policy_finalize cover session policy admission and revalidation; baseline_wait records only the time setup blocks on the concurrently captured Git baseline. mkdir, sandbox, and roots cover directory creation, sandbox preparation, and permission-root resolution. The sandbox.* checkpoints further separate session selection, state-owner admission, workspace selection, GitHub identity, pruning, workspace/skills layout, mounts, registry reads, backend provisioning, browser setup, and finalization. sandbox.layout.* separates workspace creation, skill imports, remote connections, eligibility, and synchronization. These are fixed labels and monotonic elapsed times, including failed preparation. Read siblings as consecutive phases and parents as inclusive totals; do not add a parent to its children. Prompt-media warnings beginning Native image: concern image-file access, not container-image validation, and occur outside attempt setup.

detail.attempt.system-prompt.runtime-and-references covers concurrent runtime and reference preparation. detail.attempt.system-prompt.context covers concurrent tool context, project memory, and TTS preference preparation; its nested tool-context span covers concurrent memory and watched-session reads. These group durations include overlapping work, not separate timings for each read.

Within detail.attempt.system-prompt.context, detail.project-memory.* separates runtime selection, dynamic imports, provider preparation, candidate retrieval, authority checks, rendering, provider close, and final publication. Legacy memory runtimes instead report legacy.manager, legacy.candidates, and legacy.render. The native provider owners further report detail.memory.provider-open.* (authority preparation, runtime lookup, opening, revalidation, failed-open cleanup, and binding) and detail.memory.provider-call.* (authority preparation, execution, and revalidation). Authority phases include waits for pending audience publications; candidate execution includes the provider's own storage and filtering work. These fixed labels contain no project keys or memory content, and include failed or denied operations. Parent totals overlap their children; the same provider phases can also occur during other memory operations.

Fresh managed sandbox projections also record fixed stages beneath sandbox.workspace_selection.prepare.reconcile: template selection and cloning, checkout packing, indexing and materialization, manifest scanning, publication, and cleanup. dependency.install separates package installation from dependency.retire, which includes removing the disposable builder before its files can be published. A slow reconcile total can therefore reflect container removal even after the installer has exited.

detail.worktree.managed.* and detail.worktree.sandbox.* carry the worktree owner's recorded phase durations, including templatePrepare, templateApply, templateValidate, templateRetire, checkout, runtimeQuiesce, runtimeResume, and unattributed. Template retirement includes removing an obsolete generation; it is separate from retiring a dependency builder. Runtime quiescence and resume include pausing and unpausing sandbox writers around synchronization. The top-level worktree interval can describe an earlier managed checkout; it is not the sandbox work inside attempt.setup. Use the matching detail.worktree.sandbox total when comparing setup with its nested worktree preparation. Recorded worktree summaries appear when their owner finishes; a preparation spanning acknowledgement is reported at completion.

Embedded attempt preparation

The model-dispatch prep summary extends stream-ready through history sanitization, provider validation, history windowing, context-engine assembly (including memory preparation), prompt building, prompt projection, and compaction preflight. It ends at the first provider invocation, including a compaction request when one is needed. These are elapsed checkpoints, not CPU measurements.

With timeline logging enabled, provider.request.serialized marks completion of native request-body encoding, before the transport consumes those bytes. Its runId and spanId match provider.request.started; the gap isolates request projection, worker admission, and serialization after provider invocation. The mark does not assert that the socket has sent bytes. A local mock provider's request arrival can measure the remaining transport interval without model time.

Embedded prep stages summaries separate two tool-preparation intervals:

• bundle-tools: awaited MCP/LSP preparation, tool normalization and policy projection, measured after preparation admission.
• tool-catalog: synchronous catalog construction, including Code Mode or tool search when enabled, schema projection and tool diagnostics.

tool-preparation is an inclusive checkpoint from the preceding bootstrap checkpoint. It includes both intervals, preparation admission waits and the remaining bootstrap work. These entries overlap: do not sum them or interpret them as CPU time. Later permission refreshes do not append initial-preparation entries. The conditional Code Mode and tool-search catalog messages still report their original activation/compaction events.

Older summaries charged bundle waiting to code-mode or tool-search and used bundle-tools for a later bookkeeping checkpoint. Those names do not provide the same timing boundaries as the corrected spans.

The summary keeps its existing identity fields and warning thresholds: ten seconds total or five seconds in any recorded stage; faster summaries use trace logging. A missing summary does not prove preparation completed without delay.

Session catalog provider waits

With process diagnostics enabled, the gateway/session-catalog logger records slow session catalog provider list for attempts that settle after at least one second. admissionWaitMs records initial provider admission waiting. providerElapsedMs spans the first provider invocation through final logical settlement, including waiting between steps of a stepped fill. completionDelayMs begins after final settlement and queue release. The Gateway's earlier operator-start queue is separate.

stepCount counts admitted callbacks. admittedStepMs sums their elapsed time through actual settlement, including authority checks, factory work, and I/O waits. These fields are not an exact disjoint partition and do not measure CPU time.

admitted and providerInvoked distinguish an attempt that never entered the queue's active slot from one that called the provider. Unreached intervals are omitted. outcome reports the attempt's resolution or rejection; signalAborted reports the signal independently and does not identify an error's cause or prove that native work stopped. An active provider call or next() step keeps its slot until its actual promise settles, including after cancellation.

providerIdHash hashes provider IDs of at most 256 UTF-16 units; longer IDs omit the field. It supports correlation, not anonymization or authorization. Host summaries count only returned gateway/node kinds, connection flags and error presence, inspecting at most 512 hosts. returnedHostCount reports the full array length and hostCountsComplete=false marks partial counts. No session rows, host IDs, provider labels, search text or error messages are included.

Each summary describes an underlying provider attempt. Cached and in-flight followers can receive several RPC responses from that one attempt. Later waitUntil host publications have a separate lifetime and are not included in the provider duration or returned-host counts. The log does not prove client receipt, identify which native operation was slow, or cover attempts that never settle. Missing records do not establish that there were no stalls.

Codex catalog phases

The same gateway/session-catalog logger records three Codex summaries when process diagnostics and warning-level logging are enabled. Each summary is emitted only after its observed operation settles and takes at least one second:

• slow Codex catalog list phases covers the plugin's list operation. managedSnapshotMs, controlWaitSumMs, exclusionMarkSumMs, adoptionSumMs and mappingMs identify reached work. Counts include localHostCount, controlPageCalls, exclusionMarkCalls and adoptionCalls. managedSnapshotMs is absent when the optional snapshot method or store is unavailable. nodeRegistryCalls and nodeRegistryMs measure the existing node registry invocation; pairedNodeCalls, pairedNodeSettled and nodeWaitSumMs describe the paired-node promises reached by the list. Cache counters coldStarts, refreshStarts, freshHits, staleHits and pendingJoins distinguish new producers, background refreshes, immediate cached delivery and callers awaiting an existing cold page.
• slow Codex catalog page producer measures one control-page calculation. listOperationId identifies its originating list when observed. origin is cold, refresh or uncached. controlRequestCalls counts entered control requests; inclusiveControlRequestWaitMs sums their elapsed waits and inclusiveControlRequestWaitMaxMs reports the longest one. postResponseMs covers subsequent provenance checks and page projection. provenanceChecks, provenanceCacheHits, provenanceReadCalls and provenanceMs describe the existing provenance path. provenanceReadCalls counts calls to the metadata reader, not filesystem read syscalls or chunks. stopReason, when reached, is exhausted, limit or page-bound.
• slow Codex catalog cache wait measures a caller waiting for a pending cold page. producerOperationId links it to an observed producer; listOperationId links the surrounding list when available. producerObserved=false means the producer's diagnostic identity is unavailable, not that no producer exists.

Page-producer summaries also accumulate elapsed time at the existing control phase transitions. Repeated phases, including selection retries, and multiple control calls contribute to the same page totals. Each total is rounded only when the summary emits. Unreached phases are absent; a reached phase may report zero milliseconds.

Rejected control calls can add controlFailurePhase and controlFailureCategory. The failure phase uses the same logical boundaries:

These are caller-observed intervals, frozen when that control invocation reports failure or closes. They exclude underlying work continuing after an outward timeout. A call on an already-pinned connection can omit acquisition and release because the surrounding pin owns those operations. Setup/settlement gaps and rounding mean the phase totals need not exactly equal the inclusive wait.

Categories are deadline-observed, scoped-rejection, rpc-method-unavailable (typed RPC error code -32601), rpc-error, or other. They use existing owner decisions and typed errors, without copying exception messages, stacks, response data, or arbitrary error codes. Plain startup, transport, and other unclassified errors remain other; the category does not identify their cause. Public unavailable-host messages remain sanitized.

Successful cleanup preserves an earlier error's phase unless the outer request owner observes its deadline. For deadline-observed, the phase is the active stage at that later decision, even if cleanup just completed. For example, release-client/deadline-observed can follow budget exhaustion before the client API was ever invoked; it does not prove cleanup caused the deadline. A cleanup error that replaces the request error reports release-client. Internally handled retries and successful requests do not publish failure fields, and late callbacks cannot overwrite a settled observation.

These fields do not prove a native request was written, a native process failed, or a response reached the client. The containing list can resolve with an unavailable host after a control call rejects.

operationId is local to diagnosticEpoch, PID and thread. It is not a session, native request or audit execution identity. One producer can serve several waiters, and a stale refresh can continue after a list returns. outcome=resolved means that the observed operation returned; a resolved list can include disconnected or error-bearing hosts.

All timings are elapsed time, including asynchronous waits. The inclusive control-request interval and its logical phase totals do not isolate physical request writes, wire latency or native CPU, and do not prove that a native process stopped. Several callers can be waiting on the same underlying work. Provenance time is included in post-response time, and host work can overlap, so sums need not partition the list's elapsed time. nodeWaitSumMs sums existing paired-node promise waits; it is not a disjoint node phase or proof of native completion. If the list closes while a child promise is unsettled, pairedNodeCalls can exceed pairedNodeSettled and the sum is partial. Later host publications retain their separate lifetime. Unreached timings are omitted, while a reached stage may report zero milliseconds.

The tracker admits at most 64 active diagnostic observations per JavaScript runtime isolate and shares a budget of 60 records per fixed 60-second window across these three summaries. These limits suppress observations, not catalog work. Each record's metadata is capped at 28 scalar fields and 2 KiB, excluding the logger envelope. omittedObservations reports accumulated capacity, rate or metadata-limit suppression on a later emitted record. Window-boundary bursts remain possible. Disabled diagnostics, logging levels, short operations, non-settlement or logging failures can also leave no record.

The records contain fixed labels, counts, timings and diagnostic operation identity. They omit connection fingerprints, queries, cursors, homes, paths, session/thread identifiers, titles, credentials and raw errors. Existing trace context may accompany the log; no trace or audit identity is created. These are ordinary performance logs and do not change audit collection, authorization, cache behavior or deadlines. Their sanitized attributes may flow through an already-enabled OpenTelemetry log exporter even when content capture is off. Missing logs do not prove an absence of stalls.

Lifecycle queue waits

When process diagnostics are enabled, the sessions/lifecycle logger emits session lifecycle queue waiting once when a queue acquisition is still pending after one second. It identifies the controller mutation queue or lifecycle admission/settlement wait and samples an observed current holder at that instant. The holder can have changed since the waiter entered the queue. A delayed timer that runs after acquisition emits no holder sample.

operationId and holderOperationId identify diagnostic operation instances within diagnosticEpoch, PID and thread. Operations use the fixed boundary labels lifecycle, mutation and compaction. Mutations also include the bounded mutationKind (for example, create, delete, or worktree-cleanup); queue samples include holderMutationKind. Existing request traces appear in operationTraceId/operationSpanId and separate holderTraceId/holderSpanId fields when present. Missing trace fields remain unknown; no new trace or audit execution identity is created.

identityHash is a salted digest of the already-normalized store/session identity. It correlates only inside the same JavaScript runtime isolate and diagnostic epoch. Raw session keys and paths are omitted. The digest is operational correlation, not anonymization or authorization evidence. Both records include sessionScopeHash, the salted digest of the first normalized identity, and identityCount. For batches this names one scope, not every affected session. holderSessionScopeHash identifies that same scope for the current holder. No session titles, transcript content, repository names, or raw identities are logged.

slow session lifecycle operation records serialized mutations and compactions taking at least one second through their actual owner release. Lifecycle effects can span an entire agent turn, so their elapsed time alone is not a warning: they are reported only after spending at least one second in a queue themselves or being observed as the holder by a queue-waiting diagnostic. This observation still counts if the warning budget suppresses the queue-waiting record. It separates mutationQueueWaitMs, lifecycleQueueWaitMs, completionDelayMs and phaseDurationsMs.prepare, .run and .finalize. The holder's current holderPhase can also identify activation, admission or release work. A lifecycle operation includes admission, validation, the store-writer barrier and the acquired effect lifetime. Mutation timing includes the wait for captured turns, claims and effects before its body runs. A holder can remain unknown while a mutation waits for a turn or claim; an effect sample names only an effect whose settlement that mutation is actually awaiting. completionDelayMs measures synchronous owner-release cleanup after the callback settles. A caller can cancel before all of its queued work unwinds; signalAborted reports the signal without claiming that the holder has released.

Subagent-kill mutations also report subphaseDurationsMs for awaited caller facts, caller Stop hooks, descendant discovery, durable kill intent, admission draining, runtime loading, registry settlement and obligation retirement, prefixed with prepare. or run.. These durations include rejected waits and accumulate repeated waits; they exclude each operation's synchronous work before returning its promise. Registry settlement includes its persistence and session cleanup bookkeeping. The joins for raw source settlement and child-command process cleanup occur after the mutation and are not included. The subphase totals therefore need not equal the enclosing phase durations.

The tracker preserves outer ownership across reentrant work and retires a holder only when its actual callback exits and the owner releases custody. Its metadata follows existing controller mutations and effects; it does not create another execution queue. Per runtime isolate, it retains at most 128 holder descriptors and 32 one-shot wait timers, and emits at most 60 records per minute. The mutation owner explicitly distinguishes reentry, so unobserved outer mutation holders stay unknown at capacity or after enablement. omittedObservations on a later record reports suppressed observations; missing records never prove no wait.

Controller transition records use session controller phase on the diagnostic logger at info level in the structured file log only, with the same process diagnostics gate. They omit console output because the message alone does not identify the phase or status. Slow lifecycle warnings remain on the console. Session, source and priority-source identities are salted digests, and reasons are fixed labels. These records have a separate 60-record-per-minute budget so transition traffic cannot suppress the slow warnings above. Their omittedObservations count applies to phase records independently.

Elapsed intervals can include asynchronous waits and nested work, so phase and queue totals need not form a disjoint partition. isMainThread=true identifies the emitting thread; a long .run interval does not establish event-loop blocking. Correlate it with event-loop delay or a CPU profile before attributing a stall. Work before lifecycle admission, including request preflight, is outside these intervals. A holder sample identifies who owns that queue at the sampled instant, not every predecessor responsible for the entire wait or which work consumed CPU. These are ordinary performance logs. They do not use or change audit identity, decisions, retention, principal attribution or admission authority.

Worker pool capacity

Gateway status responses include workerPools.transcriptReconciliation and workerPools.modelCatalog. Each reports maxWorkers, workers, workersCreated, activeTasks, and pendingTasks from the pool owner. Both Gateway pools admit one worker at a time. Pending tasks include queued and executing work; creation counts belong to the current pool lifetime. workerPools.modelCatalog also reports workerFailures: how many model-catalog workers have failed (run out of memory, exited, or timed out) since the Gateway started. A failed pool is replaced, so this count survives replacement. Each failure also logs one model catalog worker failed warning when it happens, with the worker's reason and the number of agent catalogs to republish on a new worker. A worker that exits while idle is counted and logged at once; the next catalog request replaces it. Shutdown and plugin retirement are not counted. The startup trace's memory.ready record also includes these pool counts.

These figures describe worker and task counts. Process RSS includes every isolate and native allocation; Node's process heap flags can override a worker's requested heap limits. Use constructor or per-isolate measurements when attributing memory growth to a particular worker.

Slow Git content reads

With process diagnostics and info-level logging enabled, git/worker emits slow Git content read after a diff, diff-baseline, or PR branch-facts operation lasting at least one second. The journal message includes the same fields as the structured file log. Records are limited to 60 per minute.

operation identifies the caller family. checkoutId is a truncated SHA-256 of the absolute checkout path; linked checkouts have different IDs. checkoutClass is managed when the caller supplies managed-index ownership, otherwise unspecified. Paths, refs, command arguments, and output contents are not logged. For checkout.diff, scope distinguishes all, uncommitted, and commit; summaryOnly distinguishes summary polling from full patch requests. These fields describe the admitted request without inspecting the checkout again.

workerQueueWaitMs measures admission wait (null if never dispatched). firstHostRequestMs includes that wait plus worker startup and work before the first host request. workerMs covers subsequent worker and host work; settlementMs covers final cleanup. summedGitQueueWaitMs measures waiting for shared content-process slots, while summedGitWallMs sums command execution including process settlement. Concurrent commands overlap, so their sum can exceed operation duration; these are wall times, not CPU times.

gitCommandCount counts started host command requests, not Git's own subprocesses. gitStdoutBytes and gitStderrBytes count captured bytes, excluding any truncated output. gitTimeoutCount counts returned timeouts; slowestGitCommand records the longest command's allowlisted name, diff mode when applicable, duration, and termination. An operation can return successfully after a command times out because optional statistics fall back to unknown. Artifact and maintenance operations contribute to aggregate Git worker metrics but do not emit this log.

Slow worktree cleanup

With process diagnostics and info-level logging enabled, two subsystems log operations lasting at least one second after they return or throw:

• agents/worktrees: slow managed worktree removal measures removal through allocation-lease settlement. admissionMs covers acquisition attempts, backoff, setup, and scheduling before the removal callback starts. bodyMs covers that callback; finalizeMs covers drainage, final authority checks, lease release, and completion delivery. Create and restore operations do not emit this record.
• git/ref-mutation: slow Git ref mutation measures shared Git-ref queue operations. resolveMs covers common-directory resolution; queueWaitMs covers time from enqueue to callback entry; queuedOperationMs covers the callback and delivery of its settlement. It can include multiple Git commands and does not identify a queue holder or every predecessor.

Removal also records the stages reached inside bodyMs:

• preparationMs: authority and removal-claim checks, repository rebinding, and worktree lock inspection or unlock.
• snapshotMs: snapshot preparation and publication, including provisioned-file capture and snapshot-failure cleanup.
• checkoutRemovalMs: deletion admission checks and physical Git worktree removal through result validation.
• bodyFinalizeMs: branch deletion, prune, empty-parent cleanup, registry finalization, or removal-claim cleanup after failure. This is distinct from finalizeMs, which measures the allocation-lease wrapper's final settlement.

Unreached stages are absent; a reached stage can report zero milliseconds. Exceptions close the active stage and include claim cleanup in bodyFinalizeMs. These fields subdivide the admitted body, not individual Git commands or CPU work. They use the same completion record and rate budget.

Both records include durationMs in integer milliseconds, callbackEntered, and outcome (returned or threw). Removal that never enters its callback reports all elapsed time as admissionMs and omits bodyMs and finalizeMs. Git directory resolution failure reports resolveMs and omits unreached queue and operation durations. Phase durations partition each record's interval before rounding. These intervals include asynchronous waits: admission is not pure lock wait, and queued operation time is not child-process CPU time. They nest within broader operations such as session-patch worktreeCleanup; do not add nested durations to the enclosing total.

Each subsystem has a separate fixed budget of 60 records per 60-second window per JavaScript runtime isolate. Bursts across window boundaries remain possible. omittedObservations reports suppressed records on the next emitted record, then resets. Pending operations emit nothing until they settle; disabled diagnostics, log levels, thresholds, and budgets can also leave no record. Missing records never prove there was no delay.

The added fields are fixed scalar timings, outcomes, counts, pid, threadId, and isMainThread. They omit repository paths, refs, arguments, raw errors, and command output. Records preserve an existing valid diagnostic trace when available; they create no trace, operation identity, or private-identity hash. Use the trace to associate nested records, without treating elapsed time as CPU attribution. These diagnostics measure cleanup without changing its ordering or completion behavior.

Slow agent database opens

Foreground Gateway startup reports a bounded set of startup phase records even without opt-in tracing. CLI records identify entry-module loading, environment selection, command imports, and state preparation before loading configuration. Bootstrap subphases separate config-guard imports, config and database admission, plugin metadata, payload verification, and lease acquisition. Gateway records separate maintenance work, listener binding, plugin loading and attachment, worker startup, model preparation, channel startup, and restored subagent activation. Measured phases log both their start and elapsed duration; total is elapsed time from process startup (or the current in-process restart). These are wall times, including asynchronous waits, not CPU measurements. Nested durations overlap their parent phases; do not add them to the parent total. Plugin service starts taking at least 250 ms always log one compact sidecars.plugin-services.<plugin>.<service> span with elapsed time and a success or failure outcome, even without opt-in tracing. Plugin and service IDs use encoded diagnostic segments; the record contains no service payload or error text. Set OPENCLAW_GATEWAY_STARTUP_TRACE=1 for the complete breakdown and per-plugin

separates the environment selector's module imports from its work; cli.environment.* breaks down trusted dotenv loading, config reads, validation, and recovery inside that work. These steps also appear in process.bootstrap. For external plugins, modulePreparationMs measures awaited source acquisition inside loadMs; it is not additional time to add to that total.

Desktop permission narrowing and session identity checks remain on their required admission paths. Transcript projection repair, orphan settlement, channel maintenance, pairing diagnostics, and restored-subagent activation run after readiness. Retired plugin captures join the existing post-ready idle cleanup. History reads use the transcript owner's bounded on-demand repair when needed. Orphan status and its failure receipt settle together in the database worker, with current run ownership checked before commit. These scheduling changes do not change stored formats or require an update migration. Each CLI admission pass shares one read-only state snapshot and releases it before live guards and writes; acquiring a preparation lease still starts a fresh pass.

The slow OpenClaw agent database open warning includes phaseDurationsMs when a persistent database open takes at least one second:

The integer millisecond durations partition elapsedMs, measured with a monotonic clock after lease acquisition. Live cache hits remain quiet. These are elapsed durations, including asynchronous waits, rather than CPU time or proof that the main event loop was blocked for the whole interval.

The structured warning also includes pid, Node's threadId, and isMainThread for the opener emitting it. Inspect each openclaw logs --json event's original raw record; warn-level console text also carries these fields as key=value pairs. An opener on the main thread may have awaited an integrity Worker, so these fields do not identify the thread performing every phase. admissionMode records the actual sync or async open driver. Async admission offloads its initial integrity check; resumed validation and repair can still run on the opener. Correlate the process ID with the log timestamp and current process; PIDs can be reused after exit.

integrityGateMs covers the initial integrity check through admission revalidation and resumption. When the driver measures its synchronous integrity and foreign-key callback, integrityCheckSyncMs reports that callback's elapsed time and integrityOutsideCheckMs reports the remaining gate time. The two integer fields partition integrityGateMs; the remainder includes admission, IPC, scheduling, and revalidation, not just a parent queue wait. These are wall durations, not CPU time. A reclamation Worker can report this synchronous check while its admissionMode is async. An asynchronous child-process check leaves both fields absent because its parent cannot measure the callback itself.

SQLite reclamation Workers also emit slow SQLite reclamation Worker operation at warn when their joined operation takes at least one second. The record is emitted after a settled Worker result or native exit and parent admission settlement. It includes the parent's pid, threadId and isMainThread, the actual Node workerThreadId, reclamationKind, elapsedMs, terminal outcome (resolved or rejected), and exitCode. Timing starts after admission to the archive Worker queue and includes startup, validation, admission waits, work, and cleanup. It does not measure CPU time, isolate a validation phase, or establish that the emitting parent thread blocked. Short writer sections can therefore remain quiet while this whole-operation warning exposes slow preparation between them. The record inherits an existing parent trace when available. Failed retained reclamation operations also include sessionIdHash (when targeting one session), error (the redacted message and causes), and errorFrame (the first stack frame). These failures emit one warning even below one second, named SQLite reclamation Worker failed; slower failures use the existing slow-operation warning. Automatic maintenance that a newer session write supersedes before commit is not a Worker failure: maintenance retries after the write quiet window, and a fast superseded Worker logs SQLite reclamation Worker superseded by newer inputs at debug level. Session identifiers use the same hash as other session SQLite diagnostics; the failure fields are redacted and bounded to 2,048 characters each. Cold-storage operations use the same warning with reclamationKind set to cold-batch (archive or externalize), cold-maintain (reclaim free pages), or cold-restore (restore a transcript). Their writer warnings carry the same Worker identity and numbered admission fields.

State lease renewal failures

The state/lease logger warns on the first non-loss failure of a parent-scheduled state lease renewal. It includes the lease scope, key, attempt number, and error name and codes, without error messages, stacks, or database paths. Further failures are counted until the lease timer closes or expires.

state lease expired warnings and the loss error distinguish a renewal still pending (with its start time and age), the last renewal failing (with its attempt and error classification), and no renewal attempted since the last committed expiry (with the age of that successful commit). Each expiry warning includes the failure count. A pending request identifies an unsettled actor operation; it does not establish which actor phase is blocked. Lease durations, retries, and ownership checks are unchanged.

SQLite transaction timing

The sqlite/transaction warnings slow SQLite transaction hold, slow SQLite transaction step, and SQLite transaction lock wait failed include database, operation, pid, Node's threadId, and isMainThread for the thread executing the transaction. Explicit labels take precedence; otherwise diagnostics use the native database path and the current Worker operation. An in-memory database is :memory:, a retired handle is unavailable, and a caller without operation context is unlabeled. Hold warnings also include mode (deferred or immediate). Inspect the original raw record in openclaw logs --json to distinguish the main thread from Workers sharing the same process. async: false describes the synchronous transaction helper; it does not identify the thread.

Hold time starts after BEGIN succeeds and includes the synchronous callback, result checks, and commit or rollback. It excludes database opening and the begin step. Host admission waits inside a transaction count toward its hold; overlapping deferred read holds do not establish that multiple writers held a lock. Successful begin and commit step timings include native execution, storage work, and scheduling delays; they do not establish lock contention. The separate SQLite transaction lock wait failed warning identifies caught SQLite lock errors. These elapsed durations do not measure SQL CPU time or establish a causal link to a nearby request.

Hold warnings label the callback as callbackKind: "application-transaction": OpenClaw's transaction body, not a SQLite busy handler or SQL function. phases.databaseAdmissionWaitMs measures synchronous metadata exchanges with the host, separately from authority waits in hostAdmissionWaitMs. Metadata waits during COMMIT belong to this field and are excluded from the hold's commitMs. The legacy sqlMs field excludes both waits and commit time, but includes JavaScript between SQL statements. The default 1,000 ms logging threshold does not establish a one-second timeout.

Older builds report session.reclamation.commit-settlement for a parent-side synchronous SQLite check after authorizing a reclamation or cold-storage commit. The parent now atomically accepts the commit after checking live authority and awaits settlement asynchronously, without that check or its lock wait.

Hot transcript reads identify their purpose in operation: session transcript <purpose> read, where <purpose> is identity, header, tail, incremental, checkpoint, events, raw rows, storage rows, or match. These fixed labels distinguish readers without retaining session IDs or transcript content. Nested reads remain part of the outer transaction's timing; older warnings use the generic session transcript hot read label.

session branch summaries read covers the snapshot read and branch-summary computation. Stored sessions perform this work in a background Worker; incognito sessions use their process-held database. Cache hits do not perform this scan.

Immediate BEGIN warnings also include beginAdmission: nativeAttempts counts actual native BEGIN IMMEDIATE calls and nativeMs measures those calls; serviceCalls counts synchronous admission-service callbacks and serviceMs measures them. A service callback may find no work, so its count does not mean that reclamation was authorized. Failed attempts and throwing callbacks retain their partial measurements. Deferred BEGIN and COMMIT have no breakdown.

These fields use the same wall clock as the unchanged elapsedMs total. Native time excludes busy-timeout configuration and restoration; other bookkeeping can leave a remainder. A service can synchronously join another transaction, whose time is already included in the outer serviceMs; do not add nested warnings together. The breakdown does not identify CPU time or a physical lock holder.

SQLite session writes

The session-sqlite subsystem emits slow SQLite session write when total elapsed time reaches 1000 ms, and SQLite session write failed when a write fails. Both warnings include operation, a label from a fixed set of semantic operation names identifying the callback that owns the SQLite writer lane.

The timing fields separate the elapsed interval into:

• queueWaitMs: time waiting to enter the writer lane.
• writerExecutionMs: the owning callback's duration, including asynchronous waits.
• completionDelayMs: time between callback completion and the caller resuming.

These fields are available when the queued callback started and finished; elapsedMs records the total duration. The warning's console line carries the same fields as key=value pairs; openclaw logs --json shows the original raw record.

slow agent database reservation covers direct, Worker, and ordered-read reservations when queueWaitMs + writerExecutionMs reaches 1000 ms, even when execution itself is fast. Completion delay alone does not trigger this warning. When a writer was holding the lane at enqueue, blockingOperation and optional blockingCaller identify that holder; blockingWorkerOperations lists its bounded Worker operation labels. This is the holder observed at enqueue, not a breakdown of every writer ahead in the queue. Reentrant borrowing does not emit a second reservation warning. No reservation warning is emitted below the threshold.

Prometheus already exposes openclaw_worker_queue_wait_seconds with kind="sqlite_writer" and bounded request_class labels. It measures Worker request enqueue-to-dispatch time; an outer reservation wait before that enqueue is visible in the reservation warning, not in this Worker histogram.

Use operation to locate the owning code path. It does not identify a specific SQL statement, measure CPU time or lock contention, or establish that a nearby RPC caused the delay. Older records may lack operation; do not infer it from adjacent log messages.

session.reclamation.worker-commit labels every numbered Worker write admission, not only its final commit. reclamationAdmissionId is the actual request ID, scoped to that Worker and process. reclamationAdmissionReleaseCause records the observed worker-release message or worker-exit event. It does not infer an initial/final phase or prove successful commit or cleanup. An early failure can leave the release cause absent because neither event has been observed yet.

For session.lifecycle.artifacts-prepare, the same warning includes a bounded artifactPreparation object. admissionMode distinguishes an existing cached handle from asynchronous acquisition; admissionMs stops when the planner receives that handle. Asynchronous acquisition may include shared admission and integrity-check waits, so it is not a CPU measurement.

The remaining millisecond fields separate node inventory and selection (nodeInventoryMs), references and entry deletion plans (referencePlanningMs), orphan selection and plans (orphanPlanningMs), and transcript marker iteration (markerScanMs). Orphan planning excludes marker time. Counts report existing node/window rows before agent or prefix filtering, referenced IDs, selected entries, entered marker queries, consumed marker rows, and deletion plans. They are observed result counts, not SQLite internal row visits. No identifiers, marker text, transcript contents, or byte counts are added. completed: false marks partial observations when preparation failed; absent fields were not completed. These fields do not change the warning threshold or prove that a nearby request caused the work. Rounding and work outside the measured subphases can leave a difference from writerExecutionMs; do not assign that remainder to a specific phase.

For session.history.archive-prune, the same slow or failure warning can include one bounded archivePruning object. Its trigger is recorded at the call site: initial, after-eviction, or final. It distinguishes pruning passes within the maintenance flow; it does not identify the request that caused maintenance.

The object aggregates observations across the pruning pass:

• admissionMs, cachedAdmissions, and asyncAdmissions measure database acquisition and count its observed modes. Admission time ends at callback entry or acquisition failure and can include shared admission and integrity-check waits. A refusal before mode selection adds admission time without incrementing either mode count.
• checkpointMs, checkpointMaxMs, and checkpointCalls report total time, longest call, and calls entered. checkpointIncomplete counts calls returning false, which can mean a busy checkpoint or an error; it does not identify a lock holder or distinguish those outcomes. A thrown checkpoint contributes to call count and time without incrementing checkpointIncomplete.
• vacuumMs, vacuumPasses, and vacuumPagesRequested measure incremental vacuum calls and their requested page counts. Requested pages are not confirmed reclaimed pages.
• queryMs covers existing archive-presence, candidate, unpublished-name, and freelist reads. rowDeletionMs covers the archive row-deletion transaction.
• fileRemovalMs, removedFiles, missingFiles, and failedRemovals report existing file-removal outcomes. removedFiles counts successful current-layout and legacy removals. missingFiles counts current-layout removal attempts that return ENOENT. Other current-layout failures and all unsuccessful legacy removals count under failedRemovals; the legacy count includes missing paths, non-files, and stat or removal failures.
• measurementMs and measurements cover awaited disk-usage measurement attempts, including failures and time queued for the measurement Worker, scanning, and returning the result. legacyInventoryMs covers legacy file inventory, filtering, and sorting.

All durations are wall time, including asynchronous waits, rather than CPU measurements. completed: false retains partial observations when pruning throws; an absent stage timing field means that stage was not entered. completed: true means the pruning pass returned normally. It does not prove that every checkpoint completed, every removal succeeded, or the high-water target was reached. Rounding and unmeasured work can leave a remainder relative to writerExecutionMs; checkpointMaxMs is already included in checkpointMs.

These fields reuse existing operations without additional store reads, per-file records, paths, names, or content. They do not change the warning threshold, checkpoint mode or timeout, or archive-retention behavior.

Slow reply preparation

Before acknowledgement, slow chat send reports requests taking at least one second, including the first turn of sessions.create. Its detail.admission.* fields separate work admission and validation inside that admission. The detail.persist.* fields separate pending input staging, transcript persistence, and dashboard authority preparation. These fixed labels contain no message content or session identifiers and require no diagnostic flag. Durations include awaited work: admission.revalidate is nested inside admission.work, so their values must not be added together or treated as SQLite execution time.

Admission subphases separate operator retention (operator), lifecycle preparation through the session writer barrier (lifecycle_prepare), ACP metadata (acp_metadata), placement preparation and refresh (placement_prepare, placement_refresh), and current session/retry validation (current_session). Lifecycle preparation includes agent admission checks and queue waits; it is not a pure SQLite lock timer. These fields use the detail.admission. prefix. Sends without restart-safe recovery skip the unused transcript lifecycle-header read; current session and caller checks still run. Restart-safe admission reuses the request's prepared session entry for ACP metadata. Final current_session validation stays ordered; a changed lifecycle causes ACP metadata to be prepared again before that validation repeats. Durable input custody still completes before acknowledgement.

After acknowledgement, runAdmission also covers steering into an active turn. Its detail.run_admission.steer_* subphases separate authority, continuation and delivery preparation, ordered admission (steer_admit), waiting for the backend outcome (steer_settlement), and finalization (steer_finalize). steer_prepare encloses preparation through ordered admission. A long stage=steer interval can therefore be an intentional wait behind older input or for its transcript commit, rather than slow startup of a new turn. Nested totals overlap.

When a reply spends a long time preparing, inspect the normal Gateway logs:

openclaw logs --follow --plain | rg 'timings|agent turn milestone|liveness warning'

Reply resolver, dispatch, and agent-turn preparation milestones include stage durations, elapsed time, and available run/session identifiers. Without profiler flags, they warn at 10 seconds elapsed or 5 seconds in one preparation stage. Codex preparation also logs each completed slow stage immediately, including failures, and emits a native-turn-handoff summary before submitting the native turn. Timing records contain stage names and identifiers, not prompts or tool arguments.

Dispatch preparation separates reply.wait_admission_ticket from reply.admit_pre_dispatch, reply.admit_dispatch, and reply.admit_command_resolution. These spans distinguish waiting behind an earlier input from waiting for the session's execution owner. A cancelled or failed request that never reaches the reply resolver still reports slow preparation under the same thresholds.

Embedded-run startup, prep, core-plugin-tool and auth stage summaries include pid, threadId and isMainThread in the message to distinguish emitters sharing a log file. These identify the summary emitter, not where every timed operation ran. Elapsed stage time can include asynchronous waits and is not CPU time.

Use the first turn_accepted, model_call_started, tool_execution_started, and assistant_output_started milestones to separate startup from later activity. Delayed first assistant/tool activity is logged once at info by default, because provider and tool latency is not itself a preparation warning. These are runtime observations: native turn acceptance does not prove that a provider request has started. Whole-turn summaries remain profiler-only because their totals include model and tool time. Compare the individual preparation stages before attributing a long turn to Gateway startup. A simultaneous liveness warning with high event-loop delay can explain delays across several sessions.

For shorter delays, profiler flags lower the warning thresholds. They are not required to diagnose a multi-second startup stall.

main-thread stall warnings include stallStartMs, stallEndMs, and reportedAtMs as Unix timestamps in milliseconds. Scoped callbacks have measured start and end times. Unattributed active-loop stalls use the observed gap between 20 ms heartbeats; these bounds describe a blocked-loop observation, not an exact JavaScript callback lifetime. Idle monitoring uses the missed wake deadline and the resumed wake instead, without adding idle heartbeats.

On supported Node runtimes, main-thread stall profile selects samples only inside the longest reported stall's saved interval. windowStartMs and windowEndMs identify that interval; reportedAtMs identifies profile emission. The recorder keeps its 25 ms sample interval and bounded profile buffers. Later work outside the interval is excluded, even when reporting is delayed. Sample counts include idle samples whose frames are omitted from inclusive, so a single listed stack does not mean that stack occupied the whole interval.

The report also includes self, up to ten sampled leaf frames as [frame, samples, sampledMs, nearestAppFrame]. These identify the work beneath shared inclusive callers; nearestAppFrame is the nearest OpenClaw frame on that sampled stack, or null when none is visible. Sampled milliseconds use the selected profile time deltas, clipped at the start of the stall; they are an attribution estimate, not an exact CPU timer. deepest contains up to three hot stack tails as [samples, frames], with up to eight frames ordered from caller to sampled leaf.

The report payload has a hard 2 KiB limit. reportTruncated=true means lower priority entries were omitted to fit: inclusive ancestors first, then extra stack tails and their outer callers, then lower-ranked self frames. JSON stays complete, and sampled leaves take priority. truncated separately describes limited native profile coverage. Code locations and symbols retain the same redaction policy as other diagnostic profiles.

On supported Node runtimes, a worker also reports a continuously blocked main thread after 30 seconds, then at most once per minute while the same stall continues. main-thread stall in progress is written directly to standard error (and the service journal), so it does not wait for the main event loop or the configured file logger. It uses the existing bounded native recording: an inspector interrupt takes the snapshot, and the worker parses, sanitizes, and formats it within the same 2 KiB profile budget. captureMs measures the native snapshot interrupt; no debugger listener or additional sampler is started. Native waits that cannot service an inspector interrupt can still prevent a JS profile; this mechanism diagnoses JavaScript spins, not every native deadlock.

Model call size and timing

Model-call diagnostics record bounded request/response measurements without capturing raw prompt or response content:

• requestPayloadBytes: UTF-8 byte size of the final model request payload. OpenAI and Anthropic Messages transports reuse the serialized body size, after payload hooks and transport adjustments. Other transports measure the payload exposed by their hook; this may differ from the final wire body. Injected Anthropic SDK clients and SDKs without a host fetch also use the hook measurement because they own encoding and network I/O.
• responseStreamBytes: UTF-8 byte size of streamed model response chunk payloads. High-frequency text, thinking, and tool-call delta events count only the incremental delta bytes instead of full partial snapshots.
• timeToFirstByteMs: elapsed time before the first streamed response event
• durationMs: total model-call duration

These fields are available to diagnostic snapshots, model-call plugin hooks, and OTEL model-call spans/metrics when diagnostics export is enabled.

Local llama.cpp and native Ollama requests show “Waiting for the local model to respond” after one second without a response event. Control UI, TUI, and channels with progress drafts show this transient status, including later model calls in the same turn. It clears on the first stream event, cancellation, failure, or completion. Fast requests do not flash a waiting status, and no server polling or user configuration is required.

If you cancel while this local-model wait is visible, the terminal acknowledgment adds “Cancellation requested; the local model may still be finishing.” Control UI and TUI show the caveat for five seconds; channels append it to their existing stop reply. The run ends immediately. This is not evidence that the backend is still busy or has released its slot, and fast cancels or cancels after output begins do not add the caveat.

“Waiting for a free model slot” is reserved for request-correlated backend evidence. The pinned llama.cpp server and Ollama do not expose that evidence for queued requests: occupied slots, model residency, and silence alone cannot distinguish queueing from loading or prompt processing. They use the generic response-wait wording.

Completed llama.cpp and Ollama calls also report providerProcessingMs and providerWaitEstimateMs in model-call diagnostics, provider request timeline events, and Gateway stability snapshots and bundles. openclaw gateway stability shows these as processing and wait~, alongside total wall time.

Processing is the server-reported prompt evaluation plus token generation time. Estimated wait is the nonnegative difference between wall time and processing: it includes backend queueing, model loading, transport, and terminal settlement overhead, so it is not an exact slot-wait measurement. Ollama's load_duration can include scheduling and is not counted as inference. These fields remain absent when final server timings are missing or invalid, including cancellations without final statistics. An aborted call does not prove the backend released its slot. No additional server polling is performed.

Prompt-cache drop warnings include requestGapMs (start-to-start time since the session's previous request), responseId, sessionId, and promptTokens when reported by the provider. The previous/current wire summaries include the requested serviceTier so tier switches can be distinguished from cache loss on an unchanged route. Drops classified as unattributed use debug; attributed drops remain warnings. providerPrefix compares final encoded request segments against the last request with usable cache-read usage: system instructions, tools, history messages, and other request parameters. It names the first differing segment or prefix-match when the previous prefix is unchanged. A match does not prove provider cache availability or retention. Unsupported transports and missing baselines report unavailable.

Prompt content is retained only as hashes, within the existing 512-entry diagnostic tracker. History comparison keeps the first 512 message hashes and one remaining-tail hash. A changed tail reports message-tail:512; a growing or shrinking tail reports unverified-after:512 because its earlier prefix cannot be verified. Background sessions sharing a provider cache key keep separate diagnostic baselines. No prompt content or digest values appear in these warnings.

OpenAI service-tier rejection and downgrade records include the requested model, account route, run/session IDs, original requested tier, and final wire-request tier. Terminal responses also include the served tier, response ID/model, total input tokens (including cached input), and cached tokens when reported. Changed account-catalog observations log at info; repeated observations use debug. Join the response ID to a prompt-cache diagnostic to inspect the same completion; a downgrade alone does not prove a cache miss. These are raw file-log fields and follow the usual redaction rules at egress.

Console styles

logging.consoleStyle accepts pretty or json:

• pretty: human-friendly, colored, with timestamps.
• json: JSON per line (for log processors).

A third rendering style, compact (tighter output, best for long sessions), is applied automatically when stdout is not a TTY. It is no longer a settable config value; openclaw doctor --fix maps a stored consoleStyle: "compact" to "pretty".

Redaction

File logs store raw records with owner-only (0600) permissions. OpenClaw masks sensitive values when logs are read through logs.tail (CLI or Control UI), printed to the console, or exported through diagnostic/support bundles and OpenTelemetry. Direct file reads are raw: protect and review them before sharing. Log messages should contain IDs, sizes, and outcomes, never message text, prompts, tool arguments, or tool results. Config snapshots, audit output, and credential-bearing URLs and command-line options retain their credential masking.

Gateway agent ingress returns replies through results and session transcripts without echoing them to service logs. Nested-agent replies are not printed to logs. Interactive openclaw agent commands still print replies to stdout. Stalled and stuck session diagnostics retain session IDs, ages, tool names, cron metadata, and recovery outcomes without copying assistant replies from transcripts.

logging.redactPatterns replaces the default regex string list for logs and diagnostics. Built-in structural protections for form bodies, structured authorization headers, and bare AWS secret access keys always apply, including when this list is customized. Redaction is best-effort; review diagnostic exports before sharing them.

If the list contains malformed or rejected patterns, OpenClaw skips those entries, keeps valid patterns, and adds the default protections. It warns once per process without printing the rejected values. Correct logging.redactPatterns in your config; directory inspection and Doctor can still report the config error.

Tool arguments and results are not secret-masked. This includes model context, session transcripts, tool events and summaries in the Control UI, session-history and search results, and reads of .env files. Source such as API_TOKEN = computeToken() reaches the model unchanged. Existing size limits, media handling, and history visibility rules still apply.

Active session transcripts live in the per-agent SQLite database. They can contain credentials and other private data returned by tools or supplied in conversation. Protect access to the state directory and review transcripts before sharing them. Already-masked historical content cannot be reconstructed.

Diagnostics and OpenTelemetry

Diagnostics are structured, machine-readable events for model runs and message-flow telemetry (webhooks, queueing, session state). They do not replace logs — they feed metrics, traces, and exporters. Events are emitted in-process by default (set diagnostics.enabled: false to turn them off); exporting them is separate.

When a session directive rejects a turn before model execution, its existing message.processed event reports outcome: "skipped" with a closed reason code and the usual channel, message, and session correlation. The rejection does not add the user's message, model token, or error reply to that event.

Two adjacent surfaces:

• OpenTelemetry export — send metrics, traces, and logs over OTLP/HTTP to any OpenTelemetry-compatible collector or backend (Datadog, Grafana, Honeycomb, New Relic, Tempo, etc.). Full configuration, signal catalog, metric/span names, env vars, and privacy model live on a dedicated page: OpenTelemetry export.
• Diagnostics flags — targeted debug-log flags that route extra logs to logging.file without raising logging.level. Flags are case-insensitive and support wildcards (telegram.*, *). Configure under diagnostics.flags or via the OPENCLAW_DIAGNOSTICS=... env override. Full guide: Diagnostics flags.

For OTLP export to a collector, see OpenTelemetry export.

Troubleshooting tips

• Gateway not reachable? Run openclaw doctor first.
• Logs empty? Check that the Gateway is running and writing to the file path in logging.file.
• Need more detail? Set logging.level to debug or trace and retry.

Related

• OpenTelemetry export — OTLP/HTTP export, metric/span catalog, privacy model
• Diagnostics flags — targeted debug-log flags
• Gateway logging internals — WS log styles, subsystem prefixes, and console capture
• Configuration reference — full diagnostics.* field reference
• openclaw logs — tail Gateway logs over RPC from the CLI