OpenClawDocs
On this page

On this page

View as MarkdownView this page as plain textOpen in ChatGPTAsk questions about this pageOpen in ClaudeAsk questions about this page

Plugin SDK reference

Plugin runtime media helpers

Speech, media understanding, generation, web search, and the low-level media utilities. Part of the Plugin runtime helpers reference.

FFmpeg command discovery

Official plugin sources use resolveFfmpegBin from the private openclaw/plugin-sdk/media-ffmpeg runtime. It resolves the same trusted system paths as the media runtime and throws an installation hint when FFmpeg is unavailable. This narrow entry point keeps media generation and agent runtimes out of source audio worker startup. It has a JavaScript-only host export; its declarations are excluded from the package. Third-party plugins retain the existing resolveFfmpegBin export from openclaw/plugin-sdk/media-runtime.

The official plugin publication builder emits media-runtime for this private import, including worker entries, so published plugins keep working on supported hosts that predate media-ffmpeg. Both paths use the host's FFmpeg resolver.

Realtime voice playback

Official audio workers use createRealtimeVoiceOutputActivityTracker and isRealtimeVoiceAudioAudible from the private openclaw/plugin-sdk/realtime-voice-playback runtime to avoid loading voice session and agent runtimes from source. The publication builder emits the existing openclaw/plugin-sdk/realtime-voice host import for these helpers, including worker entries, to preserve supported hosts. The source facade has a JavaScript-only host export; its declarations are excluded from the package.

Media and generation namespaces

api.runtime.tts

Text-to-speech synthesis.

typescript
// Standard TTSconst clip = await api.runtime.tts.textToSpeech({  text: "Hello from OpenClaw",  cfg: api.config,}); // Telephony-optimized TTSconst telephonyClip = await api.runtime.tts.textToSpeechTelephony({  text: "Hello from OpenClaw",  cfg: api.config,}); // List available voicesconst voices = await api.runtime.tts.listVoices({  provider: "elevenlabs",  cfg: api.config,});

Uses core tts configuration and provider selection. Returns PCM audio buffer + sample rate. textToSpeechStream is also available for streaming synthesis.

buildSystemPromptHintAsync(config, agentId?, options?) prepares the TTS prompt hint through the state owner and returns Promise<string | undefined>. It is optional on older host interfaces; require an upgrade when absent. See the async prompt preparation recipe.

api.runtime.mediaUnderstanding

Image, audio, and video analysis.

typescript
// Describe an imageconst image = await api.runtime.mediaUnderstanding.describeImageFile({  filePath: "/tmp/inbound-photo.jpg",  cfg: api.config,  agentDir: "/tmp/agent",}); // Prepare a capture limit before installing audio receive listeners.const budget = await api.runtime.mediaUnderstanding.resolveAudioInputBudget({  cfg: api.config,});// budget.enabled is false when audio understanding is disabled. Otherwise,// budget.maxBytes includes the container header and covers the largest fallback. // Transcribe audioconst { text } = await api.runtime.mediaUnderstanding.transcribeAudioFile({  filePath: "/tmp/inbound-audio.ogg",  cfg: api.config,  mime: "audio/ogg", // optional, for when MIME cannot be inferred}); // Describe a videoconst video = await api.runtime.mediaUnderstanding.describeVideoFile({  filePath: "/tmp/inbound-video.mp4",  cfg: api.config,}); // Generic file analysisconst result = await api.runtime.mediaUnderstanding.runFile({  filePath: "/tmp/inbound-file.pdf",  cfg: api.config,}); // Structured image extraction through a specific provider/model.// Include at least one image; text inputs are supplemental context.// receiptImageBuffer is your own image bytes, not an SDK-provided value.const evidence = await api.runtime.mediaUnderstanding.extractStructuredWithModel({  provider: "codex",  model: "gpt-6-astra",  input: [    {      type: "image",      buffer: receiptImageBuffer,      fileName: "receipt.png",      mime: "image/png",    },    { type: "text", text: "Prefer the printed total over handwritten notes." },  ],  instructions: "Extract vendor, total, and searchable tags.",  schemaName: "receipt.evidence",  jsonSchema: {    type: "object",    properties: {      vendor: { type: "string" },      total: { type: "number" },      tags: { type: "array", items: { type: "string" } },    },    required: ["vendor", "total"],  },  cfg: api.config,});

Returns { text: undefined } when no output is produced (e.g. skipped input).

describeImageFileWithModel(...) describes an already-known image through a specific provider/model, bypassing the default active-model resolution that describeImageFile(...) uses.

api.runtime.imageGeneration

Image generation.

typescript
const result = await api.runtime.imageGeneration.generate({  prompt: "A robot painting a sunset",  cfg: api.config,}); const providers = api.runtime.imageGeneration.listProviders({ cfg: api.config });
api.runtime.videoGeneration

Video generation, mirroring the image generation shape.

typescript
const result = await api.runtime.videoGeneration.generate({  prompt: "A drone shot flying over a coastline at sunrise",  cfg: api.config,}); const providers = api.runtime.videoGeneration.listProviders({ cfg: api.config });
api.runtime.musicGeneration

Music generation, mirroring the image generation shape.

typescript
const result = await api.runtime.musicGeneration.generate({  prompt: "An upbeat lo-fi track for a coding session",  cfg: api.config,}); const providers = api.runtime.musicGeneration.listProviders({ cfg: api.config });
api.runtime.webSearch

Web search.

typescript
const providers = api.runtime.webSearch.listProviders({ config: api.config }); const result = await api.runtime.webSearch.search({  config: api.config,  args: { query: "OpenClaw plugin SDK", count: 5 },});

Search callers may supply a synchronous assertCurrent callback with signal to retain their authority through provider preparation. Guarded HTTP requests check it after transport preparation and before each request or redirect. A registered search provider using another transport must call the execution context's assertCurrent before each external side effect, after awaited preparation. The callback belongs to the caller and expires when the search finishes; providers must not replace it or treat its absence as permission.

Host callers can also pass modelSelection: { provider, model, authProfileId? }. This is selection data, not an authorization grant. Model-backed search providers preserve a matching provider's model/profile through their native transport; unrelated search providers keep their own configured selection. An explicit matching credential prevents automatic fallback to another search provider. The agent tool carries this context separately from model-authored arguments.

api.runtime.media

Low-level media utilities.

typescript
const webMedia = await api.runtime.media.loadWebMedia(url);const mime = await api.runtime.media.detectMime({ buffer });const kind = api.runtime.media.mediaKindFromMime("image/jpeg"); // "image"const isVoice = api.runtime.media.isVoiceCompatibleAudio({ fileName: filePath });const metadata = await api.runtime.media.getImageMetadata(buffer);const resized = await api.runtime.media.resizeToJpeg({ buffer, maxSide: 800, quality: 85 });

QR helpers are exported by openclaw/plugin-sdk/media-runtime:

typescript
import { resolvePreferredOpenClawTmpDir } from "openclaw/plugin-sdk/temp-path"; const qr = await import("openclaw/plugin-sdk/media-runtime");const terminalQr = await qr.renderQrTerminal("https://openclaw.ai");const pngQr = await qr.renderQrPngBase64("https://openclaw.ai", {  scale: 6, // 1-12  marginModules: 4, // 0-16});const pngQrDataUrl = await qr.renderQrPngDataUrl("https://openclaw.ai");const tmpRoot = resolvePreferredOpenClawTmpDir();const pngQrFile = await qr.writeQrPngTempFile("https://openclaw.ai", {  tmpRoot,  dirPrefix: "my-plugin-qr-",  fileName: "qr.png",});
Was this useful?