AWS Polly & Transcribe

Local emulators. Real API contracts. Part of Mockingbird.

Stateful emulator of Amazon Polly and Amazon Transcribe for test suites: Polly SynthesizeSpeech and StartSpeechSynthesisStream (HTTP/2 duplex event stream), Transcribe Streaming StartStreamTranscription (HTTP/2 duplex) and Transcribe batch StartTranscriptionJob / GetTranscriptionJob. Chat voice turns run end to end with no audio leaving the machine: Transcribe "hears" what a test scripts, Polly answers with deterministic synthetic audio whose length follows the text.

• Operation coverage: SUPPORT.md
• Proven with the official clients our consumer pins: @aws-sdk/client-polly, @aws-sdk/client-transcribe-streaming, @aws-sdk/client-transcribe (3.1132.0), and the MP3 decoder our backend uses (mpg123-decoder@1.0.3).

#Install

npm install -D @crvouga/mockingbird-service-aws-speech

ESM only. Node >= 22 or Bun >= 1.2. No native dependencies. Serve it with npx mockingbird-aws-speech serve (h2c + HTTP/1.1 on one port), createServer from ./server, or createRuntime with any Fetch server (HTTP/1.1 only).

#Usage

npx mockingbird-aws-speech serve --port 8797
export AWS_ENDPOINT_URL_POLLY=http://127.0.0.1:8797
export AWS_ENDPOINT_URL_TRANSCRIBE_STREAMING=http://127.0.0.1:8797
export AWS_ENDPOINT_URL_TRANSCRIBE=http://127.0.0.1:8797

import { PollyClient, SynthesizeSpeechCommand } from "@aws-sdk/client-polly"
import {
  StartStreamTranscriptionCommand,
  TranscribeStreamingClient,
} from "@aws-sdk/client-transcribe-streaming"
import { createServer } from "@crvouga/mockingbird-service-aws-speech/server"

// Test setup: start the emulator and script what the next voice session "hears":
// partials while audio streams in, then the final.
const emulator = await createServer()
await fetch(`${emulator.url}/__admin/transcripts`, {
  method: "PUT",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({ match: { any: true }, partials: ["I have", "I have a headache"], final: "I have a headache." }),
})

// The AWS SDK clients your application uses; only the endpoint changes.
const config = {
  endpoint: emulator.url,
  region: "us-east-1",
  credentials: { accessKeyId: "fixture", secretAccessKey: "fixture" },
}
const transcribe = new TranscribeStreamingClient(config)
const polly = new PollyClient(config)

// Application code: a voice turn transcribes the member's audio, then speaks the reply.
const { TranscriptResultStream } = await transcribe.send(
  new StartStreamTranscriptionCommand({
    LanguageCode: "en-US",
    MediaEncoding: "pcm",
    MediaSampleRateHertz: 16_000,
    AudioStream: (async function* () {
      yield { AudioEvent: { AudioChunk: new Uint8Array(3_200) } }
    })(),
  }),
)
const heard: string[] = []
for await (const event of TranscriptResultStream ?? []) {
  for (const result of event.TranscriptEvent?.Transcript?.Results ?? [])
    heard.push(result.Alternatives?.[0]?.Transcript ?? "")
}
// heard: ["I have", "I have a headache", "I have a headache."]
const { AudioStream } = await polly.send(
  new SynthesizeSpeechCommand({ VoiceId: "Matthew", OutputFormat: "mp3", Text: "Rest and drink water." }),
)
const audio = await AudioStream?.transformToByteArray() // deterministic MP3 frames

await emulator.close() // after the test

#Protocols

Polly and Transcribe Streaming clients default to NodeHttp2Handler: against an http:// endpoint they speak h2c (cleartext HTTP/2, prior knowledge), and their bidirectional operations need HTTP/2 duplex. Transcribe batch uses HTTP/1.1. mockingbird-aws-speech serve and createServer sniff each connection and serve both on one port (serve --config from another service's CLI, and createRuntime behind a plain Fetch server, speak HTTP/1.1 only).

#Routes

#Transcripts (PUT /__admin/transcripts)

{match?, partials?, final, times?} (or {"transcripts": [...]}). match is {sessionIndex} (0-based count of streaming sessions in the namespace), {jobName} (a batch job), or {any: true}; an exact match wins over any. Unscripted sessions and jobs hear defaultTranscript ("Hello.", PUT /__admin/settings) and count as unscripted (GET /__admin/transcripts → stats).

#Admin (beyond the standard contract)

With --s3-endpoint <url> (transcriptStore in code) a completed job's transcript JSON is also written to that S3 (the stack's s3rver) at OutputBucketName/OutputKey, so the app's own GetObject on TranscriptFileUri finds it.

Fault presets (POST /__admin/faults {"preset": "<name>", "count"?: n}): polly_throttling (429 ThrottlingException), polly_service_failure (500), polly_stream_throttling / polly_stream_validation / polly_stream_quota (an exception event before any audio: our adapter falls back to SynthesizeSpeech), polly_stream_failure (ServiceFailureException after audio), polly_stream_no_close (no StreamClosedEvent), transcribe_bad_request (400), transcribe_limit_exceeded (429), transcribe_service_unavailable (503), transcribe_mid_stream_failure (InternalFailureException after the first partial), transcribe_job_limit (LimitExceededException), transcribe_job_failed (next job FAILED).

#Namespaces

x-mockingbird-namespace, a /__admin/ns/<name> prefix on the endpoint URL, or the SigV4 access key id: PUT /__admin/credentials {"credentials": {"<AWS_ACCESS_KEY_ID>": "<namespace>"}}. The journal records voice, engine, format, character counts and script ids — never text.

#Deliberately not modelled

• Real speech: Polly audio is a tone (PCM) or silent-but-valid MP3 frames (a tone needs a real MP3 encoder); Transcribe never listens — words come only from scripts.
• ogg_vorbis, ogg_opus, mulaw, alaw and speech-mark (json) output, SSML semantics (tags are stripped for length), lexicons, per-voice engine availability.
• Transcribe language identification, speaker labels, custom vocabularies (accepted and echoed, not applied), Call Analytics and Medical variants, batch jobs reading their media from S3.
• SigV4 signatures and event signatures are not verified; the access key id only selects a namespace.

#API

Part of mockingbird.