home-services/tts-gateway

tts-gateway

tts-gateway is the internal gRPC boundary for text-to-speech synthesis. It wraps a VITS voice model (92 Umamusume: Pretty Derby character voices, ported from tmp/reference/uma-tts-api) behind a small protobuf API.

Unlike the other three services, tts-gateway is two deployables:

• tts-gateway (this directory, Go) - the gRPC service itself: Japanese text normalization (open_jtalk CLI) and audio encoding (WAV + ffmpeg AAC transcode). CPU-only, no GPU needed.
• tts-gateway/sidecar (Python/libtorch) - a minimal HTTP service that does only net_g.infer() (the actual VITS forward pass). This exists because the checkpoint's output length is genuinely data-dependent at runtime (predicted phoneme durations), which torch.export/ONNX couldn't trace cleanly - see tmp/reference/uma-tts-api/spike/FINDINGS.md for the full investigation. Needs an Nvidia GPU to be useful; only meaningfully runs on nik-gpu.

Runtime Flow

1. tts-gateway loads .env, configures logging/telemetry, and starts gRPC on GRPC_PORT.
2. TTSService.Synthesize normalizes the request text via the open_jtalk CLI adapter (reproducing uma-tts-api's japanese_cleaners pipeline character-for-character, including its "wrong tokens" quirk - the checkpoint was trained on that exact tokenization), sends the resulting symbol IDs to the inference sidecar over HTTP, and transcodes the returned PCM to AAC via ffmpeg.
3. TTSService.ListSpeakers returns the 92-speaker roster (optionally filtered by a case-insensitive substring), no sidecar call needed.

gRPC API

Contract: proto/tts/v1/tts.proto.

• TTSService.Synthesize - speaker_name, text, optional noise_scale (default 0.37) / noise_scale_w (default 0.46) / length_scale (default 1.3) → audio bytes + mime_type (audio/aac). Unknown speaker → INVALID_ARGUMENT.
• TTSService.ListSpeakers - optional search substring → speaker names.

The server also registers gRPC health checks and reflection.

Configuration

tts-gateway environment variables:

Variable	Default	Description
GRPC_PORT	50053	gRPC listen port
TLS_DIR	empty	Enables mTLS for the gRPC server when set
OTEL_ENDPOINT	empty	OTLP gRPC collector endpoint; empty disables telemetry
LOG_LEVEL	info	debug, info, warn, or error
LOG_FORMAT	json	json or text
OPEN_JTALK_BIN	open_jtalk	Path/name of the open_jtalk CLI binary
OPEN_JTALK_DICT_DIR	empty	Dictionary dir; empty auto-discovers under /usr//var
OPEN_JTALK_VOICE	empty	.htsvoice path; empty auto-discovers under /usr/share/hts-voice
INFERENCE_SIDECAR_ADDR	localhost:50054	host:port of the inference sidecar

sidecar/server.py environment variables:

Variable	Default	Description
CONFIG_PATH	/models/uma.json	Path to the VITS hparams JSON
CHECKPOINT_PATH	/models/G_790000.pth	Path to the model checkpoint
PORT	50054	HTTP listen port

Neither the checkpoint nor the hparams file is committed to git - both must be supplied at runtime (see below). In production, they're baked into a small versioned image (tts-gateway/model/Dockerfile) that a k8s initContainer copies into a shared volume before tts-sidecar starts - see "Model artifact distribution (production)" below. For local/manual testing on nik-gpu, a plain bind mount (as shown below) is simpler.

Model artifact distribution (production)

CI can never build the model image - the checkpoint (~455MB) and hparams aren't committed to git, and nik-gpu's local disk at /data/tts-gateway is the only place they exist. Build and push it manually, directly on nik-gpu (not via docker --context nik-gpu from elsewhere - the build context is gathered client-side, so it would look for /data/tts-gateway on the wrong machine), whenever the checkpoint or config change:

ssh nik-gpu
docker build -f ~/repo/home-service/tts-gateway/model/Dockerfile \
  -t gitea.nik4nao.com/nik/tts-model:g790000-v1 /data/tts-gateway
docker push gitea.nik4nao.com/nik/tts-model:g790000-v1

Bump the tag (v2, v3, ...) rather than overwriting one - non-:latest tags default to imagePullPolicy: IfNotPresent, so a node that already pulled a tag won't re-pull an overwritten one. The k8s Deployment (~/repo/homelab/manifests/home-services/tts-gateway.yaml) references the tag explicitly in its model-init init container.

Running And Testing On nik-gpu

The Go gateway only needs ffmpeg/open_jtalk, which its image already bundles - it can run anywhere. The sidecar needs an actual Nvidia GPU, so the full system is really only testable on nik-gpu.

Option A: pull the images CI already built and pushed

docker --context nik-gpu pull gitea.nik4nao.com/nik/tts-gateway:latest
docker --context nik-gpu pull gitea.nik4nao.com/nik/tts-sidecar:latest

Option B: build from source (use the nik-gpu-sync and nik-gpu-docker-build skills)

# from the repo root, on your local machine
# 1. sync the repo to nik-gpu (nik-gpu-sync skill)
# 2. build both images there (nik-gpu-docker-build skill), e.g.:
docker --context nik-gpu build --platform linux/amd64 \
  -f tts-gateway/Dockerfile -t gitea.nik4nao.com/nik/tts-gateway:latest ~/repo/home-service
docker --context nik-gpu build --platform linux/amd64 \
  -f tts-gateway/sidecar/Dockerfile -t gitea.nik4nao.com/nik/tts-sidecar:latest ~/repo/home-service/tts-gateway/sidecar

Run both containers on nik-gpu

The checkpoint/config aren't baked into the sidecar image - bind-mount them from wherever they live on nik-gpu (e.g. a synced copy of tmp/reference/uma-tts-api/). Both containers use --network host so the gateway can reach the sidecar over localhost:

docker --context nik-gpu run -d --name tts-sidecar --network host --gpus all \
  -v /home/nik/repo/home-service/tmp/reference/uma-tts-api/G_790000.pth:/models/G_790000.pth:ro \
  -v /home/nik/repo/home-service/tmp/reference/uma-tts-api/configs/uma.json:/models/uma.json:ro \
  gitea.nik4nao.com/nik/tts-sidecar:latest

docker --context nik-gpu run -d --name tts-gateway --network host \
  -e GRPC_PORT=50053 -e INFERENCE_SIDECAR_ADDR=localhost:50054 -e LOG_FORMAT=text \
  gitea.nik4nao.com/nik/tts-gateway:latest

Check the sidecar loaded the checkpoint before testing:

docker --context nik-gpu logs tts-sidecar
# expect: "model loaded from /models/G_790000.pth" then "inference sidecar listening on :50054"

Use absolute remote paths in -v, not ~ - with a docker --context pointed at a remote host, ~ still gets expanded by your local shell before the command is sent, not by nik-gpu, so it silently resolves to a path that doesn't exist there and Docker bind-mounts an empty directory instead of the real file.

Smoke test with grpcurl

No local grpcurl needed - run it in a container against the host network:

docker --context nik-gpu run --rm --network host fullstorydev/grpcurl \
  -plaintext -d '{"search":"rice"}' localhost:50053 tts.v1.TTSService/ListSpeakers

docker --context nik-gpu run --rm --network host fullstorydev/grpcurl \
  -plaintext -d '{"speaker_name":"Rice Shower","text":"おはようございます"}' \
  localhost:50053 tts.v1.TTSService/Synthesize > /tmp/synth.json

python3 -c "
import json, base64
data = json.load(open('/tmp/synth.json'))
open('/tmp/synth.m4a', 'wb').write(base64.b64decode(data['audio']))
"
ffprobe /tmp/synth.m4a   # sanity-check duration/codec

Clean up

docker --context nik-gpu rm -f tts-gateway tts-sidecar

Test And Build

go test ./...
go build ./...

Note: go vet/go test/go build don't need ffmpeg/open_jtalk present - the tests that touch those adapters only exercise pure parsing/encoding logic (internal/adapters/secondary/jtalk, internal/adapters/secondary/ffmpeg), never the actual binaries.

Package Map

cmd/gateway/                                  # process entrypoint and wiring
internal/adapters/primary/grpc/               # gRPC service implementation
internal/adapters/secondary/jtalk/            # open_jtalk CLI text normalizer
internal/adapters/secondary/inferencesidecar/ # HTTP client for the Python sidecar
internal/adapters/secondary/ffmpeg/           # WAV + ffmpeg AAC encoder
internal/app/                                 # synthesis orchestration
internal/config/                              # environment loading
internal/core/domain/                         # domain types, symbol table, speaker roster
internal/core/ports/                          # driving and driven interfaces
internal/logger/                              # slog setup
internal/telemetry/                           # OpenTelemetry setup
sidecar/                                      # Python/libtorch inference service (see above)

Limitations

• No app-layer authorization, same as the other three services - keep this internal or protect it with mTLS.
• Concurrency is unbounded on the Go side, but the sidecar's model forward pass is inherently single-threaded per GPU; heavy concurrent load will just queue at the sidecar.
• Model artifact distribution to production is a manually-built-and-pushed versioned image (see above), not something CI can automate - the checkpoint only ever existed on nik-gpu's local disk.
• discord-bot's /speak command calls this service and plays the result in a Discord voice channel (discord-bot/internal/adapters/primary/discord/voice.go).