Add TTS model components and inference server

- Implemented core model components in `modules.py` including various convolutional layers and normalization techniques.
- Added transformation functions in `transforms.py` for piecewise rational quadratic transformations.
- Created utility functions in `utils.py` for checkpoint management, logging, and hyperparameter handling.
- Introduced monotonic alignment functionality with Cython optimization in `monotonic_align`.
- Developed a minimal inference server in `server.py` to handle synthesis requests.
- Updated requirements to include necessary dependencies for Cython and scipy.

CLAUDE.md

@@ -19,9 +19,9 @@ ai-gateway ------> Ollama
 ha-gateway
 ```
 
-- **ha-gateway** (port `50051`) — gRPC boundary for Home Assistant. Talks to HA's REST API; implements entity state and light control/discovery and switch discovery; switch control and event streaming are stubbed.
+- **ha-gateway** (port `50051`) — gRPC boundary for Home Assistant. Talks to HA's REST API; implements entity state, light control/discovery, switch control/discovery, and climate (HVAC) control/discovery; also relays SwitchBot Cloud remote commands (`RemoteApp`) when `SWITCHBOT_TOKEN`/`SWITCHBOT_SECRET` are set. Event streaming is stubbed.
 - **ai-gateway** (port `50052`) — gRPC service that turns free-form text into home actions. Calls Ollama for intent extraction, resolves intents against a cached light list from `ha-gateway`, and calls `ha-gateway` to execute approved actions.
-- **discord-bot** — registers `/light`, `/switch`, `/ai` slash commands and calls `ha-gateway`/`ai-gateway` via gRPC clients.
+- **discord-bot** — registers `/light`, `/switch`, `/ac`, `/ai` slash commands and calls `ha-gateway`/`ai-gateway` via gRPC clients.
 
 Each service is a separate Go module (own `go.mod`) joined by `go.work` at the root, plus a `gen` module for shared generated code. Module paths are `gitea.nik4nao.com/nik/home-services/{ha-gateway,ai-gateway,discord-bot,gen}`.
 
@@ -119,8 +119,24 @@ Tests use the standard library `testing` package only (no testify). Mocks are ha
 - Every service reads `TLS_DIR` to enable optional mTLS; when set, the directory must contain `tls.crt`, `tls.key`, and `ca.crt`.
 - `OTEL_ENDPOINT` enables OTLP gRPC traces/metrics; leave empty for local no-op telemetry.
 - `LOG_FORMAT=json` is the production default; `text` is easier to read locally.
+- `ha-gateway` reads `SWITCHBOT_TOKEN`/`SWITCHBOT_SECRET` (optional) to enable SwitchBot Cloud remote commands via `RemoteApp`; leave empty to disable that path.
 - None of the services implement app-layer authorization — they rely on being kept on a trusted internal network or on mTLS. Keep this in mind before adding any endpoint that wasn't previously reachable.
 
+## nik-gpu deployment target
+
+`tts-gateway` (planned in `TTS_GATEWAY_PLAN.md`) is designed to run on `nik-gpu`, a remote Nvidia
+GPU host reachable via `ssh nik-gpu`, managed through the Claude Code skills `nik-gpu-status`
+(read-only check), `nik-gpu-sync` (rsync this repo to `~/repo/home-service/` there), and
+`nik-gpu-docker-build` (build/smoke-test via a `nik-gpu` Docker context).
+
+**Never automatically run installation or other host-system-altering commands on nik-gpu** —
+`apt`/`apt-get`, `pip install` outside a container, Docker daemon config changes, driver/toolkit
+updates, or anything requiring `sudo`. Always print the exact command and ask the user to run it
+themselves (their own terminal, or `! <command>` in a Claude Code session). This applies to the
+bare nik-gpu host specifically; installing packages *inside* a Dockerfile build (e.g.
+`apt-get install open-jtalk` as a build step) is a normal container build action, not a host
+mutation, and is fine to run.
+
 ## CI
 
 `.gitea/workflows/ci.yaml` runs `go vet` and `go test` for all four modules (`gen`, `ai-gateway`, `ha-gateway`, `discord-bot`), then builds and pushes Docker images for the three services on pushes to `main`.

TTS_GATEWAY_PLAN.md

@@ -0,0 +1,242 @@
+# tts-gateway: plan & feasibility
+
+Proposal for turning `tmp/reference/uma-tts-api` (Python/Flask, VITS voice model) into a fourth
+`home-services` module, **`tts-gateway`**, written in Go, following the same hexagonal pattern as
+`ha-gateway`/`ai-gateway`/`discord-bot`, deployed on **nik-gpu** (Nvidia RTX 2080 Super).
+
+See [`tmp/reference/uma-tts-api/FINDINGS.md`](tmp/reference/uma-tts-api/FINDINGS.md) for how the
+reference implementation works today; this document is about porting it.
+
+## TL;DR feasibility
+
+| Piece | Verdict | Why |
+|---|---|---|
+| Serving/API layer (gRPC, config, health, telemetry) | **Straightforward** | Identical shape to the other three services; pure Go, no new tech. |
+| Audio postprocessing (WAV → AAC/Opus) | **Straightforward** | Write PCM/WAV by hand, shell out to `ffmpeg` (already a required system dep). |
+| Model inference (VITS forward pass) | **Feasible, one open risk** | No mature native-Go path to run a `.pth` checkpoint directly. The realistic route is: export the *inference-only* graph to **ONNX** once (offline, in Python), then run it in Go via `onnxruntime-go` with the CUDA execution provider. The 2080 Super (Turing, CC 7.5) is well inside ONNX Runtime's supported GPU range. The risk isn't "can Go do GPU inference" (yes), it's "does this specific checkpoint's graph export cleanly to ONNX" — see [Risk 1](#risk-1-onnx-export-of-the-stochastic-duration-predictor). |
+| Japanese text front-end (`pyopenjtalk` g2p) | **Feasible, one open risk** | No usable native-Go OpenJTalk/mecab g2p implementation exists. Shelling out to the `open_jtalk` CLI (a real C++ binary, not Python) from Go is the likely path, but its default output format needs to be checked against what `pyopenjtalk.g2p()` actually returns — see [Risk 2](#risk-2-text-front-end-parity). |
+| Overall "100% Go, no Python at runtime" | **Likely, not guaranteed** | Contingent on Risks 1 and 2 resolving cleanly. Both have a concrete fallback (below) that still ships a real Go service, just with a small non-Go component for one stage. |
+
+Bottom line: this is buildable and the GPU is not a blocker. The two things that could force a
+hybrid design instead of pure Go are named explicitly below, and Phase 0 exists specifically to
+answer them *before* any Go code is written.
+
+## What "parity" means
+
+The new service should support what `uma-tts-api` supports today:
+
+- Synthesize speech for one of 92 Umamusume speakers given Japanese text, with the same
+  `noise_scale` / `noise_scale_w` / `length_scale` knobs.
+- List/search speakers.
+- Health check.
+
+Not required to carry over: training code, the `monotonic_align` Cython extension (only used by
+the *training* loss, confirmed below — never called from `infer()`), or CPU fallback as a
+first-class target (nik-gpu has a GPU; CPU-only would defeat the point of moving it there).
+
+## Why ONNX Runtime instead of a Go libtorch binding
+
+Two ways exist to run a PyTorch model's math from Go:
+
+1. **cgo bindings to libtorch** (e.g. community "gotorch"-style projects). Rejected: these bind
+   directly to PyTorch's C++ ABI, which is unstable across versions, poorly maintained as Go
+   packages, and would require hand-porting the whole `SynthesizerTrn` forward pass into
+   Go/C++ rather than reusing a serialized graph. High effort, fragile, no real upside here.
+2. **Export to ONNX once, run via `onnxruntime-go`** (e.g. `github.com/yalue/onnxruntime_go`),
+   using Microsoft's prebuilt `onnxruntime-linux-x64-gpu` shared library with the CUDA execution
+   provider. This is the standard, maintained path for "run a PyTorch model from a non-Python
+   language," and it converts the problem from "port a neural net to Go" into "export a graph
+   once, then call a stable C API." **Recommended.**
+
+## Risk 1: ONNX export of the stochastic duration predictor
+
+Checked in the reference code: `configs/uma.json`'s `model` section doesn't set `use_sdp`, and
+`SynthesizerTrn.__init__` defaults `use_sdp=True` (`models/models.py:413`), so this checkpoint
+*does* use the flow-based `StochasticDurationPredictor`, not the simpler deterministic one.
+
+That module's reverse (inference) path chains several `ConvFlow`s, which call into
+`transforms.py`'s `rational_quadratic_spline` / `unconstrained_rational_quadratic_spline`. Those
+functions use boolean-mask indexed assignment (`outputs[outside_interval_mask] = ...`,
+`transforms.py:68-95`) to handle spline boundary conditions. This pattern is a known rough edge
+for `torch.onnx.export`'s tracer — it's data-dependent control flow that doesn't always lower to
+a single static ONNX graph cleanly, unlike the rest of the model (convolutions, attention,
+LayerNorm, ConvTranspose1d) which are all completely standard and export fine.
+
+The custom `torch.searchsorted`-alike in the same file (`transforms.py:47`) is actually
+implemented with plain `torch.sum`/comparison ops rather than the real `torch.searchsorted` op —
+that's good news, since it sidesteps an op with historically inconsistent ONNX opset support.
+
+**What this means practically:** export is very likely possible (nothing here is fundamentally
+unexportable), but it may need one of: PyTorch's newer `dynamo`-based exporter instead of the
+legacy tracer, a higher opset version, or a small rewrite of the masked-assignment lines to
+`torch.where(...)` equivalents (behavior-preserving, just export-friendlier). This is a
+half-day-to-a-few-days spike, not a redesign, but it's not knowable in advance without trying it
+— hence Phase 0.
+
+**Fallback if export genuinely can't be made to match:** run inference in a small, minimal
+Python (or C++/libtorch) sidecar container that does *only* `net_g.infer(...)` — no Flask, no
+Japanese text handling — and have the Go `tts-gateway` call it over a loopback gRPC/HTTP call
+from its `onnxengine`-equivalent secondary adapter. The service is still a real Go hexagonal
+service at the architecture and API level; only the tensor math lives elsewhere, hidden behind a
+port like any other outbound dependency (same shape as how `ai-gateway` calls out to Ollama).
+
+## Risk 2: text front-end parity
+
+`text/cleaners.py::japanese_cleaners` calls `pyopenjtalk.g2p(text, kana=False)`, which wraps the
+OpenJTalk C++ library (NAIST Japanese dictionary + internal mecab) and returns a
+space-stripped phoneme string. There is no maintained native-Go equivalent of this — it's not
+just "run mecab," it's OpenJTalk's full NJD → JPCommon phoneme pipeline.
+
+Two realistic options, in order of preference:
+
+1. **Shell out to the `open_jtalk` CLI binary** from Go (`os/exec`), which is a real,
+   independently-installable C++ binary (not Python) — same idea as this repo already shelling
+   out to `ffmpeg`. Needs verification that the CLI's phoneme/label output can be mapped 1:1 to
+   what `pyopenjtalk.g2p(..., kana=False)` produces (the CLI's default output is full
+   HTS-style context labels, not the bare phoneme string `pyopenjtalk` returns — extracting the
+   phoneme string from labels is a known, small parsing task, but it needs to be checked against
+   real output before relying on it).
+2. **A tiny, persistent Python process that does only text normalization** (just
+   `cleaners.py` + `pyopenjtalk`, no `torch`, no Flask) called over loopback — smaller and lower
+   risk than option 1 if the CLI mapping turns out to be awkward, but reintroduces a Python
+   runtime dependency.
+
+The `unidecode` call in the same cleaner only ever runs on the small set of punctuation
+characters matched by `_japanese_marks` (`text/cleaners.py:9`), not arbitrary Unicode — that part
+is trivial to reimplement as a small Go lookup table, no library needed. The symbol vocabulary
+(`text/symbols.py`) is a straight port to a Go slice/string constant.
+
+## Proposed architecture
+
+New module `tts-gateway`, added to `go.work`, mirroring `ha-gateway`'s layout:
+
+```text
+tts-gateway/
+  cmd/gateway/                       entrypoint: load .env, wire adapters, start gRPC
+  internal/core/domain/              Speaker, SynthesisParams, AudioClip
+  internal/core/ports/driven/        TTSEngine (Synthesize), TextNormalizer (Normalize),
+                                      AudioEncoder (Encode)
+  internal/core/ports/driving/       SynthesizeUseCase-shaped interface for the gRPC adapter
+  internal/app/                      orchestration: normalize -> engine.Synthesize -> encode
+  internal/adapters/primary/grpc/    tts.v1.TTSService server (Synthesize, ListSpeakers)
+  internal/adapters/secondary/onnxengine/   onnxruntime-go + CUDA EP, loads the exported .onnx
+  internal/adapters/secondary/jtalk/        open_jtalk CLI subprocess adapter (or sidecar client)
+  internal/adapters/secondary/ffmpeg/       WAV write + ffmpeg transcode to AAC/Opus
+  internal/config/, internal/logger/, internal/telemetry/   copied pattern from ha-gateway
+```
+
+New proto package `proto/tts/v1/tts.proto` (buf module, same convention as `proto/ha`,
+`proto/ai`):
+
+```protobuf
+service TTSService {
+  rpc Synthesize(SynthesizeRequest) returns (SynthesizeResponse);
+  rpc ListSpeakers(ListSpeakersRequest) returns (ListSpeakersResponse);
+}
+```
+
+Suggested port: `50053` (next free after `50051`/`50052`). Same `.env` conventions as the other
+three services: `TLS_DIR` for optional mTLS, `OTEL_ENDPOINT`, `LOG_FORMAT`, plus new
+`ONNX_MODEL_PATH`, `CUDA_DEVICE_ID`, `OPEN_JTALK_BIN` (or sidecar address).
+
+## Phased plan
+
+**Phase 0 — Spike & de-risk (Python, inside `tmp/reference/uma-tts-api`, throwaway code)**
+Export `net_g`'s inference path to ONNX and run it standalone with the `onnxruntime-gpu` Python
+package against a CUDA GPU; diff the resulting audio against `app.py`'s current output for a
+handful of speaker/text/parameter combinations. Separately, run the `open_jtalk` CLI on the same
+sample texts and check whether its output can be turned into the same phoneme string
+`pyopenjtalk.g2p()` produces. **This phase's outcome decides whether Phases 1+ build "full Go +
+ONNX" or "Go gateway + inference/text sidecar."** Don't start the Go work before this answers
+both questions — everything downstream depends on it.
+
+**Phase 1 — Proto + service skeleton**
+Add `proto/tts/v1/tts.proto`, `buf generate`, scaffold the `tts-gateway` module (`go.mod`,
+`go.work` entry), hexagonal skeleton, health check + reflection, config/logger/telemetry copied
+from `ha-gateway`. No inference yet — `Synthesize` can return a canned tone to prove the
+plumbing.
+
+**Phase 2 — Text front-end adapter**
+Port the symbol table and punctuation handling; wire the g2p adapter chosen in Phase 0.
+
+**Phase 3 — Inference adapter**
+Wire `onnxruntime-go` with the CUDA execution provider, load the Phase-0-exported `.onnx` file,
+implement `TTSEngine.Synthesize`. Validate against the Phase 0 reference outputs — expect small
+numeric drift between PyTorch-CPU/GPU and ONNX Runtime-GPU (different kernels), so compare on
+waveform length / gross spectral similarity rather than bit-exact equality, consistent with this
+repo's hand-written-mock testing convention rather than golden-byte comparison.
+
+**Phase 4 — Audio + gRPC wiring**
+WAV writer, `ffmpeg` transcode adapter, full `Synthesize` RPC wiring, speaker-not-found →
+`INVALID_ARGUMENT` (mirroring `app.py`'s current 400 behavior), `ListSpeakers` RPC.
+
+**Phase 5 — Containerize & deploy to nik-gpu**
+See [GPU/deployment notes](#gpu-and-deployment-notes) below — this phase needs a Dockerfile that
+diverges from the other three services' pattern, plus a decision on how the model artifact
+reaches the container.
+
+**Phase 6 — Client integration (optional, separate follow-up)**
+A `/speak` (or similar) Discord command in `discord-bot` calling this gateway, following the
+same secondary-adapter + gRPC-client pattern already used for its `ha-gateway`/`ai-gateway`
+clients. Not assumed in scope here — a deliberate follow-on decision once the service itself
+works.
+
+## GPU and deployment notes
+
+This would be the first GPU-bound, host-specific service in the repo, which breaks an assumption
+the other three quietly rely on (any image can run on any host):
+
+- **Dockerfile can't reuse the existing pattern.** `ha-gateway`/`ai-gateway`/`discord-bot` all
+  build with `CGO_ENABLED=0` onto `gcr.io/distroless/static:nonroot` — deliberately tiny, fully
+  static, portable. `onnxruntime-go` needs cgo (it links against `libonnxruntime.so`), and the
+  CUDA execution provider needs the CUDA/cuDNN runtime present in the image. This service needs
+  its own Dockerfile based on something like `nvidia/cuda:<version>-runtime` (or a slim CUDA
+  runtime variant), with `CGO_ENABLED=1`, `ffmpeg`, and the matching `onnxruntime-linux-x64-gpu`
+  release unpacked into it.
+- **Version matching matters.** `onnxruntime-gpu` prebuilt releases are pinned to specific
+  CUDA/cuDNN major versions. Check nik-gpu's installed Nvidia driver/CUDA version *before*
+  picking an ONNX Runtime release, not after — this is the one place a Turing-generation card
+  (CC 7.5, i.e. no Ampere-only op requirements) plus older driver could bite, not because the
+  2080 Super is unsupported, but because of a driver/toolkit version mismatch.
+- **`--gpus all` / nvidia-container-toolkit required.** nik-gpu needs the Nvidia Container
+  Toolkit installed and Docker's nvidia runtime configured; the compose file needs a
+  `deploy.resources.reservations.devices` (or `--gpus`) entry, unlike any existing
+  `docker-compose.yml` in this repo.
+- **Model artifact distribution.** The exported `.onnx` file (likely similar order of magnitude
+  to the current 455 MB `.pth`) shouldn't be baked into a git-tracked Dockerfile context. Decide
+  between a build-time download step, a bind-mounted volume on nik-gpu, or an artifact registry
+  — this is a real decision to make in Phase 5, not a detail to skip.
+- **CI implications.** `.gitea/workflows/ci.yaml` currently builds/pushes all three service
+  images generically. This service's image is only meaningful on a CUDA host — worth deciding
+  whether it belongs in that same generic build/push step at all, or gets its own workflow that
+  only runs when explicitly triggered, since a successful build here says nothing about whether
+  it actually runs correctly on nik-gpu's GPU.
+
+## Working conventions on nik-gpu
+
+Any command that installs or otherwise changes the nik-gpu **host** itself (apt packages, a
+Python/pip install outside a container, Docker daemon config, driver/CUDA toolkit updates, or
+anything needing `sudo`) must never be run automatically — print the exact command and ask the
+user to run it themselves. This mirrors the hard rule already built into the `nik-gpu-sync` and
+`nik-gpu-docker-build` skills, restated here because Phases 0 and 5 are the two points in this
+plan most likely to want something installed on the host directly (e.g. if the Phase 0 spike is
+run bare-metal instead of inside a container). Prefer pushing installs into a Dockerfile/build
+step wherever possible (`open-jtalk`, `ffmpeg`, `onnxruntime-gpu`, CUDA runtime libs) — that keeps
+nik-gpu's host state untouched and the setup reproducible from the Dockerfile alone.
+
+## Open questions (product/design, not technical blockers)
+
+- **Output codec:** keep AAC (parity with today) or switch to Opus? If the eventual consumer is
+  `discord-bot`, Opus is Discord's native voice codec — worth deciding once Phase 6 is actually
+  on the table, not before.
+- **Concurrency:** the reference implementation serializes all inference behind one lock. Same
+  starting point is reasonable here — revisit only if real latency/throughput needs show up.
+- **Auth:** consistent with every other service in this repo, no app-layer authorization —
+  acceptable only if nik-gpu stays on the same trusted internal network (or gets mTLS via
+  `TLS_DIR`, matching the other three).
+
+## Non-goals
+
+- No training/fine-tuning support — this is inference-only, same as the reference.
+- No multi-GPU or batched-request scheduling in the first version.
+- No change to the voice roster or model weights — same 92 speakers, same checkpoint.

gen/tts/v1/tts.pb.go

@@ -0,0 +1,325 @@
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// versions:
+// 	protoc-gen-go v1.36.11
+// 	protoc        (unknown)
+// source: tts/v1/tts.proto
+
+package ttsv1
+
+import (
+	protoreflect "google.golang.org/protobuf/reflect/protoreflect"
+	protoimpl "google.golang.org/protobuf/runtime/protoimpl"
+	reflect "reflect"
+	sync "sync"
+	unsafe "unsafe"
+)
+
+const (
+	// Verify that this generated code is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
+	// Verify that runtime/protoimpl is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
+)
+
+// optional fields let the gateway apply its own defaults (matching the
+// reference uma-tts-api's noise_scale=0.37, noise_scale_w=0.46,
+// length_scale=1.3) when a client omits them.
+type SynthesizeRequest struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	SpeakerName   string                 `protobuf:"bytes,1,opt,name=speaker_name,json=speakerName,proto3" json:"speaker_name,omitempty"`
+	Text          string                 `protobuf:"bytes,2,opt,name=text,proto3" json:"text,omitempty"`
+	NoiseScale    *float32               `protobuf:"fixed32,3,opt,name=noise_scale,json=noiseScale,proto3,oneof" json:"noise_scale,omitempty"`
+	NoiseScaleW   *float32               `protobuf:"fixed32,4,opt,name=noise_scale_w,json=noiseScaleW,proto3,oneof" json:"noise_scale_w,omitempty"`
+	LengthScale   *float32               `protobuf:"fixed32,5,opt,name=length_scale,json=lengthScale,proto3,oneof" json:"length_scale,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *SynthesizeRequest) Reset() {
+	*x = SynthesizeRequest{}
+	mi := &file_tts_v1_tts_proto_msgTypes[0]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *SynthesizeRequest) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*SynthesizeRequest) ProtoMessage() {}
+
+func (x *SynthesizeRequest) ProtoReflect() protoreflect.Message {
+	mi := &file_tts_v1_tts_proto_msgTypes[0]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use SynthesizeRequest.ProtoReflect.Descriptor instead.
+func (*SynthesizeRequest) Descriptor() ([]byte, []int) {
+	return file_tts_v1_tts_proto_rawDescGZIP(), []int{0}
+}
+
+func (x *SynthesizeRequest) GetSpeakerName() string {
+	if x != nil {
+		return x.SpeakerName
+	}
+	return ""
+}
+
+func (x *SynthesizeRequest) GetText() string {
+	if x != nil {
+		return x.Text
+	}
+	return ""
+}
+
+func (x *SynthesizeRequest) GetNoiseScale() float32 {
+	if x != nil && x.NoiseScale != nil {
+		return *x.NoiseScale
+	}
+	return 0
+}
+
+func (x *SynthesizeRequest) GetNoiseScaleW() float32 {
+	if x != nil && x.NoiseScaleW != nil {
+		return *x.NoiseScaleW
+	}
+	return 0
+}
+
+func (x *SynthesizeRequest) GetLengthScale() float32 {
+	if x != nil && x.LengthScale != nil {
+		return *x.LengthScale
+	}
+	return 0
+}
+
+type SynthesizeResponse struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	Audio         []byte                 `protobuf:"bytes,1,opt,name=audio,proto3" json:"audio,omitempty"`
+	MimeType      string                 `protobuf:"bytes,2,opt,name=mime_type,json=mimeType,proto3" json:"mime_type,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *SynthesizeResponse) Reset() {
+	*x = SynthesizeResponse{}
+	mi := &file_tts_v1_tts_proto_msgTypes[1]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *SynthesizeResponse) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*SynthesizeResponse) ProtoMessage() {}
+
+func (x *SynthesizeResponse) ProtoReflect() protoreflect.Message {
+	mi := &file_tts_v1_tts_proto_msgTypes[1]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use SynthesizeResponse.ProtoReflect.Descriptor instead.
+func (*SynthesizeResponse) Descriptor() ([]byte, []int) {
+	return file_tts_v1_tts_proto_rawDescGZIP(), []int{1}
+}
+
+func (x *SynthesizeResponse) GetAudio() []byte {
+	if x != nil {
+		return x.Audio
+	}
+	return nil
+}
+
+func (x *SynthesizeResponse) GetMimeType() string {
+	if x != nil {
+		return x.MimeType
+	}
+	return ""
+}
+
+type ListSpeakersRequest struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	Search        string                 `protobuf:"bytes,1,opt,name=search,proto3" json:"search,omitempty"` // case-insensitive substring filter; empty returns all speakers
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *ListSpeakersRequest) Reset() {
+	*x = ListSpeakersRequest{}
+	mi := &file_tts_v1_tts_proto_msgTypes[2]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ListSpeakersRequest) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ListSpeakersRequest) ProtoMessage() {}
+
+func (x *ListSpeakersRequest) ProtoReflect() protoreflect.Message {
+	mi := &file_tts_v1_tts_proto_msgTypes[2]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ListSpeakersRequest.ProtoReflect.Descriptor instead.
+func (*ListSpeakersRequest) Descriptor() ([]byte, []int) {
+	return file_tts_v1_tts_proto_rawDescGZIP(), []int{2}
+}
+
+func (x *ListSpeakersRequest) GetSearch() string {
+	if x != nil {
+		return x.Search
+	}
+	return ""
+}
+
+type ListSpeakersResponse struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	SpeakerNames  []string               `protobuf:"bytes,1,rep,name=speaker_names,json=speakerNames,proto3" json:"speaker_names,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *ListSpeakersResponse) Reset() {
+	*x = ListSpeakersResponse{}
+	mi := &file_tts_v1_tts_proto_msgTypes[3]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ListSpeakersResponse) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ListSpeakersResponse) ProtoMessage() {}
+
+func (x *ListSpeakersResponse) ProtoReflect() protoreflect.Message {
+	mi := &file_tts_v1_tts_proto_msgTypes[3]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ListSpeakersResponse.ProtoReflect.Descriptor instead.
+func (*ListSpeakersResponse) Descriptor() ([]byte, []int) {
+	return file_tts_v1_tts_proto_rawDescGZIP(), []int{3}
+}
+
+func (x *ListSpeakersResponse) GetSpeakerNames() []string {
+	if x != nil {
+		return x.SpeakerNames
+	}
+	return nil
+}
+
+var File_tts_v1_tts_proto protoreflect.FileDescriptor
+
+const file_tts_v1_tts_proto_rawDesc = "" +
+	"\n" +
+	"\x10tts/v1/tts.proto\x12\x06tts.v1\"\xf4\x01\n" +
+	"\x11SynthesizeRequest\x12!\n" +
+	"\fspeaker_name\x18\x01 \x01(\tR\vspeakerName\x12\x12\n" +
+	"\x04text\x18\x02 \x01(\tR\x04text\x12$\n" +
+	"\vnoise_scale\x18\x03 \x01(\x02H\x00R\n" +
+	"noiseScale\x88\x01\x01\x12'\n" +
+	"\rnoise_scale_w\x18\x04 \x01(\x02H\x01R\vnoiseScaleW\x88\x01\x01\x12&\n" +
+	"\flength_scale\x18\x05 \x01(\x02H\x02R\vlengthScale\x88\x01\x01B\x0e\n" +
+	"\f_noise_scaleB\x10\n" +
+	"\x0e_noise_scale_wB\x0f\n" +
+	"\r_length_scale\"G\n" +
+	"\x12SynthesizeResponse\x12\x14\n" +
+	"\x05audio\x18\x01 \x01(\fR\x05audio\x12\x1b\n" +
+	"\tmime_type\x18\x02 \x01(\tR\bmimeType\"-\n" +
+	"\x13ListSpeakersRequest\x12\x16\n" +
+	"\x06search\x18\x01 \x01(\tR\x06search\";\n" +
+	"\x14ListSpeakersResponse\x12#\n" +
+	"\rspeaker_names\x18\x01 \x03(\tR\fspeakerNames2\x9c\x01\n" +
+	"\n" +
+	"TTSService\x12C\n" +
+	"\n" +
+	"Synthesize\x12\x19.tts.v1.SynthesizeRequest\x1a\x1a.tts.v1.SynthesizeResponse\x12I\n" +
+	"\fListSpeakers\x12\x1b.tts.v1.ListSpeakersRequest\x1a\x1c.tts.v1.ListSpeakersResponseB6Z4gitea.nik4nao.com/nik/home-services/gen/tts/v1;ttsv1b\x06proto3"
+
+var (
+	file_tts_v1_tts_proto_rawDescOnce sync.Once
+	file_tts_v1_tts_proto_rawDescData []byte
+)
+
+func file_tts_v1_tts_proto_rawDescGZIP() []byte {
+	file_tts_v1_tts_proto_rawDescOnce.Do(func() {
+		file_tts_v1_tts_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_tts_v1_tts_proto_rawDesc), len(file_tts_v1_tts_proto_rawDesc)))
+	})
+	return file_tts_v1_tts_proto_rawDescData
+}
+
+var file_tts_v1_tts_proto_msgTypes = make([]protoimpl.MessageInfo, 4)
+var file_tts_v1_tts_proto_goTypes = []any{
+	(*SynthesizeRequest)(nil),    // 0: tts.v1.SynthesizeRequest
+	(*SynthesizeResponse)(nil),   // 1: tts.v1.SynthesizeResponse
+	(*ListSpeakersRequest)(nil),  // 2: tts.v1.ListSpeakersRequest
+	(*ListSpeakersResponse)(nil), // 3: tts.v1.ListSpeakersResponse
+}
+var file_tts_v1_tts_proto_depIdxs = []int32{
+	0, // 0: tts.v1.TTSService.Synthesize:input_type -> tts.v1.SynthesizeRequest
+	2, // 1: tts.v1.TTSService.ListSpeakers:input_type -> tts.v1.ListSpeakersRequest
+	1, // 2: tts.v1.TTSService.Synthesize:output_type -> tts.v1.SynthesizeResponse
+	3, // 3: tts.v1.TTSService.ListSpeakers:output_type -> tts.v1.ListSpeakersResponse
+	2, // [2:4] is the sub-list for method output_type
+	0, // [0:2] is the sub-list for method input_type
+	0, // [0:0] is the sub-list for extension type_name
+	0, // [0:0] is the sub-list for extension extendee
+	0, // [0:0] is the sub-list for field type_name
+}
+
+func init() { file_tts_v1_tts_proto_init() }
+func file_tts_v1_tts_proto_init() {
+	if File_tts_v1_tts_proto != nil {
+		return
+	}
+	file_tts_v1_tts_proto_msgTypes[0].OneofWrappers = []any{}
+	type x struct{}
+	out := protoimpl.TypeBuilder{
+		File: protoimpl.DescBuilder{
+			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
+			RawDescriptor: unsafe.Slice(unsafe.StringData(file_tts_v1_tts_proto_rawDesc), len(file_tts_v1_tts_proto_rawDesc)),
+			NumEnums:      0,
+			NumMessages:   4,
+			NumExtensions: 0,
+			NumServices:   1,
+		},
+		GoTypes:           file_tts_v1_tts_proto_goTypes,
+		DependencyIndexes: file_tts_v1_tts_proto_depIdxs,
+		MessageInfos:      file_tts_v1_tts_proto_msgTypes,
+	}.Build()
+	File_tts_v1_tts_proto = out.File
+	file_tts_v1_tts_proto_goTypes = nil
+	file_tts_v1_tts_proto_depIdxs = nil
+}

gen/tts/v1/tts_grpc.pb.go

@@ -0,0 +1,159 @@
+// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
+// versions:
+// - protoc-gen-go-grpc v1.6.2
+// - protoc             (unknown)
+// source: tts/v1/tts.proto
+
+package ttsv1
+
+import (
+	context "context"
+	grpc "google.golang.org/grpc"
+	codes "google.golang.org/grpc/codes"
+	status "google.golang.org/grpc/status"
+)
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the grpc package it is being compiled against.
+// Requires gRPC-Go v1.64.0 or later.
+const _ = grpc.SupportPackageIsVersion9
+
+const (
+	TTSService_Synthesize_FullMethodName   = "/tts.v1.TTSService/Synthesize"
+	TTSService_ListSpeakers_FullMethodName = "/tts.v1.TTSService/ListSpeakers"
+)
+
+// TTSServiceClient is the client API for TTSService service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type TTSServiceClient interface {
+	Synthesize(ctx context.Context, in *SynthesizeRequest, opts ...grpc.CallOption) (*SynthesizeResponse, error)
+	ListSpeakers(ctx context.Context, in *ListSpeakersRequest, opts ...grpc.CallOption) (*ListSpeakersResponse, error)
+}
+
+type tTSServiceClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewTTSServiceClient(cc grpc.ClientConnInterface) TTSServiceClient {
+	return &tTSServiceClient{cc}
+}
+
+func (c *tTSServiceClient) Synthesize(ctx context.Context, in *SynthesizeRequest, opts ...grpc.CallOption) (*SynthesizeResponse, error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	out := new(SynthesizeResponse)
+	err := c.cc.Invoke(ctx, TTSService_Synthesize_FullMethodName, in, out, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+func (c *tTSServiceClient) ListSpeakers(ctx context.Context, in *ListSpeakersRequest, opts ...grpc.CallOption) (*ListSpeakersResponse, error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	out := new(ListSpeakersResponse)
+	err := c.cc.Invoke(ctx, TTSService_ListSpeakers_FullMethodName, in, out, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// TTSServiceServer is the server API for TTSService service.
+// All implementations must embed UnimplementedTTSServiceServer
+// for forward compatibility.
+type TTSServiceServer interface {
+	Synthesize(context.Context, *SynthesizeRequest) (*SynthesizeResponse, error)
+	ListSpeakers(context.Context, *ListSpeakersRequest) (*ListSpeakersResponse, error)
+	mustEmbedUnimplementedTTSServiceServer()
+}
+
+// UnimplementedTTSServiceServer must be embedded to have
+// forward compatible implementations.
+//
+// NOTE: this should be embedded by value instead of pointer to avoid a nil
+// pointer dereference when methods are called.
+type UnimplementedTTSServiceServer struct{}
+
+func (UnimplementedTTSServiceServer) Synthesize(context.Context, *SynthesizeRequest) (*SynthesizeResponse, error) {
+	return nil, status.Error(codes.Unimplemented, "method Synthesize not implemented")
+}
+func (UnimplementedTTSServiceServer) ListSpeakers(context.Context, *ListSpeakersRequest) (*ListSpeakersResponse, error) {
+	return nil, status.Error(codes.Unimplemented, "method ListSpeakers not implemented")
+}
+func (UnimplementedTTSServiceServer) mustEmbedUnimplementedTTSServiceServer() {}
+func (UnimplementedTTSServiceServer) testEmbeddedByValue()                    {}
+
+// UnsafeTTSServiceServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to TTSServiceServer will
+// result in compilation errors.
+type UnsafeTTSServiceServer interface {
+	mustEmbedUnimplementedTTSServiceServer()
+}
+
+func RegisterTTSServiceServer(s grpc.ServiceRegistrar, srv TTSServiceServer) {
+	// If the following call panics, it indicates UnimplementedTTSServiceServer was
+	// embedded by pointer and is nil.  This will cause panics if an
+	// unimplemented method is ever invoked, so we test this at initialization
+	// time to prevent it from happening at runtime later due to I/O.
+	if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
+		t.testEmbeddedByValue()
+	}
+	s.RegisterService(&TTSService_ServiceDesc, srv)
+}
+
+func _TTSService_Synthesize_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(SynthesizeRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(TTSServiceServer).Synthesize(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: TTSService_Synthesize_FullMethodName,
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(TTSServiceServer).Synthesize(ctx, req.(*SynthesizeRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+func _TTSService_ListSpeakers_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(ListSpeakersRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(TTSServiceServer).ListSpeakers(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: TTSService_ListSpeakers_FullMethodName,
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(TTSServiceServer).ListSpeakers(ctx, req.(*ListSpeakersRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// TTSService_ServiceDesc is the grpc.ServiceDesc for TTSService service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var TTSService_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "tts.v1.TTSService",
+	HandlerType: (*TTSServiceServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "Synthesize",
+			Handler:    _TTSService_Synthesize_Handler,
+		},
+		{
+			MethodName: "ListSpeakers",
+			Handler:    _TTSService_ListSpeakers_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "tts/v1/tts.proto",
+}

go.work

@@ -5,4 +5,5 @@ use (
 	./discord-bot
 	./gen
 	./ha-gateway
+	./tts-gateway
 )

proto/tts/v1/tts.proto

@@ -0,0 +1,32 @@
+syntax = "proto3";
+package tts.v1;
+option go_package = "gitea.nik4nao.com/nik/home-services/gen/tts/v1;ttsv1";
+
+service TTSService {
+  rpc Synthesize(SynthesizeRequest)     returns (SynthesizeResponse);
+  rpc ListSpeakers(ListSpeakersRequest) returns (ListSpeakersResponse);
+}
+
+// optional fields let the gateway apply its own defaults (matching the
+// reference uma-tts-api's noise_scale=0.37, noise_scale_w=0.46,
+// length_scale=1.3) when a client omits them.
+message SynthesizeRequest {
+  string          speaker_name  = 1;
+  string          text          = 2;
+  optional float  noise_scale   = 3;
+  optional float  noise_scale_w = 4;
+  optional float  length_scale  = 5;
+}
+
+message SynthesizeResponse {
+  bytes  audio     = 1;
+  string mime_type = 2;
+}
+
+message ListSpeakersRequest {
+  string search = 1; // case-insensitive substring filter; empty returns all speakers
+}
+
+message ListSpeakersResponse {
+  repeated string speaker_names = 1;
+}

tts-gateway/.env.example

@@ -0,0 +1,9 @@
+GRPC_PORT=50053
+TLS_DIR=
+OTEL_ENDPOINT=
+LOG_LEVEL=info
+LOG_FORMAT=text
+OPEN_JTALK_BIN=open_jtalk
+OPEN_JTALK_DICT_DIR=
+OPEN_JTALK_VOICE=
+INFERENCE_SIDECAR_ADDR=localhost:50054

tts-gateway/Dockerfile.dev

@@ -0,0 +1,32 @@
+# Dev/smoke-test image for tts-gateway (Phase 4 of TTS_GATEWAY_PLAN.md).
+# NOT the polished production image - Phase 5 (deferred) decides the final
+# base image/multi-stage layout. This one just needs to prove the pipeline
+# end-to-end: unlike ha-gateway/ai-gateway/discord-bot, tts-gateway needs
+# `ffmpeg` and the `open_jtalk` CLI (+ dictionary + voice) present at
+# runtime, which rules out a distroless base for now.
+FROM golang:1.26-bookworm AS builder
+WORKDIR /workspace
+
+COPY go.work go.work.sum ./
+COPY gen/ ./gen/
+COPY ai-gateway/ ./ai-gateway/
+COPY ha-gateway/ ./ha-gateway/
+COPY discord-bot/ ./discord-bot/
+COPY tts-gateway/ ./tts-gateway/
+
+WORKDIR /workspace/tts-gateway
+RUN go mod download
+RUN CGO_ENABLED=0 GOOS=linux go build -o /tts-gateway ./cmd/gateway
+
+FROM ubuntu:22.04
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    ffmpeg \
+    open-jtalk \
+    open-jtalk-mecab-naist-jdic \
+    hts-voice-nitech-jp-atr503-m001 \
+    ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=builder /tts-gateway /tts-gateway
+EXPOSE 50053
+ENTRYPOINT ["/tts-gateway"]

tts-gateway/cmd/gateway/main.go

@@ -0,0 +1,159 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"net"
+	"os"
+	"os/signal"
+	"path/filepath"
+	"syscall"
+
+	"github.com/joho/godotenv"
+	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/credentials"
+	"google.golang.org/grpc/health"
+	grpc_health_v1 "google.golang.org/grpc/health/grpc_health_v1"
+	"google.golang.org/grpc/reflection"
+
+	ttsv1 "gitea.nik4nao.com/nik/home-services/gen/tts/v1"
+	grpcadapter "gitea.nik4nao.com/nik/home-services/tts-gateway/internal/adapters/primary/grpc"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/adapters/secondary/ffmpeg"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/adapters/secondary/inferencesidecar"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/adapters/secondary/jtalk"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/app"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/config"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/logger"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/telemetry"
+)
+
+// MEMO: auth is not implemented - see ha-gateway/cmd/gateway/main.go's memo
+// for the same options (shared API key vs mTLS) before exposing this
+// service to any untrusted network.
+
+// version is set at build time via -ldflags "-X main.version=<tag>".
+var version = "dev"
+
+func main() {
+	_ = godotenv.Load()
+
+	cfg, err := config.Load()
+	if err != nil {
+		os.Stderr.WriteString("config error: " + err.Error() + "\n")
+		os.Exit(1)
+	}
+
+	log := logger.New(cfg.LogFormat, cfg.LogLevel)
+	log.Info("starting tts-gateway",
+		"version", version,
+		"grpc_port", cfg.GRPCPort,
+		"tls_dir", cfg.TLSDir,
+		"otel_endpoint", cfg.OTELEndpoint,
+		"log_level", cfg.LogLevel,
+		"log_format", cfg.LogFormat,
+		"inference_sidecar_addr", cfg.InferenceSidecarAddr,
+	)
+
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
+	defer stop()
+	ctx = logger.WithLogger(ctx, log)
+
+	shutdown, err := telemetry.Setup(ctx, "tts-gateway", version, cfg)
+	if err != nil {
+		log.Error("telemetry setup failed", "err", err)
+		os.Exit(1)
+	}
+	if cfg.OTELEndpoint != "" {
+		log.Info("telemetry enabled", "endpoint", cfg.OTELEndpoint)
+	} else {
+		log.Debug("telemetry disabled")
+	}
+
+	normalizer, err := jtalk.NewClient(cfg)
+	if err != nil {
+		log.Error("open_jtalk client setup failed", "err", err)
+		os.Exit(1)
+	}
+	engine := inferencesidecar.NewClient(cfg.InferenceSidecarAddr)
+	encoder := ffmpeg.NewEncoder()
+
+	ttsApp := app.NewTTSApp(normalizer, engine, encoder)
+
+	serverOpts := []grpc.ServerOption{
+		grpc.StatsHandler(otelgrpc.NewServerHandler()),
+		grpc.ChainUnaryInterceptor(grpcadapter.LoggingUnaryInterceptor(log)),
+	}
+	if cfg.TLSDir != "" {
+		creds, err := loadServerCredentials(cfg.TLSDir)
+		if err != nil {
+			log.Error("load mTLS credentials failed", "tls_dir", cfg.TLSDir, "err", err)
+			os.Exit(1)
+		}
+		serverOpts = append(serverOpts, grpc.Creds(creds))
+		log.Info("mTLS enabled", "tls_dir", cfg.TLSDir)
+	} else {
+		log.Info("mTLS disabled")
+	}
+
+	srv := grpc.NewServer(serverOpts...)
+	healthSrv := health.NewServer()
+	healthSrv.SetServingStatus("", grpc_health_v1.HealthCheckResponse_SERVING)
+
+	ttsv1.RegisterTTSServiceServer(srv, grpcadapter.NewTTSGRPC(ttsApp))
+	grpc_health_v1.RegisterHealthServer(srv, healthSrv)
+	reflection.Register(srv)
+
+	lis, err := net.Listen("tcp", ":"+cfg.GRPCPort)
+	if err != nil {
+		log.Error("listen failed", "err", err)
+		os.Exit(1)
+	}
+
+	go func() {
+		log.Info("tts-gateway listening", "addr", lis.Addr().String())
+		if err := srv.Serve(lis); err != nil {
+			log.Error("serve failed", "err", err)
+		}
+	}()
+
+	<-ctx.Done()
+	log.Info("shutdown signal received, draining")
+
+	healthSrv.SetServingStatus("", grpc_health_v1.HealthCheckResponse_NOT_SERVING)
+	srv.GracefulStop()
+	log.Info("shutdown complete")
+
+	if err := shutdown(context.Background()); err != nil {
+		log.Error("telemetry shutdown error", "err", err)
+	}
+}
+
+func loadServerCredentials(tlsDir string) (credentials.TransportCredentials, error) {
+	cert, err := tls.LoadX509KeyPair(
+		filepath.Join(tlsDir, "tls.crt"),
+		filepath.Join(tlsDir, "tls.key"),
+	)
+	if err != nil {
+		return nil, fmt.Errorf("load server key pair: %w", err)
+	}
+
+	caPEM, err := os.ReadFile(filepath.Join(tlsDir, "ca.crt"))
+	if err != nil {
+		return nil, fmt.Errorf("read client CA: %w", err)
+	}
+
+	clientCAs := x509.NewCertPool()
+	if !clientCAs.AppendCertsFromPEM(caPEM) {
+		return nil, fmt.Errorf("append client CA: invalid PEM")
+	}
+
+	return credentials.NewTLS(&tls.Config{
+		Certificates: []tls.Certificate{cert},
+		ClientCAs:    clientCAs,
+		ClientAuth:   tls.RequireAndVerifyClientCert,
+		MinVersion:   tls.VersionTLS13,
+	}), nil
+}

tts-gateway/go.mod

@@ -0,0 +1,37 @@
+module gitea.nik4nao.com/nik/home-services/tts-gateway
+
+go 1.26
+
+require (
+	gitea.nik4nao.com/nik/home-services/gen v0.0.0
+	github.com/joho/godotenv v1.5.1
+	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.60.0
+	go.opentelemetry.io/otel v1.39.0
+	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.35.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.35.0
+	go.opentelemetry.io/otel/metric v1.39.0
+	go.opentelemetry.io/otel/sdk v1.39.0
+	go.opentelemetry.io/otel/sdk/metric v1.39.0
+	go.opentelemetry.io/otel/trace v1.39.0
+	google.golang.org/grpc v1.79.3
+)
+
+require (
+	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.35.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.5.0 // indirect
+	golang.org/x/net v0.48.0 // indirect
+	golang.org/x/sys v0.39.0 // indirect
+	golang.org/x/text v0.32.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20251202230838-ff82c1b0f217 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
+	google.golang.org/protobuf v1.36.11 // indirect
+)
+
+replace gitea.nik4nao.com/nik/home-services/gen => ../gen

tts-gateway/internal/adapters/primary/grpc/interceptor.go

@@ -0,0 +1,61 @@
+package grpc
+
+import (
+	"context"
+	"log/slog"
+	"time"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/logger"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/peer"
+	"google.golang.org/grpc/status"
+)
+
+// LoggingUnaryInterceptor logs one completion record for each unary gRPC call.
+func LoggingUnaryInterceptor(log *slog.Logger) grpc.UnaryServerInterceptor {
+	return func(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
+		method, ok := grpc.Method(ctx)
+		if !ok {
+			method = info.FullMethod
+		}
+		reqLog := requestLogger(ctx, log, method)
+		ctx = logger.WithLogger(ctx, reqLog)
+
+		start := time.Now()
+		resp, err := handler(ctx, req)
+		logCompletion(reqLog, "grpc call completed", status.Code(err), time.Since(start), err)
+		return resp, err
+	}
+}
+
+// requestLogger derives a child logger so every downstream component sees the
+// same gRPC method and peer metadata through context propagation.
+func requestLogger(ctx context.Context, log *slog.Logger, method string) *slog.Logger {
+	peerAddr := ""
+	if p, ok := peer.FromContext(ctx); ok && p.Addr != nil {
+		peerAddr = p.Addr.String()
+	}
+	return log.With("grpc.method", method, "grpc.peer", peerAddr)
+}
+
+// logCompletion keeps severity consistent with gRPC status semantics so
+// expected client-facing errors do not look like infrastructure failures.
+func logCompletion(log *slog.Logger, msg string, code codes.Code, duration time.Duration, err error) {
+	attrs := []any{
+		"duration_ms", duration.Milliseconds(),
+		"grpc.code", code.String(),
+	}
+	if err != nil {
+		attrs = append(attrs, "error", err.Error())
+	}
+
+	switch code {
+	case codes.OK:
+		log.Info(msg, attrs...)
+	case codes.NotFound, codes.InvalidArgument, codes.Unimplemented:
+		log.Warn(msg, attrs...)
+	default:
+		log.Error(msg, attrs...)
+	}
+}

tts-gateway/internal/adapters/primary/grpc/tts.go

@@ -0,0 +1,64 @@
+package grpc
+
+import (
+	"context"
+	"errors"
+
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	ttsv1 "gitea.nik4nao.com/nik/home-services/gen/tts/v1"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/app"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/domain"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/ports/driving"
+)
+
+type TTSGRPC struct {
+	ttsv1.UnimplementedTTSServiceServer
+	svc driving.TTSService
+}
+
+// NewTTSGRPC constructs the gRPC adapter for TTSService.
+func NewTTSGRPC(svc driving.TTSService) *TTSGRPC {
+	return &TTSGRPC{svc: svc}
+}
+
+// Synthesize translates a protobuf request into a domain call, applying
+// uma-tts-api's sampling defaults for any knob the client omits.
+func (h *TTSGRPC) Synthesize(ctx context.Context, req *ttsv1.SynthesizeRequest) (*ttsv1.SynthesizeResponse, error) {
+	params := domain.SynthesisParams{
+		NoiseScale:  domain.DefaultNoiseScale,
+		NoiseScaleW: domain.DefaultNoiseScaleW,
+		LengthScale: domain.DefaultLengthScale,
+	}
+	if req.NoiseScale != nil {
+		params.NoiseScale = *req.NoiseScale
+	}
+	if req.NoiseScaleW != nil {
+		params.NoiseScaleW = *req.NoiseScaleW
+	}
+	if req.LengthScale != nil {
+		params.LengthScale = *req.LengthScale
+	}
+
+	clip, err := h.svc.Synthesize(ctx, req.SpeakerName, req.Text, params)
+	if err != nil {
+		return nil, grpcError(err)
+	}
+	return &ttsv1.SynthesizeResponse{Audio: clip.Data, MimeType: clip.MimeType}, nil
+}
+
+// ListSpeakers returns the (optionally filtered) speaker roster.
+func (h *TTSGRPC) ListSpeakers(ctx context.Context, req *ttsv1.ListSpeakersRequest) (*ttsv1.ListSpeakersResponse, error) {
+	names := h.svc.ListSpeakers(ctx, req.Search)
+	return &ttsv1.ListSpeakersResponse{SpeakerNames: names}, nil
+}
+
+// grpcError maps domain errors to appropriate gRPC status codes, mirroring
+// uma-tts-api's /synthesize 400 response for an unknown speaker name.
+func grpcError(err error) error {
+	if errors.Is(err, app.ErrSpeakerNotFound) {
+		return status.Errorf(codes.InvalidArgument, "%v", err)
+	}
+	return status.Errorf(codes.Internal, "%v", err)
+}

tts-gateway/internal/adapters/secondary/ffmpeg/ffmpeg.go

@@ -0,0 +1,55 @@
+// Package ffmpeg implements driven.AudioEncoder: WAV framing done by hand,
+// then a shell-out to the system ffmpeg binary to transcode to AAC/M4A,
+// matching uma-tts-api's pydub `.export(..., format="ipod")` step
+// (mimetype audio/aac, despite the "ipod" format name being an M4A/AAC
+// container alias).
+package ffmpeg
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"os"
+	"os/exec"
+)
+
+type Encoder struct{}
+
+func NewEncoder() *Encoder {
+	return &Encoder{}
+}
+
+// Encode transcodes PCM samples to AAC. The output goes to a temp file
+// rather than a stdout pipe because the M4A/MP4 muxer needs a seekable
+// destination to write its moov atom.
+func (e *Encoder) Encode(ctx context.Context, pcm []float32, sampleRate int) ([]byte, string, error) {
+	wavBytes := encodeWAV(pcm, sampleRate)
+
+	tmpFile, err := os.CreateTemp("", "tts-gateway-*.m4a")
+	if err != nil {
+		return nil, "", fmt.Errorf("ffmpeg: create temp output file: %w", err)
+	}
+	tmpPath := tmpFile.Name()
+	_ = tmpFile.Close()
+	defer os.Remove(tmpPath)
+
+	cmd := exec.CommandContext(ctx, "ffmpeg",
+		"-y",
+		"-f", "wav", "-i", "pipe:0",
+		"-f", "ipod",
+		tmpPath,
+	)
+	cmd.Stdin = bytes.NewReader(wavBytes)
+	var stderr bytes.Buffer
+	cmd.Stderr = &stderr
+
+	if err := cmd.Run(); err != nil {
+		return nil, "", fmt.Errorf("ffmpeg: transcode failed: %w: %s", err, stderr.String())
+	}
+
+	data, err := os.ReadFile(tmpPath)
+	if err != nil {
+		return nil, "", fmt.Errorf("ffmpeg: read transcoded output: %w", err)
+	}
+	return data, "audio/aac", nil
+}

tts-gateway/internal/adapters/secondary/ffmpeg/wav.go

@@ -0,0 +1,52 @@
+package ffmpeg
+
+import (
+	"bytes"
+	"encoding/binary"
+)
+
+// encodeWAV writes float32 PCM samples (range [-1, 1]) as a 16-bit PCM mono
+// WAV file, matching how uma-tts-api's soundfile.write step feeds pydub
+// (samples get truncated/clamped rather than wrapped on overflow).
+func encodeWAV(pcm []float32, sampleRate int) []byte {
+	const (
+		numChannels   = 1
+		bitsPerSample = 16
+	)
+	byteRate := sampleRate * numChannels * bitsPerSample / 8
+	blockAlign := numChannels * bitsPerSample / 8
+	dataSize := len(pcm) * 2
+
+	var buf bytes.Buffer
+	buf.WriteString("RIFF")
+	_ = binary.Write(&buf, binary.LittleEndian, uint32(36+dataSize))
+	buf.WriteString("WAVE")
+
+	buf.WriteString("fmt ")
+	_ = binary.Write(&buf, binary.LittleEndian, uint32(16)) // PCM fmt chunk size
+	_ = binary.Write(&buf, binary.LittleEndian, uint16(1))  // PCM format tag
+	_ = binary.Write(&buf, binary.LittleEndian, uint16(numChannels))
+	_ = binary.Write(&buf, binary.LittleEndian, uint32(sampleRate))
+	_ = binary.Write(&buf, binary.LittleEndian, uint32(byteRate))
+	_ = binary.Write(&buf, binary.LittleEndian, uint16(blockAlign))
+	_ = binary.Write(&buf, binary.LittleEndian, uint16(bitsPerSample))
+
+	buf.WriteString("data")
+	_ = binary.Write(&buf, binary.LittleEndian, uint32(dataSize))
+	for _, sample := range pcm {
+		_ = binary.Write(&buf, binary.LittleEndian, int16(clampSample(sample)*32767))
+	}
+
+	return buf.Bytes()
+}
+
+func clampSample(s float32) float32 {
+	switch {
+	case s > 1:
+		return 1
+	case s < -1:
+		return -1
+	default:
+		return s
+	}
+}

tts-gateway/internal/adapters/secondary/ffmpeg/wav_test.go

@@ -0,0 +1,39 @@
+package ffmpeg
+
+import (
+	"encoding/binary"
+	"testing"
+)
+
+func TestEncodeWAV(t *testing.T) {
+	pcm := []float32{0, 1, -1, 2, -2} // last two exercise clamping
+	sampleRate := 22050
+
+	data := encodeWAV(pcm, sampleRate)
+
+	if string(data[0:4]) != "RIFF" || string(data[8:12]) != "WAVE" {
+		t.Fatalf("encodeWAV() missing RIFF/WAVE header: %q", data[:12])
+	}
+	if string(data[12:16]) != "fmt " || string(data[36:40]) != "data" {
+		t.Fatalf("encodeWAV() missing fmt/data chunk headers")
+	}
+
+	gotSampleRate := binary.LittleEndian.Uint32(data[24:28])
+	if gotSampleRate != uint32(sampleRate) {
+		t.Fatalf("encodeWAV() sample rate = %d, want %d", gotSampleRate, sampleRate)
+	}
+
+	dataSize := binary.LittleEndian.Uint32(data[40:44])
+	if int(dataSize) != len(pcm)*2 {
+		t.Fatalf("encodeWAV() data size = %d, want %d", dataSize, len(pcm)*2)
+	}
+
+	samples := data[44:]
+	want := []int16{0, 32767, -32767, 32767, -32767} // clamped to [-1, 1] before scaling
+	for i, w := range want {
+		got := int16(binary.LittleEndian.Uint16(samples[i*2 : i*2+2]))
+		if got != w {
+			t.Errorf("sample %d = %d, want %d", i, got, w)
+		}
+	}
+}

tts-gateway/internal/adapters/secondary/inferencesidecar/client.go

@@ -0,0 +1,96 @@
+// Package inferencesidecar implements driven.TTSEngine by calling a small
+// Python/libtorch sidecar (tts-gateway/sidecar) over HTTP - the fallback
+// documented in TTS_GATEWAY_PLAN.md's Risk 1 and adopted after the Phase 0
+// spike found the checkpoint's data-dependent output length isn't cleanly
+// exportable via torch.export. Same shape as ai-gateway's Ollama client:
+// the tensor math lives outside Go, hidden behind this port like any other
+// outbound dependency.
+package inferencesidecar
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"math"
+	"net/http"
+	"strconv"
+	"time"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/domain"
+)
+
+type Client struct {
+	baseURL    string
+	httpClient *http.Client
+}
+
+// NewClient constructs the sidecar HTTP client. addr is a host:port, e.g.
+// "localhost:50054" (config.Config.InferenceSidecarAddr).
+func NewClient(addr string) *Client {
+	return &Client{
+		baseURL:    "http://" + addr,
+		httpClient: &http.Client{Timeout: 60 * time.Second},
+	}
+}
+
+type synthesizeRequest struct {
+	SpeakerID   int     `json:"speaker_id"`
+	SymbolIDs   []int64 `json:"symbol_ids"`
+	NoiseScale  float32 `json:"noise_scale"`
+	NoiseScaleW float32 `json:"noise_scale_w"`
+	LengthScale float32 `json:"length_scale"`
+}
+
+// Synthesize posts symbol IDs + params to the sidecar's /synthesize
+// endpoint and decodes its raw little-endian float32 PCM response.
+func (c *Client) Synthesize(ctx context.Context, speakerID int, symbolIDs []int64, params domain.SynthesisParams) ([]float32, int, error) {
+	reqBody, err := json.Marshal(synthesizeRequest{
+		SpeakerID:   speakerID,
+		SymbolIDs:   symbolIDs,
+		NoiseScale:  params.NoiseScale,
+		NoiseScaleW: params.NoiseScaleW,
+		LengthScale: params.LengthScale,
+	})
+	if err != nil {
+		return nil, 0, fmt.Errorf("inferencesidecar: marshal request: %w", err)
+	}
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/synthesize", bytes.NewReader(reqBody))
+	if err != nil {
+		return nil, 0, fmt.Errorf("inferencesidecar: build request: %w", err)
+	}
+	req.Header.Set("Content-Type", "application/json")
+
+	resp, err := c.httpClient.Do(req)
+	if err != nil {
+		return nil, 0, fmt.Errorf("inferencesidecar: request failed: %w", err)
+	}
+	defer resp.Body.Close()
+
+	data, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, 0, fmt.Errorf("inferencesidecar: read response: %w", err)
+	}
+	if resp.StatusCode != http.StatusOK {
+		return nil, 0, fmt.Errorf("inferencesidecar: %s: %s", resp.Status, string(data))
+	}
+
+	sampleRate, err := strconv.Atoi(resp.Header.Get("X-Sample-Rate"))
+	if err != nil {
+		return nil, 0, fmt.Errorf("inferencesidecar: missing/invalid X-Sample-Rate header: %w", err)
+	}
+	if len(data)%4 != 0 {
+		return nil, 0, fmt.Errorf("inferencesidecar: response length %d not a multiple of 4", len(data))
+	}
+
+	pcm := make([]float32, len(data)/4)
+	for i := range pcm {
+		bits := binary.LittleEndian.Uint32(data[i*4 : i*4+4])
+		pcm[i] = math.Float32frombits(bits)
+	}
+
+	return pcm, sampleRate, nil
+}

tts-gateway/internal/adapters/secondary/inferencesidecar/client_test.go

@@ -0,0 +1,71 @@
+package inferencesidecar
+
+import (
+	"encoding/binary"
+	"encoding/json"
+	"math"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/domain"
+)
+
+func TestClientSynthesize(t *testing.T) {
+	t.Run("happy path decodes little-endian float32 PCM and sample rate header", func(t *testing.T) {
+		wantPCM := []float32{0.1, -0.2, 0.3}
+
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			if r.URL.Path != "/synthesize" {
+				t.Fatalf("request path = %q, want /synthesize", r.URL.Path)
+			}
+			var body synthesizeRequest
+			if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
+				t.Fatalf("decode request body: %v", err)
+			}
+			if body.SpeakerID != 29 || len(body.SymbolIDs) != 2 {
+				t.Fatalf("request body = %+v, unexpected", body)
+			}
+
+			payload := make([]byte, len(wantPCM)*4)
+			for i, f := range wantPCM {
+				binary.LittleEndian.PutUint32(payload[i*4:], math.Float32bits(f))
+			}
+			w.Header().Set("X-Sample-Rate", "22050")
+			w.WriteHeader(http.StatusOK)
+			_, _ = w.Write(payload)
+		}))
+		defer srv.Close()
+
+		c := NewClient(srv.Listener.Addr().String())
+		gotPCM, gotSampleRate, err := c.Synthesize(t.Context(), 29, []int64{1, 2}, domain.SynthesisParams{NoiseScale: 0.37})
+		if err != nil {
+			t.Fatalf("Synthesize() error = %v", err)
+		}
+		if gotSampleRate != 22050 {
+			t.Fatalf("Synthesize() sampleRate = %d, want 22050", gotSampleRate)
+		}
+		if len(gotPCM) != len(wantPCM) {
+			t.Fatalf("Synthesize() pcm = %v, want %v", gotPCM, wantPCM)
+		}
+		for i, want := range wantPCM {
+			if gotPCM[i] != want {
+				t.Errorf("pcm[%d] = %v, want %v", i, gotPCM[i], want)
+			}
+		}
+	})
+
+	t.Run("non-200 response is surfaced as an error with the body", func(t *testing.T) {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.WriteHeader(http.StatusBadRequest)
+			_, _ = w.Write([]byte("bad speaker id"))
+		}))
+		defer srv.Close()
+
+		c := NewClient(srv.Listener.Addr().String())
+		_, _, err := c.Synthesize(t.Context(), 999, []int64{1}, domain.SynthesisParams{})
+		if err == nil {
+			t.Fatal("Synthesize() error = nil, want non-nil")
+		}
+	})
+}

tts-gateway/internal/adapters/secondary/jtalk/jtalk.go

@@ -0,0 +1,287 @@
+// Package jtalk implements driven.TextNormalizer by shelling out to the
+// open_jtalk CLI, reproducing uma-tts-api's japanese_cleaners pipeline
+// (text/cleaners.py) exactly - verified rune-for-rune against the real
+// Python implementation during TTS_GATEWAY_PLAN.md's Phase 0 spike. See
+// tmp/reference/uma-tts-api/spike/FINDINGS.md for the verification detail.
+package jtalk
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"io/fs"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"strings"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/config"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/domain"
+)
+
+type Client struct {
+	bin     string
+	dictDir string
+	voice   string
+}
+
+// NewClient constructs the open_jtalk CLI adapter, auto-discovering the
+// dictionary directory and voice file when the config leaves them empty
+// (matches how the Phase 0 spike located them on nik-gpu's open-jtalk +
+// open-jtalk-mecab-naist-jdic + hts-voice-* apt packages).
+func NewClient(cfg *config.Config) (*Client, error) {
+	dictDir := cfg.OpenJTalkDictDir
+	if dictDir == "" {
+		var err error
+		dictDir, err = findDictDir()
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	voice := cfg.OpenJTalkVoice
+	if voice == "" {
+		var err error
+		voice, err = findVoice()
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return &Client{bin: cfg.OpenJTalkBin, dictDir: dictDir, voice: voice}, nil
+}
+
+var dictDirCandidates = []string{
+	"/usr/lib/*/open_jtalk/open_jtalk_dic_utf_8-*",
+	"/usr/share/open_jtalk/dic/*",
+	"/var/lib/mecab/dic/open-jtalk/naist-jdic",
+}
+
+func findDictDir() (string, error) {
+	for _, pattern := range dictDirCandidates {
+		matches, err := filepath.Glob(pattern)
+		if err != nil {
+			continue
+		}
+		for _, m := range matches {
+			if info, err := os.Stat(m); err == nil && info.IsDir() {
+				return m, nil
+			}
+		}
+	}
+	return "", fmt.Errorf("no open_jtalk dictionary dir found under %v", dictDirCandidates)
+}
+
+func findVoice() (string, error) {
+	const root = "/usr/share/hts-voice"
+	var found string
+	err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
+		if err != nil || found != "" {
+			return nil
+		}
+		if !d.IsDir() && strings.HasSuffix(path, ".htsvoice") {
+			found = path
+		}
+		return nil
+	})
+	if err != nil || found == "" {
+		return "", fmt.Errorf("no .htsvoice file found under %s", root)
+	}
+	return found, nil
+}
+
+// Normalize reproduces japanese_cleaners(text) then maps the result
+// character-by-character to symbol IDs (text_to_sequence's actual
+// behavior - see FINDINGS.md on why this is intentional-if-buggy fidelity,
+// not a mistake), and finally intersperses blank tokens (add_blank=true).
+func (c *Client) Normalize(ctx context.Context, text string) ([]int64, error) {
+	cleaned, err := c.japaneseCleaners(ctx, text)
+	if err != nil {
+		return nil, err
+	}
+
+	seq := make([]int64, 0, len(cleaned))
+	for _, r := range cleaned {
+		id, ok := domain.SymbolToID[r]
+		if !ok {
+			return nil, fmt.Errorf("jtalk: no symbol for rune %q in cleaned text %q", r, cleaned)
+		}
+		seq = append(seq, int64(id))
+	}
+
+	return domain.Intersperse(seq, 0), nil
+}
+
+// japaneseCleaners ports text/cleaners.py::japanese_cleaners rune-for-rune.
+func (c *Client) japaneseCleaners(ctx context.Context, text string) (string, error) {
+	spans, marks := splitByMarks(text)
+
+	var b strings.Builder
+	for i, mark := range marks {
+		if spans[i] != "" {
+			phonemes, err := c.g2p(ctx, spans[i])
+			if err != nil {
+				return "", err
+			}
+			b.WriteString(phonemes)
+		}
+		b.WriteString(markToASCII(mark))
+	}
+	if last := spans[len(spans)-1]; last != "" {
+		phonemes, err := c.g2p(ctx, last)
+		if err != nil {
+			return "", err
+		}
+		b.WriteString(phonemes)
+	}
+
+	out := b.String()
+	if out != "" {
+		r := []rune(out)
+		last := r[len(r)-1]
+		if (last >= 'A' && last <= 'Z') || (last >= 'a' && last <= 'z') {
+			out += "."
+		}
+	}
+	return out, nil
+}
+
+// splitByMarks reproduces re.split(_japanese_marks, text) / re.findall(...):
+// every individual non-Japanese-classified rune is its own split point, so
+// consecutive marks yield an empty span between them. len(spans) is always
+// len(marks)+1.
+func splitByMarks(text string) (spans []string, marks []rune) {
+	var current strings.Builder
+	for _, r := range text {
+		if isJapaneseChar(r) {
+			current.WriteRune(r)
+			continue
+		}
+		spans = append(spans, current.String())
+		marks = append(marks, r)
+		current.Reset()
+	}
+	spans = append(spans, current.String())
+	return spans, marks
+}
+
+// isJapaneseChar mirrors pyopenjtalk's cleaner regex
+// `[A-Za-z\d々぀-ヿ一-鿿１-９Ａ-Ｚａ-ｚｦ-ﾝ]` exactly, including its quirks (e.g.
+// fullwidth "０" (U+FF10) is NOT included, only "１"-"９").
+func isJapaneseChar(r rune) bool {
+	switch {
+	case r >= 'A' && r <= 'Z', r >= 'a' && r <= 'z', r >= '0' && r <= '9':
+		return true
+	case r == '々': // U+3005
+		return true
+	case r >= '぀' && r <= 'ヿ': // ぀-ヿ: Hiragana + Katakana
+		return true
+	case r >= '一' && r <= '鿿': // 一-鿿: CJK Unified Ideographs
+		return true
+	case r >= '１' && r <= '９': // １-９ (fullwidth 1-9, NOT 0)
+		return true
+	case r >= 'Ａ' && r <= 'Ｚ': // Ａ-Ｚ
+		return true
+	case r >= 'ａ' && r <= 'ｚ': // ａ-ｚ
+		return true
+	case r >= 'ｦ' && r <= 'ﾝ': // ｦ-ﾝ (halfwidth katakana)
+		return true
+	default:
+		return false
+	}
+}
+
+// markToASCII reproduces unidecode(mark).replace(' ', ”) for the small set
+// of punctuation marks _japanese_marks actually matches in practice (verified
+// against real unidecode output during the Phase 0 spike). ASCII runes are
+// unidecode no-ops so pass through unchanged; anything else not in this
+// table is dropped (empty string) rather than guessing a transliteration.
+func markToASCII(r rune) string {
+	if r < 128 {
+		return string(r)
+	}
+	if s, ok := markTable[r]; ok {
+		return s
+	}
+	return ""
+}
+
+var markTable = map[rune]string{
+	'、': ",",
+	'。': ".",
+	'「': "[",
+	'」': "]",
+	'『': "{",
+	'』': "}",
+	'・': "*",
+	'〜': "~",
+	'！': "!",
+	'？': "?",
+	'，': ",",
+	'．': ".",
+	'　': " ", // fullwidth space
+	'…': "...",
+	'―': "--",
+}
+
+var labelPhonemeRe = regexp.MustCompile(`-([^+]+)\+`)
+
+// g2p runs the open_jtalk CLI on one Japanese-only span and returns the
+// concatenated phoneme string with 'pau' tokens and spaces stripped, matching
+// pyopenjtalk.g2p(span, kana=False).replace('pau',”).replace(' ',”).
+func (c *Client) g2p(ctx context.Context, span string) (string, error) {
+	cmd := exec.CommandContext(ctx, c.bin, "-x", c.dictDir, "-m", c.voice, "-ot", "/dev/stdout", "-ow", "/dev/null")
+	cmd.Stdin = strings.NewReader(span)
+	var stdout bytes.Buffer
+	cmd.Stdout = &stdout
+	if err := cmd.Run(); err != nil {
+		return "", fmt.Errorf("jtalk: open_jtalk run: %w", err)
+	}
+
+	phonemes, err := parseOutputLabelPhonemes(stdout.String())
+	if err != nil {
+		return "", err
+	}
+
+	var b strings.Builder
+	for _, p := range phonemes {
+		if p == "pau" {
+			continue
+		}
+		b.WriteString(p)
+	}
+	return b.String(), nil
+}
+
+// parseOutputLabelPhonemes extracts the current-phoneme field from each HTS
+// full-context label line within open_jtalk's verbose "-ot /dev/stdout"
+// output, scoped to only the [Output label] section - the rest of the
+// verbose dump ([Global parameter] etc.) can contain lines that spuriously
+// match the same "-X+" pattern, which silently corrupted an earlier version
+// of this parser (see FINDINGS.md). Boundary sil tokens are stripped.
+func parseOutputLabelPhonemes(rawStdout string) ([]string, error) {
+	const startMarker = "[Output label]"
+	_, section, ok := strings.Cut(rawStdout, startMarker)
+	if !ok {
+		return nil, fmt.Errorf("jtalk: %q not found in open_jtalk output", startMarker)
+	}
+	if end := strings.IndexByte(section, '['); end != -1 {
+		section = section[:end]
+	}
+
+	var phonemes []string
+	for line := range strings.SplitSeq(strings.TrimSpace(section), "\n") {
+		m := labelPhonemeRe.FindStringSubmatch(line)
+		if m != nil {
+			phonemes = append(phonemes, m[1])
+		}
+	}
+	if len(phonemes) > 0 && phonemes[0] == "sil" {
+		phonemes = phonemes[1:]
+	}
+	if len(phonemes) > 0 && phonemes[len(phonemes)-1] == "sil" {
+		phonemes = phonemes[:len(phonemes)-1]
+	}
+	return phonemes, nil
+}

tts-gateway/internal/adapters/secondary/jtalk/jtalk_test.go

@@ -0,0 +1,130 @@
+package jtalk
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestIsJapaneseChar(t *testing.T) {
+	tests := []struct {
+		name string
+		r    rune
+		want bool
+	}{
+		{"ascii letter", 'a', true},
+		{"ascii digit", '5', true},
+		{"hiragana", 'お', true},
+		{"katakana", 'ー', true}, // U+30FC, within ぀-ヿ
+		{"kanji", '様', true},
+		{"iteration mark", '々', true},
+		{"fullwidth digit 1-9", '１', true},
+		{"fullwidth digit 0 excluded (quirk in the original regex)", '０', false},
+		{"fullwidth letter", 'Ａ', true},
+		{"halfwidth katakana", 'ｦ', true},
+		{"japanese comma is a mark, not a char", '、', false},
+		{"japanese period is a mark", '。', false},
+		{"ascii punctuation is a mark", '!', false},
+		{"space is a mark", ' ', false},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := isJapaneseChar(tt.r); got != tt.want {
+				t.Errorf("isJapaneseChar(%q) = %v, want %v", tt.r, got, tt.want)
+			}
+		})
+	}
+}
+
+func TestSplitByMarks(t *testing.T) {
+	tests := []struct {
+		name      string
+		text      string
+		wantSpans []string
+		wantMarks []rune
+	}{
+		{
+			name:      "no marks",
+			text:      "おはよう",
+			wantSpans: []string{"おはよう"},
+			wantMarks: nil,
+		},
+		{
+			name:      "one internal mark",
+			text:      "おにー様、すきです",
+			wantSpans: []string{"おにー様", "すきです"},
+			wantMarks: []rune{'、'},
+		},
+		{
+			name:      "consecutive marks produce an empty span between them",
+			text:      "すごい!!すごい",
+			wantSpans: []string{"すごい", "", "すごい"},
+			wantMarks: []rune{'!', '!'},
+		},
+		{
+			name:      "trailing mark leaves an empty final span",
+			text:      "ありがとう。",
+			wantSpans: []string{"ありがとう", ""},
+			wantMarks: []rune{'。'},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			gotSpans, gotMarks := splitByMarks(tt.text)
+			if !reflect.DeepEqual(gotSpans, tt.wantSpans) {
+				t.Errorf("splitByMarks() spans = %#v, want %#v", gotSpans, tt.wantSpans)
+			}
+			if !reflect.DeepEqual(gotMarks, tt.wantMarks) {
+				t.Errorf("splitByMarks() marks = %#v, want %#v", gotMarks, tt.wantMarks)
+			}
+		})
+	}
+}
+
+func TestMarkToASCII(t *testing.T) {
+	tests := []struct {
+		r    rune
+		want string
+	}{
+		{'、', ","},
+		{'。', "."},
+		{'！', "!"},
+		{'!', "!"}, // ASCII passthrough
+		{'(', "("}, // ASCII passthrough
+		{'鳥', ""},  // unmapped non-ASCII rune falls back to empty, not a guess
+	}
+	for _, tt := range tests {
+		if got := markToASCII(tt.r); got != tt.want {
+			t.Errorf("markToASCII(%q) = %q, want %q", tt.r, got, tt.want)
+		}
+	}
+}
+
+func TestParseOutputLabelPhonemes(t *testing.T) {
+	t.Run("strips boundary sil and ignores content after the section", func(t *testing.T) {
+		raw := "[Text analysis result]\n" +
+			"some,morphological,analysis,-1\n" +
+			"\n[Output label]\n" +
+			"0 100 xx^xx-sil+o=n/A:xx+xx+xx\n" +
+			"100 200 xx^sil-o+n=i/A:0+1+5\n" +
+			"200 300 sil^o-n+i=i/A:1+2+4\n" +
+			"300 400 o^n-i+sil=xx/A:1+2+4\n" +
+			"\n[Global parameter]\n" +
+			"Some parameter -> value+with+plusses\n"
+
+		got, err := parseOutputLabelPhonemes(raw)
+		if err != nil {
+			t.Fatalf("parseOutputLabelPhonemes() error = %v", err)
+		}
+		want := []string{"o", "n", "i"}
+		if !reflect.DeepEqual(got, want) {
+			t.Fatalf("parseOutputLabelPhonemes() = %#v, want %#v (must not include [Global parameter] content)", got, want)
+		}
+	})
+
+	t.Run("missing section returns an error", func(t *testing.T) {
+		_, err := parseOutputLabelPhonemes("no label section here")
+		if err == nil {
+			t.Fatal("parseOutputLabelPhonemes() error = nil, want non-nil")
+		}
+	})
+}

tts-gateway/internal/app/tts.go

@@ -0,0 +1,69 @@
+package app
+
+import (
+	"context"
+	"errors"
+	"strings"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/domain"
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/ports/driven"
+)
+
+// ErrSpeakerNotFound is returned when the requested speaker name has no
+// matching entry in domain.Speakers.
+var ErrSpeakerNotFound = errors.New("speaker not found")
+
+type TTSApp struct {
+	normalizer driven.TextNormalizer
+	engine     driven.TTSEngine
+	encoder    driven.AudioEncoder
+	nameToID   map[string]int
+}
+
+// NewTTSApp constructs the synthesis application service.
+func NewTTSApp(normalizer driven.TextNormalizer, engine driven.TTSEngine, encoder driven.AudioEncoder) *TTSApp {
+	nameToID := make(map[string]int, len(domain.Speakers))
+	for _, s := range domain.Speakers {
+		nameToID[s.Name] = s.ID
+	}
+	return &TTSApp{normalizer: normalizer, engine: engine, encoder: encoder, nameToID: nameToID}
+}
+
+// Synthesize normalizes text, runs the TTS engine, and encodes the result.
+func (a *TTSApp) Synthesize(ctx context.Context, speakerName, text string, params domain.SynthesisParams) (*domain.AudioClip, error) {
+	speakerID, ok := a.nameToID[speakerName]
+	if !ok {
+		return nil, ErrSpeakerNotFound
+	}
+
+	symbolIDs, err := a.normalizer.Normalize(ctx, text)
+	if err != nil {
+		return nil, err
+	}
+
+	pcm, sampleRate, err := a.engine.Synthesize(ctx, speakerID, symbolIDs, params)
+	if err != nil {
+		return nil, err
+	}
+
+	data, mimeType, err := a.encoder.Encode(ctx, pcm, sampleRate)
+	if err != nil {
+		return nil, err
+	}
+
+	return &domain.AudioClip{Data: data, MimeType: mimeType}, nil
+}
+
+// ListSpeakers returns speaker names filtered by a case-insensitive substring
+// match, or the full roster when search is empty - matching uma-tts-api's
+// /speakers endpoint.
+func (a *TTSApp) ListSpeakers(_ context.Context, search string) []string {
+	search = strings.ToLower(search)
+	names := make([]string, 0, len(domain.Speakers))
+	for _, s := range domain.Speakers {
+		if search == "" || strings.Contains(strings.ToLower(s.Name), search) {
+			names = append(names, s.Name)
+		}
+	}
+	return names
+}

tts-gateway/internal/app/tts_test.go

@@ -0,0 +1,134 @@
+package app
+
+import (
+	"context"
+	"errors"
+	"testing"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/domain"
+)
+
+type mockNormalizer struct {
+	normalizeFunc func(ctx context.Context, text string) ([]int64, error)
+}
+
+func (m *mockNormalizer) Normalize(ctx context.Context, text string) ([]int64, error) {
+	return m.normalizeFunc(ctx, text)
+}
+
+type mockEngine struct {
+	synthesizeFunc func(ctx context.Context, speakerID int, symbolIDs []int64, params domain.SynthesisParams) ([]float32, int, error)
+}
+
+func (m *mockEngine) Synthesize(ctx context.Context, speakerID int, symbolIDs []int64, params domain.SynthesisParams) ([]float32, int, error) {
+	return m.synthesizeFunc(ctx, speakerID, symbolIDs, params)
+}
+
+type mockEncoder struct {
+	encodeFunc func(ctx context.Context, pcm []float32, sampleRate int) ([]byte, string, error)
+}
+
+func (m *mockEncoder) Encode(ctx context.Context, pcm []float32, sampleRate int) ([]byte, string, error) {
+	return m.encodeFunc(ctx, pcm, sampleRate)
+}
+
+func TestTTSAppSynthesize(t *testing.T) {
+	t.Run("unknown speaker returns ErrSpeakerNotFound without calling downstream ports", func(t *testing.T) {
+		a := NewTTSApp(
+			&mockNormalizer{normalizeFunc: func(context.Context, string) ([]int64, error) {
+				t.Fatal("Normalize should not be called for an unknown speaker")
+				return nil, nil
+			}},
+			&mockEngine{},
+			&mockEncoder{},
+		)
+
+		_, err := a.Synthesize(context.Background(), "Not A Real Speaker", "hello", domain.SynthesisParams{})
+		if !errors.Is(err, ErrSpeakerNotFound) {
+			t.Fatalf("Synthesize() error = %v, want %v", err, ErrSpeakerNotFound)
+		}
+	})
+
+	t.Run("happy path wires normalize -> engine -> encode", func(t *testing.T) {
+		wantParams := domain.SynthesisParams{NoiseScale: 0.1, NoiseScaleW: 0.2, LengthScale: 0.3}
+		wantSymbolIDs := []int64{1, 2, 3}
+		wantPCM := []float32{0.5, -0.5}
+		wantSampleRate := 22050
+		wantAudio := []byte{1, 2, 3, 4}
+		wantMimeType := "audio/aac"
+
+		a := NewTTSApp(
+			&mockNormalizer{normalizeFunc: func(_ context.Context, text string) ([]int64, error) {
+				if text != "hello" {
+					t.Fatalf("Normalize() text = %q, want %q", text, "hello")
+				}
+				return wantSymbolIDs, nil
+			}},
+			&mockEngine{synthesizeFunc: func(_ context.Context, speakerID int, symbolIDs []int64, params domain.SynthesisParams) ([]float32, int, error) {
+				if speakerID != 29 {
+					t.Fatalf("Synthesize() speakerID = %d, want 29 (Rice Shower)", speakerID)
+				}
+				if len(symbolIDs) != len(wantSymbolIDs) {
+					t.Fatalf("Synthesize() symbolIDs = %v, want %v", symbolIDs, wantSymbolIDs)
+				}
+				if params != wantParams {
+					t.Fatalf("Synthesize() params = %v, want %v", params, wantParams)
+				}
+				return wantPCM, wantSampleRate, nil
+			}},
+			&mockEncoder{encodeFunc: func(_ context.Context, pcm []float32, sampleRate int) ([]byte, string, error) {
+				if sampleRate != wantSampleRate {
+					t.Fatalf("Encode() sampleRate = %d, want %d", sampleRate, wantSampleRate)
+				}
+				return wantAudio, wantMimeType, nil
+			}},
+		)
+
+		got, err := a.Synthesize(context.Background(), "Rice Shower", "hello", wantParams)
+		if err != nil {
+			t.Fatalf("Synthesize() error = %v", err)
+		}
+		if string(got.Data) != string(wantAudio) || got.MimeType != wantMimeType {
+			t.Fatalf("Synthesize() = %+v, want Data=%v MimeType=%q", got, wantAudio, wantMimeType)
+		}
+	})
+
+	t.Run("propagates normalizer error", func(t *testing.T) {
+		wantErr := errors.New("normalize failed")
+		a := NewTTSApp(
+			&mockNormalizer{normalizeFunc: func(context.Context, string) ([]int64, error) { return nil, wantErr }},
+			&mockEngine{},
+			&mockEncoder{},
+		)
+
+		_, err := a.Synthesize(context.Background(), "Rice Shower", "hello", domain.SynthesisParams{})
+		if !errors.Is(err, wantErr) {
+			t.Fatalf("Synthesize() error = %v, want %v", err, wantErr)
+		}
+	})
+}
+
+func TestTTSAppListSpeakers(t *testing.T) {
+	a := NewTTSApp(&mockNormalizer{}, &mockEngine{}, &mockEncoder{})
+
+	t.Run("empty search returns full roster", func(t *testing.T) {
+		got := a.ListSpeakers(context.Background(), "")
+		if len(got) != len(domain.Speakers) {
+			t.Fatalf("ListSpeakers(\"\") returned %d names, want %d", len(got), len(domain.Speakers))
+		}
+	})
+
+	t.Run("case-insensitive substring filter", func(t *testing.T) {
+		got := a.ListSpeakers(context.Background(), "rice")
+		if len(got) != 1 || got[0] != "Rice Shower" {
+			t.Fatalf("ListSpeakers(\"rice\") = %v, want [\"Rice Shower\"]", got)
+		}
+	})
+
+	t.Run("no match returns empty, not nil roster", func(t *testing.T) {
+		got := a.ListSpeakers(context.Background(), "definitely not a speaker")
+		if len(got) != 0 {
+			t.Fatalf("ListSpeakers() = %v, want empty", got)
+		}
+	})
+}

tts-gateway/internal/config/config.go

@@ -0,0 +1,43 @@
+package config
+
+import "os"
+
+// Config holds runtime configuration for the TTS gRPC gateway.
+type Config struct {
+	GRPCPort     string // GRPC_PORT, default "50053"
+	TLSDir       string // TLS_DIR, empty disables mTLS for local dev
+	OTELEndpoint string // OTEL_ENDPOINT, e.g. "otel-collector.monitoring.svc:4317"
+	LogLevel     string // LOG_LEVEL, default "info"
+	LogFormat    string // LOG_FORMAT, default "json"
+
+	OpenJTalkBin     string // OPEN_JTALK_BIN, default "open_jtalk"
+	OpenJTalkDictDir string // OPEN_JTALK_DICT_DIR, empty auto-discovers under /usr and /var
+	OpenJTalkVoice   string // OPEN_JTALK_VOICE, empty auto-discovers the first *.htsvoice found
+
+	InferenceSidecarAddr string // INFERENCE_SIDECAR_ADDR, e.g. "localhost:50054"
+}
+
+// Load reads configuration from environment variables and applies defaults.
+func Load() (*Config, error) {
+	return &Config{
+		GRPCPort:     getenvDefault("GRPC_PORT", "50053"),
+		TLSDir:       os.Getenv("TLS_DIR"),
+		OTELEndpoint: os.Getenv("OTEL_ENDPOINT"),
+		LogLevel:     getenvDefault("LOG_LEVEL", "info"),
+		LogFormat:    getenvDefault("LOG_FORMAT", "json"),
+
+		OpenJTalkBin:     getenvDefault("OPEN_JTALK_BIN", "open_jtalk"),
+		OpenJTalkDictDir: os.Getenv("OPEN_JTALK_DICT_DIR"),
+		OpenJTalkVoice:   os.Getenv("OPEN_JTALK_VOICE"),
+
+		InferenceSidecarAddr: getenvDefault("INFERENCE_SIDECAR_ADDR", "localhost:50054"),
+	}, nil
+}
+
+// getenvDefault keeps config loading concise for optional variables with defaults.
+func getenvDefault(key, fallback string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return fallback
+}

tts-gateway/internal/core/domain/domain.go

@@ -0,0 +1,27 @@
+package domain
+
+// Speaker identifies one of the checkpoint's trained voices.
+type Speaker struct {
+	ID   int
+	Name string
+}
+
+// SynthesisParams are the VITS sampling knobs exposed to clients, matching
+// uma-tts-api's defaults (noise_scale=0.37, noise_scale_w=0.46, length_scale=1.3).
+type SynthesisParams struct {
+	NoiseScale  float32
+	NoiseScaleW float32
+	LengthScale float32
+}
+
+// AudioClip is an encoded audio payload ready to return to a client.
+type AudioClip struct {
+	Data     []byte
+	MimeType string
+}
+
+const (
+	DefaultNoiseScale  float32 = 0.37
+	DefaultNoiseScaleW float32 = 0.46
+	DefaultLengthScale float32 = 1.3
+)

tts-gateway/internal/core/domain/speakers.go

@@ -0,0 +1,99 @@
+package domain
+
+// Speakers is the checkpoint's trained voice roster, ported verbatim from
+// uma-tts-api's constant.py::speakerList (92-speaker embedding table; id 74
+// has no name in the original list, so it's intentionally absent here too -
+// not a transcription error).
+var Speakers = []Speaker{
+	{ID: 0, Name: "Special Week"},
+	{ID: 1, Name: "Silence Suzuka"},
+	{ID: 2, Name: "Tokai Teio"},
+	{ID: 3, Name: "Maruzensky"},
+	{ID: 4, Name: "Fuji Kiseki"},
+	{ID: 5, Name: "Oguri Cap"},
+	{ID: 6, Name: "Gold Ship"},
+	{ID: 7, Name: "Vodka"},
+	{ID: 8, Name: "Daiwa Scarlet"},
+	{ID: 9, Name: "Taiki Shuttle"},
+	{ID: 10, Name: "Grass Wonder"},
+	{ID: 11, Name: "Hishi Amazon"},
+	{ID: 12, Name: "Mejiro McQueen"},
+	{ID: 13, Name: "El Condor Pasa"},
+	{ID: 14, Name: "TM Opera O"},
+	{ID: 15, Name: "Narita Brian"},
+	{ID: 16, Name: "Symboli Rudolf"},
+	{ID: 17, Name: "Air Groove"},
+	{ID: 18, Name: "Agnes Digital"},
+	{ID: 19, Name: "Seiun Sky"},
+	{ID: 20, Name: "Tamamo Cross"},
+	{ID: 21, Name: "Fine Motion"},
+	{ID: 22, Name: "Biwa Hayahide"},
+	{ID: 23, Name: "Mayano Top Gun"},
+	{ID: 24, Name: "Manhattan Cafe"},
+	{ID: 25, Name: "Mihono Bourbon"},
+	{ID: 26, Name: "Mejiro Ryan"},
+	{ID: 27, Name: "Hishi Akebono"},
+	{ID: 28, Name: "Yukino Bijin"},
+	{ID: 29, Name: "Rice Shower"},
+	{ID: 30, Name: "Ines Fujin"},
+	{ID: 31, Name: "Agnes Tachyon"},
+	{ID: 32, Name: "Admire Vega"},
+	{ID: 33, Name: "Inari One"},
+	{ID: 34, Name: "Winning Ticket"},
+	{ID: 35, Name: "Air Shakur"},
+	{ID: 36, Name: "Eishin Flash"},
+	{ID: 37, Name: "Curren Chan"},
+	{ID: 38, Name: "Kawakami Princess"},
+	{ID: 39, Name: "Gold City"},
+	{ID: 40, Name: "Sakura Bakushin O"},
+	{ID: 41, Name: "Seeking the Pearl"},
+	{ID: 42, Name: "Shinko Windy"},
+	{ID: 43, Name: "Sweep Tosho"},
+	{ID: 44, Name: "Super Creek"},
+	{ID: 45, Name: "Smart Falcon"},
+	{ID: 46, Name: "Zenno Rob Roy"},
+	{ID: 47, Name: "Tosen Jordan"},
+	{ID: 48, Name: "Nakayama Festa"},
+	{ID: 49, Name: "Narita Taishin"},
+	{ID: 50, Name: "Nishino Flower"},
+	{ID: 51, Name: "Haru Urara"},
+	{ID: 52, Name: "Bamboo Memory"},
+	{ID: 53, Name: "Biko Pegasus"},
+	{ID: 54, Name: "Marvelous Sunday"},
+	{ID: 55, Name: "Matikanefukukitaru"},
+	{ID: 56, Name: "Mr. C.B."},
+	{ID: 57, Name: "Meisho doto"},
+	{ID: 58, Name: "Mejiro Dober"},
+	{ID: 59, Name: "Nice Nature"},
+	{ID: 60, Name: "King Halo"},
+	{ID: 61, Name: "Machikane Tannhauser"},
+	{ID: 62, Name: "Ikuno Dictus"},
+	{ID: 63, Name: "Mejiro Palmer"},
+	{ID: 64, Name: "Daitaku Helios"},
+	{ID: 65, Name: "Twin Turbo"},
+	{ID: 66, Name: "Satono Diamond"},
+	{ID: 67, Name: "Kitasan Black"},
+	{ID: 68, Name: "Sakura Chiyono O"},
+	{ID: 69, Name: "Sirius Symboli"},
+	{ID: 70, Name: "Mejiro Ardan"},
+	{ID: 71, Name: "Yaeno Muteki"},
+	{ID: 72, Name: "Tsurumaru Tsuyoshi"},
+	{ID: 73, Name: "Mejiro Bright"},
+	{ID: 75, Name: "Sakura Laurel"},
+	{ID: 76, Name: "Narita Top Road"},
+	{ID: 77, Name: "Yamanin Zephyr"},
+	{ID: 78, Name: "Daiichi Ruby"},
+	{ID: 79, Name: "Aston Machan"},
+	{ID: 80, Name: "K.S. Miracle"},
+	{ID: 81, Name: "Copano Rickey"},
+	{ID: 82, Name: "Hokko Tarumae"},
+	{ID: 83, Name: "Wonder Acute"},
+	{ID: 84, Name: "Montjeu"},
+	{ID: 85, Name: "Hayakawa Tazuna"},
+	{ID: 86, Name: "Akikawa Yayoi(President)"},
+	{ID: 87, Name: "Otonasi Etuko"},
+	{ID: 88, Name: "Kiryuin Aoi"},
+	{ID: 89, Name: "Anshinzawa Sasami"},
+	{ID: 90, Name: "Kashimoto Rico"},
+	{ID: 91, Name: "Light Hello"},
+}

tts-gateway/internal/core/domain/symbols.go

@@ -0,0 +1,48 @@
+package domain
+
+// Symbols is the checkpoint's trained vocabulary, ported rune-for-rune from
+// text/symbols.py (`[_pad] + list(_special) + list(_punctuation) +
+// list(_letters) + _dummy`, including the literal backslash inside
+// _punctuation's raw Python triple-quoted string, and the 84 "wrong tokens"
+// placeholder entries the original training pipeline never sorted out - see
+// its own comment: "I trained with wrong tokens... these thing is for that").
+var Symbols = buildSymbols()
+
+// SymbolToID mirrors `_symbol_to_id = {s: i for i, s in enumerate(symbols)}`:
+// later duplicate runes (e.g. '-' appears in both _special and _punctuation)
+// overwrite earlier ones, matching Python dict-construction semantics exactly.
+var SymbolToID = buildSymbolToID()
+
+func buildSymbols() []rune {
+	var symbols []rune
+	symbols = append(symbols, '_')                                                                         // _pad
+	symbols = append(symbols, []rune("-~%#@&*$")...)                                                       // _special
+	symbols = append(symbols, []rune("!\"'(),.:;?{}<>\\^[]/+- ")...)                                       // _punctuation
+	symbols = append(symbols, []rune("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890")...) // _letters
+	for range 84 {
+		symbols = append(symbols, '=') // _dummy
+	}
+	return symbols
+}
+
+func buildSymbolToID() map[rune]int {
+	m := make(map[rune]int, len(Symbols))
+	for i, s := range Symbols {
+		m[s] = i
+	}
+	return m
+}
+
+// Intersperse ports commons.py::intersperse: pads seq with item between every
+// element and at both ends (used when hps.data.add_blank is true, which it
+// is for this checkpoint's configs/uma.json).
+func Intersperse(seq []int64, item int64) []int64 {
+	result := make([]int64, len(seq)*2+1)
+	for i := range result {
+		result[i] = item
+	}
+	for i, v := range seq {
+		result[i*2+1] = v
+	}
+	return result
+}

tts-gateway/internal/core/domain/symbols_test.go

@@ -0,0 +1,31 @@
+package domain
+
+import "testing"
+
+func TestIntersperse(t *testing.T) {
+	got := Intersperse([]int64{1, 2, 3}, 0)
+	want := []int64{0, 1, 0, 2, 0, 3, 0}
+	if len(got) != len(want) {
+		t.Fatalf("Intersperse() = %v, want %v", got, want)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("Intersperse() = %v, want %v", got, want)
+		}
+	}
+}
+
+func TestSymbolToIDCoversLetters(t *testing.T) {
+	for _, r := range "ABCabc012" {
+		if _, ok := SymbolToID[r]; !ok {
+			t.Errorf("SymbolToID missing letter/digit %q", r)
+		}
+	}
+}
+
+func TestSymbolToIDPad(t *testing.T) {
+	id, ok := SymbolToID['_']
+	if !ok || id != 0 {
+		t.Fatalf("SymbolToID['_'] = (%d, %v), want (0, true) - pad must be symbol ID 0", id, ok)
+	}
+}

tts-gateway/internal/core/ports/driven/driven.go

@@ -0,0 +1,25 @@
+package driven
+
+import (
+	"context"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/domain"
+)
+
+// TextNormalizer turns raw input text into the symbol-ID sequence the
+// TTSEngine expects, reproducing uma-tts-api's japanese_cleaners pipeline
+// (g2p, blank-token interspersing, char-level symbol mapping) exactly.
+type TextNormalizer interface {
+	Normalize(ctx context.Context, text string) ([]int64, error)
+}
+
+// TTSEngine runs the VITS forward pass for one speaker/phoneme-sequence
+// combination and returns raw PCM samples.
+type TTSEngine interface {
+	Synthesize(ctx context.Context, speakerID int, symbolIDs []int64, params domain.SynthesisParams) (pcm []float32, sampleRate int, err error)
+}
+
+// AudioEncoder transcodes raw PCM samples into a client-facing audio format.
+type AudioEncoder interface {
+	Encode(ctx context.Context, pcm []float32, sampleRate int) (data []byte, mimeType string, err error)
+}

tts-gateway/internal/core/ports/driving/driving.go

@@ -0,0 +1,16 @@
+package driving
+
+import (
+	"context"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/core/domain"
+)
+
+// TTSService is the use case the primary gRPC adapter drives.
+type TTSService interface {
+	// Synthesize renders speech for one speaker/text/params combination.
+	Synthesize(ctx context.Context, speakerName, text string, params domain.SynthesisParams) (*domain.AudioClip, error)
+	// ListSpeakers returns the speaker roster, optionally filtered by a
+	// case-insensitive substring match against speaker name.
+	ListSpeakers(ctx context.Context, search string) []string
+}

tts-gateway/internal/logger/logger.go

@@ -0,0 +1,38 @@
+package logger
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"os"
+)
+
+type contextKey struct{}
+
+// New constructs a root logger for the configured format and level.
+func New(format, level string) *slog.Logger {
+	var parsed slog.Level
+	if err := parsed.UnmarshalText([]byte(level)); err != nil {
+		_, _ = fmt.Fprintf(os.Stderr, "invalid log level %q, falling back to info\n", level)
+		parsed = slog.LevelInfo
+	}
+
+	opts := &slog.HandlerOptions{Level: parsed}
+	if format == "json" {
+		return slog.New(slog.NewJSONHandler(os.Stdout, opts))
+	}
+	return slog.New(slog.NewTextHandler(os.Stdout, opts))
+}
+
+// WithLogger attaches a logger to the provided context.
+func WithLogger(ctx context.Context, l *slog.Logger) context.Context {
+	return context.WithValue(ctx, contextKey{}, l)
+}
+
+// FromContext retrieves a logger from context and falls back to slog.Default().
+func FromContext(ctx context.Context) *slog.Logger {
+	if l, ok := ctx.Value(contextKey{}).(*slog.Logger); ok && l != nil {
+		return l
+	}
+	return slog.Default()
+}

tts-gateway/internal/telemetry/telemetry.go

@@ -0,0 +1,85 @@
+package telemetry
+
+import (
+	"context"
+	"errors"
+	"time"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/logger"
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
+	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
+	metricnoop "go.opentelemetry.io/otel/metric/noop"
+	"go.opentelemetry.io/otel/propagation"
+	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
+	"go.opentelemetry.io/otel/sdk/resource"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
+	tracenoop "go.opentelemetry.io/otel/trace/noop"
+
+	"gitea.nik4nao.com/nik/home-services/tts-gateway/internal/config"
+)
+
+// Setup initialises OTel trace and metric providers for one service.
+func Setup(ctx context.Context, serviceName, version string, cfg *config.Config) (shutdown func(context.Context) error, err error) {
+	if cfg.OTELEndpoint == "" {
+		otel.SetTracerProvider(tracenoop.NewTracerProvider())
+		otel.SetMeterProvider(metricnoop.NewMeterProvider())
+		logger.FromContext(ctx).Debug("otel disabled — OTEL_ENDPOINT not set")
+		return func(context.Context) error { return nil }, nil
+	}
+
+	res := resource.NewWithAttributes(
+		semconv.SchemaURL,
+		semconv.ServiceNameKey.String(serviceName),
+		semconv.ServiceVersionKey.String(version),
+	)
+
+	traceExp, err := otlptracegrpc.New(ctx,
+		otlptracegrpc.WithEndpoint(cfg.OTELEndpoint),
+		otlptracegrpc.WithInsecure(),
+	)
+	if err != nil {
+		return nil, err
+	}
+	tp := sdktrace.NewTracerProvider(
+		sdktrace.WithBatcher(traceExp),
+		sdktrace.WithResource(res),
+		sdktrace.WithSampler(sdktrace.AlwaysSample()),
+	)
+	otel.SetTracerProvider(tp)
+	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
+		propagation.TraceContext{},
+		propagation.Baggage{},
+	))
+
+	// Metric exporter.
+	metricExp, err := otlpmetricgrpc.New(ctx,
+		otlpmetricgrpc.WithEndpoint(cfg.OTELEndpoint),
+		otlpmetricgrpc.WithInsecure(),
+	)
+	if err != nil {
+		_ = tp.Shutdown(ctx)
+		return nil, err
+	}
+	mp := sdkmetric.NewMeterProvider(
+		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExp,
+			sdkmetric.WithInterval(30*time.Second))),
+		sdkmetric.WithResource(res),
+	)
+	otel.SetMeterProvider(mp)
+
+	return func(ctx context.Context) error {
+		// Shutdown is bounded so exporter flushes cannot stall process exit forever.
+		shutdownCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
+		defer cancel()
+		var shutdownErr error
+		if err := tp.Shutdown(shutdownCtx); err != nil {
+			shutdownErr = errors.Join(shutdownErr, err)
+		}
+		if err := mp.Shutdown(shutdownCtx); err != nil {
+			shutdownErr = errors.Join(shutdownErr, err)
+		}
+		return shutdownErr
+	}, nil
+}

tts-gateway/sidecar/Dockerfile

@@ -0,0 +1,27 @@
+# Dev/smoke-test image for the tts-gateway inference sidecar (Phase 3/4 of
+# TTS_GATEWAY_PLAN.md). NOT the polished production image - Phase 5 (deferred)
+# covers the final containerization/CI/model-artifact-distribution decisions.
+# Base image matches nik-gpu's driver 570.211.01 / CUDA 12.8.
+FROM pytorch/pytorch:2.9.1-cuda12.8-cudnn9-runtime
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    cmake \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY . .
+
+# models/models.py does a module-level `import monotonic_align`, even though
+# this sidecar's infer()-only path never calls it (only the training-time
+# forward() does) - the extension still has to build for the import itself to
+# succeed. See tmp/reference/uma-tts-api/spike/FINDINGS.md for the identical
+# issue hit during the Phase 0 spike.
+RUN cd monotonic_align && python setup.py build_ext --inplace
+
+EXPOSE 50054
+CMD ["python", "server.py"]

tts-gateway/sidecar/commons.py

@@ -0,0 +1,161 @@
+import math
+import numpy as np
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+
+def init_weights(m, mean=0.0, std=0.01):
+  classname = m.__class__.__name__
+  if classname.find("Conv") != -1:
+    m.weight.data.normal_(mean, std)
+
+
+def get_padding(kernel_size, dilation=1):
+  return int((kernel_size*dilation - dilation)/2)
+
+
+def convert_pad_shape(pad_shape):
+  l = pad_shape[::-1]
+  pad_shape = [item for sublist in l for item in sublist]
+  return pad_shape
+
+
+def intersperse(lst, item):
+  result = [item] * (len(lst) * 2 + 1)
+  result[1::2] = lst
+  return result
+
+
+def kl_divergence(m_p, logs_p, m_q, logs_q):
+  """KL(P||Q)"""
+  kl = (logs_q - logs_p) - 0.5
+  kl += 0.5 * (torch.exp(2. * logs_p) + ((m_p - m_q)**2)) * torch.exp(-2. * logs_q)
+  return kl
+
+
+def rand_gumbel(shape):
+  """Sample from the Gumbel distribution, protect from overflows."""
+  uniform_samples = torch.rand(shape) * 0.99998 + 0.00001
+  return -torch.log(-torch.log(uniform_samples))
+
+
+def rand_gumbel_like(x):
+  g = rand_gumbel(x.size()).to(dtype=x.dtype, device=x.device)
+  return g
+
+
+def slice_segments(x, ids_str, segment_size=4):
+  ret = torch.zeros_like(x[:, :, :segment_size])
+  for i in range(x.size(0)):
+    idx_str = ids_str[i]
+    idx_end = idx_str + segment_size
+    ret[i] = x[i, :, idx_str:idx_end]
+  return ret
+
+
+def rand_slice_segments(x, x_lengths=None, segment_size=4):
+  b, d, t = x.size()
+  if x_lengths is None:
+    x_lengths = t
+  ids_str_max = x_lengths - segment_size + 1
+  ids_str = (torch.rand([b]).to(device=x.device) * ids_str_max).to(dtype=torch.long)
+  ret = slice_segments(x, ids_str, segment_size)
+  return ret, ids_str
+
+
+def get_timing_signal_1d(
+    length, channels, min_timescale=1.0, max_timescale=1.0e4):
+  position = torch.arange(length, dtype=torch.float)
+  num_timescales = channels // 2
+  log_timescale_increment = (
+      math.log(float(max_timescale) / float(min_timescale)) /
+      (num_timescales - 1))
+  inv_timescales = min_timescale * torch.exp(
+      torch.arange(num_timescales, dtype=torch.float) * -log_timescale_increment)
+  scaled_time = position.unsqueeze(0) * inv_timescales.unsqueeze(1)
+  signal = torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], 0)
+  signal = F.pad(signal, [0, 0, 0, channels % 2])
+  signal = signal.view(1, channels, length)
+  return signal
+
+
+def add_timing_signal_1d(x, min_timescale=1.0, max_timescale=1.0e4):
+  b, channels, length = x.size()
+  signal = get_timing_signal_1d(length, channels, min_timescale, max_timescale)
+  return x + signal.to(dtype=x.dtype, device=x.device)
+
+
+def cat_timing_signal_1d(x, min_timescale=1.0, max_timescale=1.0e4, axis=1):
+  b, channels, length = x.size()
+  signal = get_timing_signal_1d(length, channels, min_timescale, max_timescale)
+  return torch.cat([x, signal.to(dtype=x.dtype, device=x.device)], axis)
+
+
+def subsequent_mask(length):
+  mask = torch.tril(torch.ones(length, length)).unsqueeze(0).unsqueeze(0)
+  return mask
+
+
+@torch.jit.script
+def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
+  n_channels_int = n_channels[0]
+  in_act = input_a + input_b
+  t_act = torch.tanh(in_act[:, :n_channels_int, :])
+  s_act = torch.sigmoid(in_act[:, n_channels_int:, :])
+  acts = t_act * s_act
+  return acts
+
+
+def convert_pad_shape(pad_shape):
+  l = pad_shape[::-1]
+  pad_shape = [item for sublist in l for item in sublist]
+  return pad_shape
+
+
+def shift_1d(x):
+  x = F.pad(x, convert_pad_shape([[0, 0], [0, 0], [1, 0]]))[:, :, :-1]
+  return x
+
+
+def sequence_mask(length, max_length=None):
+  if max_length is None:
+    max_length = length.max()
+  x = torch.arange(max_length, dtype=length.dtype, device=length.device)
+  return x.unsqueeze(0) < length.unsqueeze(1)
+
+
+def generate_path(duration, mask):
+  """
+  duration: [b, 1, t_x]
+  mask: [b, 1, t_y, t_x]
+  """
+  device = duration.device
+  
+  b, _, t_y, t_x = mask.shape
+  cum_duration = torch.cumsum(duration, -1)
+  
+  cum_duration_flat = cum_duration.view(b * t_x)
+  path = sequence_mask(cum_duration_flat, t_y).to(mask.dtype)
+  path = path.view(b, t_x, t_y)
+  path = path - F.pad(path, convert_pad_shape([[0, 0], [1, 0], [0, 0]]))[:, :-1]
+  path = path.unsqueeze(1).transpose(2,3) * mask
+  return path
+
+
+def clip_grad_value_(parameters, clip_value, norm_type=2):
+  if isinstance(parameters, torch.Tensor):
+    parameters = [parameters]
+  parameters = list(filter(lambda p: p.grad is not None, parameters))
+  norm_type = float(norm_type)
+  if clip_value is not None:
+    clip_value = float(clip_value)
+
+  total_norm = 0
+  for p in parameters:
+    param_norm = p.grad.data.norm(norm_type)
+    total_norm += param_norm.item() ** norm_type
+    if clip_value is not None:
+      p.grad.data.clamp_(min=-clip_value, max=clip_value)
+  total_norm = total_norm ** (1. / norm_type)
+  return total_norm

tts-gateway/sidecar/configs/uma.json

@@ -0,0 +1,53 @@
+{
+  "train": {
+    "log_interval": 200,
+    "eval_interval": 1000,
+    "seed": 1234,
+    "epochs": 10000,
+    "learning_rate": 2e-4,
+    "betas": [0.8, 0.99],
+    "eps": 1e-9,
+    "batch_size": 12,
+    "fp16_run": true,
+    "lr_decay": 0.999875,
+    "segment_size": 8192,
+    "init_lr_ratio": 1,
+    "warmup_epochs": 0,
+    "c_mel": 45,
+    "c_kl": 1.0
+  },
+  "data": {
+    "training_files":"filelists/uma_text_train.txt.cleaned",
+    "validation_files":"filelists/uma_text_val.txt.cleaned",
+    "text_cleaners":["japanese_cleaners"],
+    "max_wav_value": 32768.0,
+    "sampling_rate": 22050,
+    "filter_length": 1024,
+    "hop_length": 256,
+    "win_length": 1024,
+    "n_mel_channels": 80,
+    "mel_fmin": 0.0,
+    "mel_fmax": null,
+    "add_blank": true,
+    "n_speakers": 92,
+    "cleaned_text": true
+  },
+  "model": {
+    "inter_channels": 192,
+    "hidden_channels": 192,
+    "filter_channels": 768,
+    "n_heads": 2,
+    "n_layers": 6,
+    "kernel_size": 3,
+    "p_dropout": 0.1,
+    "resblock": "1",
+    "resblock_kernel_sizes": [3,7,11],
+    "resblock_dilation_sizes": [[1,3,5], [1,3,5], [1,3,5]],
+    "upsample_rates": [8,8,2,2],
+    "upsample_initial_channel": 512,
+    "upsample_kernel_sizes": [16,16,4,4],
+    "n_layers_q": 3,
+    "use_spectral_norm": false,
+    "gin_channels": 256
+  }
+}

tts-gateway/sidecar/models/attentions.py

@@ -0,0 +1,303 @@
+import copy
+import math
+import numpy as np
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+import commons
+from models import modules
+from models.modules import LayerNorm
+   
+
+class Encoder(nn.Module):
+  def __init__(self, hidden_channels, filter_channels, n_heads, n_layers, kernel_size=1, p_dropout=0., window_size=4, **kwargs):
+    super().__init__()
+    self.hidden_channels = hidden_channels
+    self.filter_channels = filter_channels
+    self.n_heads = n_heads
+    self.n_layers = n_layers
+    self.kernel_size = kernel_size
+    self.p_dropout = p_dropout
+    self.window_size = window_size
+
+    self.drop = nn.Dropout(p_dropout)
+    self.attn_layers = nn.ModuleList()
+    self.norm_layers_1 = nn.ModuleList()
+    self.ffn_layers = nn.ModuleList()
+    self.norm_layers_2 = nn.ModuleList()
+    for i in range(self.n_layers):
+      self.attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout, window_size=window_size))
+      self.norm_layers_1.append(LayerNorm(hidden_channels))
+      self.ffn_layers.append(FFN(hidden_channels, hidden_channels, filter_channels, kernel_size, p_dropout=p_dropout))
+      self.norm_layers_2.append(LayerNorm(hidden_channels))
+
+  def forward(self, x, x_mask):
+    attn_mask = x_mask.unsqueeze(2) * x_mask.unsqueeze(-1)
+    x = x * x_mask
+    for i in range(self.n_layers):
+      y = self.attn_layers[i](x, x, attn_mask)
+      y = self.drop(y)
+      x = self.norm_layers_1[i](x + y)
+
+      y = self.ffn_layers[i](x, x_mask)
+      y = self.drop(y)
+      x = self.norm_layers_2[i](x + y)
+    x = x * x_mask
+    return x
+
+
+class Decoder(nn.Module):
+  def __init__(self, hidden_channels, filter_channels, n_heads, n_layers, kernel_size=1, p_dropout=0., proximal_bias=False, proximal_init=True, **kwargs):
+    super().__init__()
+    self.hidden_channels = hidden_channels
+    self.filter_channels = filter_channels
+    self.n_heads = n_heads
+    self.n_layers = n_layers
+    self.kernel_size = kernel_size
+    self.p_dropout = p_dropout
+    self.proximal_bias = proximal_bias
+    self.proximal_init = proximal_init
+
+    self.drop = nn.Dropout(p_dropout)
+    self.self_attn_layers = nn.ModuleList()
+    self.norm_layers_0 = nn.ModuleList()
+    self.encdec_attn_layers = nn.ModuleList()
+    self.norm_layers_1 = nn.ModuleList()
+    self.ffn_layers = nn.ModuleList()
+    self.norm_layers_2 = nn.ModuleList()
+    for i in range(self.n_layers):
+      self.self_attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout, proximal_bias=proximal_bias, proximal_init=proximal_init))
+      self.norm_layers_0.append(LayerNorm(hidden_channels))
+      self.encdec_attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout))
+      self.norm_layers_1.append(LayerNorm(hidden_channels))
+      self.ffn_layers.append(FFN(hidden_channels, hidden_channels, filter_channels, kernel_size, p_dropout=p_dropout, causal=True))
+      self.norm_layers_2.append(LayerNorm(hidden_channels))
+
+  def forward(self, x, x_mask, h, h_mask):
+    """
+    x: decoder input
+    h: encoder output
+    """
+    self_attn_mask = commons.subsequent_mask(x_mask.size(2)).to(device=x.device, dtype=x.dtype)
+    encdec_attn_mask = h_mask.unsqueeze(2) * x_mask.unsqueeze(-1)
+    x = x * x_mask
+    for i in range(self.n_layers):
+      y = self.self_attn_layers[i](x, x, self_attn_mask)
+      y = self.drop(y)
+      x = self.norm_layers_0[i](x + y)
+
+      y = self.encdec_attn_layers[i](x, h, encdec_attn_mask)
+      y = self.drop(y)
+      x = self.norm_layers_1[i](x + y)
+      
+      y = self.ffn_layers[i](x, x_mask)
+      y = self.drop(y)
+      x = self.norm_layers_2[i](x + y)
+    x = x * x_mask
+    return x
+
+
+class MultiHeadAttention(nn.Module):
+  def __init__(self, channels, out_channels, n_heads, p_dropout=0., window_size=None, heads_share=True, block_length=None, proximal_bias=False, proximal_init=False):
+    super().__init__()
+    assert channels % n_heads == 0
+
+    self.channels = channels
+    self.out_channels = out_channels
+    self.n_heads = n_heads
+    self.p_dropout = p_dropout
+    self.window_size = window_size
+    self.heads_share = heads_share
+    self.block_length = block_length
+    self.proximal_bias = proximal_bias
+    self.proximal_init = proximal_init
+    self.attn = None
+
+    self.k_channels = channels // n_heads
+    self.conv_q = nn.Conv1d(channels, channels, 1)
+    self.conv_k = nn.Conv1d(channels, channels, 1)
+    self.conv_v = nn.Conv1d(channels, channels, 1)
+    self.conv_o = nn.Conv1d(channels, out_channels, 1)
+    self.drop = nn.Dropout(p_dropout)
+
+    if window_size is not None:
+      n_heads_rel = 1 if heads_share else n_heads
+      rel_stddev = self.k_channels**-0.5
+      self.emb_rel_k = nn.Parameter(torch.randn(n_heads_rel, window_size * 2 + 1, self.k_channels) * rel_stddev)
+      self.emb_rel_v = nn.Parameter(torch.randn(n_heads_rel, window_size * 2 + 1, self.k_channels) * rel_stddev)
+
+    nn.init.xavier_uniform_(self.conv_q.weight)
+    nn.init.xavier_uniform_(self.conv_k.weight)
+    nn.init.xavier_uniform_(self.conv_v.weight)
+    if proximal_init:
+      with torch.no_grad():
+        self.conv_k.weight.copy_(self.conv_q.weight)
+        self.conv_k.bias.copy_(self.conv_q.bias)
+      
+  def forward(self, x, c, attn_mask=None):
+    q = self.conv_q(x)
+    k = self.conv_k(c)
+    v = self.conv_v(c)
+    
+    x, self.attn = self.attention(q, k, v, mask=attn_mask)
+
+    x = self.conv_o(x)
+    return x
+
+  def attention(self, query, key, value, mask=None):
+    # reshape [b, d, t] -> [b, n_h, t, d_k]
+    b, d, t_s, t_t = (*key.size(), query.size(2))
+    query = query.view(b, self.n_heads, self.k_channels, t_t).transpose(2, 3)
+    key = key.view(b, self.n_heads, self.k_channels, t_s).transpose(2, 3)
+    value = value.view(b, self.n_heads, self.k_channels, t_s).transpose(2, 3)
+
+    scores = torch.matmul(query / math.sqrt(self.k_channels), key.transpose(-2, -1))
+    if self.window_size is not None:
+      assert t_s == t_t, "Relative attention is only available for self-attention."
+      key_relative_embeddings = self._get_relative_embeddings(self.emb_rel_k, t_s)
+      rel_logits = self._matmul_with_relative_keys(query /math.sqrt(self.k_channels), key_relative_embeddings)
+      scores_local = self._relative_position_to_absolute_position(rel_logits)
+      scores = scores + scores_local
+    if self.proximal_bias:
+      assert t_s == t_t, "Proximal bias is only available for self-attention."
+      scores = scores + self._attention_bias_proximal(t_s).to(device=scores.device, dtype=scores.dtype)
+    if mask is not None:
+      scores = scores.masked_fill(mask == 0, -1e4)
+      if self.block_length is not None:
+        assert t_s == t_t, "Local attention is only available for self-attention."
+        block_mask = torch.ones_like(scores).triu(-self.block_length).tril(self.block_length)
+        scores = scores.masked_fill(block_mask == 0, -1e4)
+    p_attn = F.softmax(scores, dim=-1) # [b, n_h, t_t, t_s]
+    p_attn = self.drop(p_attn)
+    output = torch.matmul(p_attn, value)
+    if self.window_size is not None:
+      relative_weights = self._absolute_position_to_relative_position(p_attn)
+      value_relative_embeddings = self._get_relative_embeddings(self.emb_rel_v, t_s)
+      output = output + self._matmul_with_relative_values(relative_weights, value_relative_embeddings)
+    output = output.transpose(2, 3).contiguous().view(b, d, t_t) # [b, n_h, t_t, d_k] -> [b, d, t_t]
+    return output, p_attn
+
+  def _matmul_with_relative_values(self, x, y):
+    """
+    x: [b, h, l, m]
+    y: [h or 1, m, d]
+    ret: [b, h, l, d]
+    """
+    ret = torch.matmul(x, y.unsqueeze(0))
+    return ret
+
+  def _matmul_with_relative_keys(self, x, y):
+    """
+    x: [b, h, l, d]
+    y: [h or 1, m, d]
+    ret: [b, h, l, m]
+    """
+    ret = torch.matmul(x, y.unsqueeze(0).transpose(-2, -1))
+    return ret
+
+  def _get_relative_embeddings(self, relative_embeddings, length):
+    max_relative_position = 2 * self.window_size + 1
+    # Pad first before slice to avoid using cond ops.
+    pad_length = max(length - (self.window_size + 1), 0)
+    slice_start_position = max((self.window_size + 1) - length, 0)
+    slice_end_position = slice_start_position + 2 * length - 1
+    if pad_length > 0:
+      padded_relative_embeddings = F.pad(
+          relative_embeddings,
+          commons.convert_pad_shape([[0, 0], [pad_length, pad_length], [0, 0]]))
+    else:
+      padded_relative_embeddings = relative_embeddings
+    used_relative_embeddings = padded_relative_embeddings[:,slice_start_position:slice_end_position]
+    return used_relative_embeddings
+
+  def _relative_position_to_absolute_position(self, x):
+    """
+    x: [b, h, l, 2*l-1]
+    ret: [b, h, l, l]
+    """
+    batch, heads, length, _ = x.size()
+    # Concat columns of pad to shift from relative to absolute indexing.
+    x = F.pad(x, commons.convert_pad_shape([[0,0],[0,0],[0,0],[0,1]]))
+
+    # Concat extra elements so to add up to shape (len+1, 2*len-1).
+    x_flat = x.view([batch, heads, length * 2 * length])
+    x_flat = F.pad(x_flat, commons.convert_pad_shape([[0,0],[0,0],[0,length-1]]))
+
+    # Reshape and slice out the padded elements.
+    x_final = x_flat.view([batch, heads, length+1, 2*length-1])[:, :, :length, length-1:]
+    return x_final
+
+  def _absolute_position_to_relative_position(self, x):
+    """
+    x: [b, h, l, l]
+    ret: [b, h, l, 2*l-1]
+    """
+    batch, heads, length, _ = x.size()
+    # padd along column
+    x = F.pad(x, commons.convert_pad_shape([[0, 0], [0, 0], [0, 0], [0, length-1]]))
+    x_flat = x.view([batch, heads, length**2 + length*(length -1)])
+    # add 0's in the beginning that will skew the elements after reshape
+    x_flat = F.pad(x_flat, commons.convert_pad_shape([[0, 0], [0, 0], [length, 0]]))
+    x_final = x_flat.view([batch, heads, length, 2*length])[:,:,:,1:]
+    return x_final
+
+  def _attention_bias_proximal(self, length):
+    """Bias for self-attention to encourage attention to close positions.
+    Args:
+      length: an integer scalar.
+    Returns:
+      a Tensor with shape [1, 1, length, length]
+    """
+    r = torch.arange(length, dtype=torch.float32)
+    diff = torch.unsqueeze(r, 0) - torch.unsqueeze(r, 1)
+    return torch.unsqueeze(torch.unsqueeze(-torch.log1p(torch.abs(diff)), 0), 0)
+
+
+class FFN(nn.Module):
+  def __init__(self, in_channels, out_channels, filter_channels, kernel_size, p_dropout=0., activation=None, causal=False):
+    super().__init__()
+    self.in_channels = in_channels
+    self.out_channels = out_channels
+    self.filter_channels = filter_channels
+    self.kernel_size = kernel_size
+    self.p_dropout = p_dropout
+    self.activation = activation
+    self.causal = causal
+
+    if causal:
+      self.padding = self._causal_padding
+    else:
+      self.padding = self._same_padding
+
+    self.conv_1 = nn.Conv1d(in_channels, filter_channels, kernel_size)
+    self.conv_2 = nn.Conv1d(filter_channels, out_channels, kernel_size)
+    self.drop = nn.Dropout(p_dropout)
+
+  def forward(self, x, x_mask):
+    x = self.conv_1(self.padding(x * x_mask))
+    if self.activation == "gelu":
+      x = x * torch.sigmoid(1.702 * x)
+    else:
+      x = torch.relu(x)
+    x = self.drop(x)
+    x = self.conv_2(self.padding(x * x_mask))
+    return x * x_mask
+  
+  def _causal_padding(self, x):
+    if self.kernel_size == 1:
+      return x
+    pad_l = self.kernel_size - 1
+    pad_r = 0
+    padding = [[0, 0], [0, 0], [pad_l, pad_r]]
+    x = F.pad(x, commons.convert_pad_shape(padding))
+    return x
+
+  def _same_padding(self, x):
+    if self.kernel_size == 1:
+      return x
+    pad_l = (self.kernel_size - 1) // 2
+    pad_r = self.kernel_size // 2
+    padding = [[0, 0], [0, 0], [pad_l, pad_r]]
+    x = F.pad(x, commons.convert_pad_shape(padding))
+    return x

tts-gateway/sidecar/models/models.py

@@ -0,0 +1,533 @@
+import copy
+import math
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+import commons
+from models import modules, attentions
+import monotonic_align
+
+from torch.nn import Conv1d, ConvTranspose1d, AvgPool1d, Conv2d
+from torch.nn.utils import weight_norm, remove_weight_norm, spectral_norm
+from commons import init_weights, get_padding
+
+
+class StochasticDurationPredictor(nn.Module):
+  def __init__(self, in_channels, filter_channels, kernel_size, p_dropout, n_flows=4, gin_channels=0):
+    super().__init__()
+    filter_channels = in_channels # it needs to be removed from future version.
+    self.in_channels = in_channels
+    self.filter_channels = filter_channels
+    self.kernel_size = kernel_size
+    self.p_dropout = p_dropout
+    self.n_flows = n_flows
+    self.gin_channels = gin_channels
+
+    self.log_flow = modules.Log()
+    self.flows = nn.ModuleList()
+    self.flows.append(modules.ElementwiseAffine(2))
+    for i in range(n_flows):
+      self.flows.append(modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3))
+      self.flows.append(modules.Flip())
+
+    self.post_pre = nn.Conv1d(1, filter_channels, 1)
+    self.post_proj = nn.Conv1d(filter_channels, filter_channels, 1)
+    self.post_convs = modules.DDSConv(filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout)
+    self.post_flows = nn.ModuleList()
+    self.post_flows.append(modules.ElementwiseAffine(2))
+    for i in range(4):
+      self.post_flows.append(modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3))
+      self.post_flows.append(modules.Flip())
+
+    self.pre = nn.Conv1d(in_channels, filter_channels, 1)
+    self.proj = nn.Conv1d(filter_channels, filter_channels, 1)
+    self.convs = modules.DDSConv(filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout)
+    if gin_channels != 0:
+      self.cond = nn.Conv1d(gin_channels, filter_channels, 1)
+
+  def forward(self, x, x_mask, w=None, g=None, reverse=False, noise_scale=1.0):
+    x = torch.detach(x)
+    x = self.pre(x)
+    if g is not None:
+      g = torch.detach(g)
+      x = x + self.cond(g)
+    x = self.convs(x, x_mask)
+    x = self.proj(x) * x_mask
+
+    if not reverse:
+      flows = self.flows
+      assert w is not None
+
+      logdet_tot_q = 0 
+      h_w = self.post_pre(w)
+      h_w = self.post_convs(h_w, x_mask)
+      h_w = self.post_proj(h_w) * x_mask
+      e_q = torch.randn(w.size(0), 2, w.size(2)).to(device=x.device, dtype=x.dtype) * x_mask
+      z_q = e_q
+      for flow in self.post_flows:
+        z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w))
+        logdet_tot_q += logdet_q
+      z_u, z1 = torch.split(z_q, [1, 1], 1) 
+      u = torch.sigmoid(z_u) * x_mask
+      z0 = (w - u) * x_mask
+      logdet_tot_q += torch.sum((F.logsigmoid(z_u) + F.logsigmoid(-z_u)) * x_mask, [1,2])
+      logq = torch.sum(-0.5 * (math.log(2*math.pi) + (e_q**2)) * x_mask, [1,2]) - logdet_tot_q
+
+      logdet_tot = 0
+      z0, logdet = self.log_flow(z0, x_mask)
+      logdet_tot += logdet
+      z = torch.cat([z0, z1], 1)
+      for flow in flows:
+        z, logdet = flow(z, x_mask, g=x, reverse=reverse)
+        logdet_tot = logdet_tot + logdet
+      nll = torch.sum(0.5 * (math.log(2*math.pi) + (z**2)) * x_mask, [1,2]) - logdet_tot
+      return nll + logq # [b]
+    else:
+      flows = list(reversed(self.flows))
+      flows = flows[:-2] + [flows[-1]] # remove a useless vflow
+      z = torch.randn(x.size(0), 2, x.size(2)).to(device=x.device, dtype=x.dtype) * noise_scale
+      for flow in flows:
+        z = flow(z, x_mask, g=x, reverse=reverse)
+      z0, z1 = torch.split(z, [1, 1], 1)
+      logw = z0
+      return logw
+
+
+class DurationPredictor(nn.Module):
+  def __init__(self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0):
+    super().__init__()
+
+    self.in_channels = in_channels
+    self.filter_channels = filter_channels
+    self.kernel_size = kernel_size
+    self.p_dropout = p_dropout
+    self.gin_channels = gin_channels
+
+    self.drop = nn.Dropout(p_dropout)
+    self.conv_1 = nn.Conv1d(in_channels, filter_channels, kernel_size, padding=kernel_size//2)
+    self.norm_1 = modules.LayerNorm(filter_channels)
+    self.conv_2 = nn.Conv1d(filter_channels, filter_channels, kernel_size, padding=kernel_size//2)
+    self.norm_2 = modules.LayerNorm(filter_channels)
+    self.proj = nn.Conv1d(filter_channels, 1, 1)
+
+    if gin_channels != 0:
+      self.cond = nn.Conv1d(gin_channels, in_channels, 1)
+
+  def forward(self, x, x_mask, g=None):
+    x = torch.detach(x)
+    if g is not None:
+      g = torch.detach(g)
+      x = x + self.cond(g)
+    x = self.conv_1(x * x_mask)
+    x = torch.relu(x)
+    x = self.norm_1(x)
+    x = self.drop(x)
+    x = self.conv_2(x * x_mask)
+    x = torch.relu(x)
+    x = self.norm_2(x)
+    x = self.drop(x)
+    x = self.proj(x * x_mask)
+    return x * x_mask
+
+
+class TextEncoder(nn.Module):
+  def __init__(self,
+      n_vocab,
+      out_channels,
+      hidden_channels,
+      filter_channels,
+      n_heads,
+      n_layers,
+      kernel_size,
+      p_dropout):
+    super().__init__()
+    self.n_vocab = n_vocab
+    self.out_channels = out_channels
+    self.hidden_channels = hidden_channels
+    self.filter_channels = filter_channels
+    self.n_heads = n_heads
+    self.n_layers = n_layers
+    self.kernel_size = kernel_size
+    self.p_dropout = p_dropout
+
+    self.emb = nn.Embedding(n_vocab, hidden_channels)
+    nn.init.normal_(self.emb.weight, 0.0, hidden_channels**-0.5)
+
+    self.encoder = attentions.Encoder(
+      hidden_channels,
+      filter_channels,
+      n_heads,
+      n_layers,
+      kernel_size,
+      p_dropout)
+    self.proj= nn.Conv1d(hidden_channels, out_channels * 2, 1)
+
+  def forward(self, x, x_lengths):
+    x = self.emb(x) * math.sqrt(self.hidden_channels) # [b, t, h]
+    x = torch.transpose(x, 1, -1) # [b, h, t]
+    x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(x.dtype)
+
+    x = self.encoder(x * x_mask, x_mask)
+    stats = self.proj(x) * x_mask
+
+    m, logs = torch.split(stats, self.out_channels, dim=1)
+    return x, m, logs, x_mask
+
+
+class ResidualCouplingBlock(nn.Module):
+  def __init__(self,
+      channels,
+      hidden_channels,
+      kernel_size,
+      dilation_rate,
+      n_layers,
+      n_flows=4,
+      gin_channels=0):
+    super().__init__()
+    self.channels = channels
+    self.hidden_channels = hidden_channels
+    self.kernel_size = kernel_size
+    self.dilation_rate = dilation_rate
+    self.n_layers = n_layers
+    self.n_flows = n_flows
+    self.gin_channels = gin_channels
+
+    self.flows = nn.ModuleList()
+    for i in range(n_flows):
+      self.flows.append(modules.ResidualCouplingLayer(channels, hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=gin_channels, mean_only=True))
+      self.flows.append(modules.Flip())
+
+  def forward(self, x, x_mask, g=None, reverse=False):
+    if not reverse:
+      for flow in self.flows:
+        x, _ = flow(x, x_mask, g=g, reverse=reverse)
+    else:
+      for flow in reversed(self.flows):
+        x = flow(x, x_mask, g=g, reverse=reverse)
+    return x
+
+
+class PosteriorEncoder(nn.Module):
+  def __init__(self,
+      in_channels,
+      out_channels,
+      hidden_channels,
+      kernel_size,
+      dilation_rate,
+      n_layers,
+      gin_channels=0):
+    super().__init__()
+    self.in_channels = in_channels
+    self.out_channels = out_channels
+    self.hidden_channels = hidden_channels
+    self.kernel_size = kernel_size
+    self.dilation_rate = dilation_rate
+    self.n_layers = n_layers
+    self.gin_channels = gin_channels
+
+    self.pre = nn.Conv1d(in_channels, hidden_channels, 1)
+    self.enc = modules.WN(hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=gin_channels)
+    self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
+
+  def forward(self, x, x_lengths, g=None):
+    x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(x.dtype)
+    x = self.pre(x) * x_mask
+    x = self.enc(x, x_mask, g=g)
+    stats = self.proj(x) * x_mask
+    m, logs = torch.split(stats, self.out_channels, dim=1)
+    z = (m + torch.randn_like(m) * torch.exp(logs)) * x_mask
+    return z, m, logs, x_mask
+
+
+class Generator(torch.nn.Module):
+    def __init__(self, initial_channel, resblock, resblock_kernel_sizes, resblock_dilation_sizes, upsample_rates, upsample_initial_channel, upsample_kernel_sizes, gin_channels=0):
+        super(Generator, self).__init__()
+        self.num_kernels = len(resblock_kernel_sizes)
+        self.num_upsamples = len(upsample_rates)
+        self.conv_pre = Conv1d(initial_channel, upsample_initial_channel, 7, 1, padding=3)
+        resblock = modules.ResBlock1 if resblock == '1' else modules.ResBlock2
+
+        self.ups = nn.ModuleList()
+        for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
+            self.ups.append(weight_norm(
+                ConvTranspose1d(upsample_initial_channel//(2**i), upsample_initial_channel//(2**(i+1)),
+                                k, u, padding=(k-u)//2)))
+
+        self.resblocks = nn.ModuleList()
+        for i in range(len(self.ups)):
+            ch = upsample_initial_channel//(2**(i+1))
+            for j, (k, d) in enumerate(zip(resblock_kernel_sizes, resblock_dilation_sizes)):
+                self.resblocks.append(resblock(ch, k, d))
+
+        self.conv_post = Conv1d(ch, 1, 7, 1, padding=3, bias=False)
+        self.ups.apply(init_weights)
+
+        if gin_channels != 0:
+            self.cond = nn.Conv1d(gin_channels, upsample_initial_channel, 1)
+
+    def forward(self, x, g=None):
+        x = self.conv_pre(x)
+        if g is not None:
+          x = x + self.cond(g)
+
+        for i in range(self.num_upsamples):
+            x = F.leaky_relu(x, modules.LRELU_SLOPE)
+            x = self.ups[i](x)
+            xs = None
+            for j in range(self.num_kernels):
+                if xs is None:
+                    xs = self.resblocks[i*self.num_kernels+j](x)
+                else:
+                    xs += self.resblocks[i*self.num_kernels+j](x)
+            x = xs / self.num_kernels
+        x = F.leaky_relu(x)
+        x = self.conv_post(x)
+        x = torch.tanh(x)
+
+        return x
+
+    def remove_weight_norm(self):
+        print('Removing weight norm...')
+        for l in self.ups:
+            remove_weight_norm(l)
+        for l in self.resblocks:
+            l.remove_weight_norm()
+
+
+class DiscriminatorP(torch.nn.Module):
+    def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False):
+        super(DiscriminatorP, self).__init__()
+        self.period = period
+        self.use_spectral_norm = use_spectral_norm
+        norm_f = weight_norm if use_spectral_norm == False else spectral_norm
+        self.convs = nn.ModuleList([
+            norm_f(Conv2d(1, 32, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))),
+            norm_f(Conv2d(32, 128, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))),
+            norm_f(Conv2d(128, 512, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))),
+            norm_f(Conv2d(512, 1024, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))),
+            norm_f(Conv2d(1024, 1024, (kernel_size, 1), 1, padding=(get_padding(kernel_size, 1), 0))),
+        ])
+        self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0)))
+
+    def forward(self, x):
+        fmap = []
+
+        # 1d to 2d
+        b, c, t = x.shape
+        if t % self.period != 0: # pad first
+            n_pad = self.period - (t % self.period)
+            x = F.pad(x, (0, n_pad), "reflect")
+            t = t + n_pad
+        x = x.view(b, c, t // self.period, self.period)
+
+        for l in self.convs:
+            x = l(x)
+            x = F.leaky_relu(x, modules.LRELU_SLOPE)
+            fmap.append(x)
+        x = self.conv_post(x)
+        fmap.append(x)
+        x = torch.flatten(x, 1, -1)
+
+        return x, fmap
+
+
+class DiscriminatorS(torch.nn.Module):
+    def __init__(self, use_spectral_norm=False):
+        super(DiscriminatorS, self).__init__()
+        norm_f = weight_norm if use_spectral_norm == False else spectral_norm
+        self.convs = nn.ModuleList([
+            norm_f(Conv1d(1, 16, 15, 1, padding=7)),
+            norm_f(Conv1d(16, 64, 41, 4, groups=4, padding=20)),
+            norm_f(Conv1d(64, 256, 41, 4, groups=16, padding=20)),
+            norm_f(Conv1d(256, 1024, 41, 4, groups=64, padding=20)),
+            norm_f(Conv1d(1024, 1024, 41, 4, groups=256, padding=20)),
+            norm_f(Conv1d(1024, 1024, 5, 1, padding=2)),
+        ])
+        self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1))
+
+    def forward(self, x):
+        fmap = []
+
+        for l in self.convs:
+            x = l(x)
+            x = F.leaky_relu(x, modules.LRELU_SLOPE)
+            fmap.append(x)
+        x = self.conv_post(x)
+        fmap.append(x)
+        x = torch.flatten(x, 1, -1)
+
+        return x, fmap
+
+
+class MultiPeriodDiscriminator(torch.nn.Module):
+    def __init__(self, use_spectral_norm=False):
+        super(MultiPeriodDiscriminator, self).__init__()
+        periods = [2,3,5,7,11]
+
+        discs = [DiscriminatorS(use_spectral_norm=use_spectral_norm)]
+        discs = discs + [DiscriminatorP(i, use_spectral_norm=use_spectral_norm) for i in periods]
+        self.discriminators = nn.ModuleList(discs)
+
+    def forward(self, y, y_hat):
+        y_d_rs = []
+        y_d_gs = []
+        fmap_rs = []
+        fmap_gs = []
+        for i, d in enumerate(self.discriminators):
+            y_d_r, fmap_r = d(y)
+            y_d_g, fmap_g = d(y_hat)
+            y_d_rs.append(y_d_r)
+            y_d_gs.append(y_d_g)
+            fmap_rs.append(fmap_r)
+            fmap_gs.append(fmap_g)
+
+        return y_d_rs, y_d_gs, fmap_rs, fmap_gs
+
+
+
+class SynthesizerTrn(nn.Module):
+  """
+  Synthesizer for Training
+  """
+
+  def __init__(self, 
+    n_vocab,
+    spec_channels,
+    segment_size,
+    inter_channels,
+    hidden_channels,
+    filter_channels,
+    n_heads,
+    n_layers,
+    kernel_size,
+    p_dropout,
+    resblock, 
+    resblock_kernel_sizes, 
+    resblock_dilation_sizes, 
+    upsample_rates, 
+    upsample_initial_channel, 
+    upsample_kernel_sizes,
+    n_speakers=0,
+    gin_channels=0,
+    use_sdp=True,
+    **kwargs):
+
+    super().__init__()
+    self.n_vocab = n_vocab
+    self.spec_channels = spec_channels
+    self.inter_channels = inter_channels
+    self.hidden_channels = hidden_channels
+    self.filter_channels = filter_channels
+    self.n_heads = n_heads
+    self.n_layers = n_layers
+    self.kernel_size = kernel_size
+    self.p_dropout = p_dropout
+    self.resblock = resblock
+    self.resblock_kernel_sizes = resblock_kernel_sizes
+    self.resblock_dilation_sizes = resblock_dilation_sizes
+    self.upsample_rates = upsample_rates
+    self.upsample_initial_channel = upsample_initial_channel
+    self.upsample_kernel_sizes = upsample_kernel_sizes
+    self.segment_size = segment_size
+    self.n_speakers = n_speakers
+    self.gin_channels = gin_channels
+
+    self.use_sdp = use_sdp
+
+    self.enc_p = TextEncoder(n_vocab,
+        inter_channels,
+        hidden_channels,
+        filter_channels,
+        n_heads,
+        n_layers,
+        kernel_size,
+        p_dropout)
+    self.dec = Generator(inter_channels, resblock, resblock_kernel_sizes, resblock_dilation_sizes, upsample_rates, upsample_initial_channel, upsample_kernel_sizes, gin_channels=gin_channels)
+    self.enc_q = PosteriorEncoder(spec_channels, inter_channels, hidden_channels, 5, 1, 16, gin_channels=gin_channels)
+    self.flow = ResidualCouplingBlock(inter_channels, hidden_channels, 5, 1, 4, gin_channels=gin_channels)
+
+    if use_sdp:
+      self.dp = StochasticDurationPredictor(hidden_channels, 192, 3, 0.5, 4, gin_channels=gin_channels)
+    else:
+      self.dp = DurationPredictor(hidden_channels, 256, 3, 0.5, gin_channels=gin_channels)
+
+    if n_speakers > 1:
+      self.emb_g = nn.Embedding(n_speakers, gin_channels)
+
+  def forward(self, x, x_lengths, y, y_lengths, sid=None):
+
+    x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths)
+    if self.n_speakers > 0:
+      g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
+    else:
+      g = None
+
+    z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
+    z_p = self.flow(z, y_mask, g=g)
+
+    with torch.no_grad():
+      # negative cross-entropy
+      s_p_sq_r = torch.exp(-2 * logs_p) # [b, d, t]
+      neg_cent1 = torch.sum(-0.5 * math.log(2 * math.pi) - logs_p, [1], keepdim=True) # [b, 1, t_s]
+      neg_cent2 = torch.matmul(-0.5 * (z_p ** 2).transpose(1, 2), s_p_sq_r) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
+      neg_cent3 = torch.matmul(z_p.transpose(1, 2), (m_p * s_p_sq_r)) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
+      neg_cent4 = torch.sum(-0.5 * (m_p ** 2) * s_p_sq_r, [1], keepdim=True) # [b, 1, t_s]
+      neg_cent = neg_cent1 + neg_cent2 + neg_cent3 + neg_cent4
+
+      attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
+      attn = monotonic_align.maximum_path(neg_cent, attn_mask.squeeze(1)).unsqueeze(1).detach()
+
+    w = attn.sum(2)
+    if self.use_sdp:
+      l_length = self.dp(x, x_mask, w, g=g)
+      l_length = l_length / torch.sum(x_mask)
+    else:
+      logw_ = torch.log(w + 1e-6) * x_mask
+      logw = self.dp(x, x_mask, g=g)
+      l_length = torch.sum((logw - logw_)**2, [1,2]) / torch.sum(x_mask) # for averaging 
+
+    # expand prior
+    m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2)
+    logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1, 2)
+
+    z_slice, ids_slice = commons.rand_slice_segments(z, y_lengths, self.segment_size)
+    o = self.dec(z_slice, g=g)
+    return o, l_length, attn, ids_slice, x_mask, y_mask, (z, z_p, m_p, logs_p, m_q, logs_q)
+
+  def infer(self, x, x_lengths, sid=None, noise_scale=1, length_scale=1, noise_scale_w=1., max_len=None):
+    x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths)
+    if self.n_speakers > 0:
+      g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
+    else:
+      g = None
+
+    if self.use_sdp:
+      logw = self.dp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w)
+    else:
+      logw = self.dp(x, x_mask, g=g)
+    w = torch.exp(logw) * x_mask * length_scale
+    w_ceil = torch.ceil(w)
+    y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long()
+    y_mask = torch.unsqueeze(commons.sequence_mask(y_lengths, None), 1).to(x_mask.dtype)
+    attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
+    attn = commons.generate_path(w_ceil, attn_mask)
+
+    m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2) # [b, t', t], [b, t, d] -> [b, d, t']
+    logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1, 2) # [b, t', t], [b, t, d] -> [b, d, t']
+
+    z_p = m_p + torch.randn_like(m_p) * torch.exp(logs_p) * noise_scale
+    z = self.flow(z_p, y_mask, g=g, reverse=True)
+    o = self.dec((z * y_mask)[:,:,:max_len], g=g)
+    return o, attn, y_mask, (z, z_p, m_p, logs_p)
+
+  def voice_conversion(self, y, y_lengths, sid_src, sid_tgt):
+    assert self.n_speakers > 0, "n_speakers have to be larger than 0."
+    g_src = self.emb_g(sid_src).unsqueeze(-1)
+    g_tgt = self.emb_g(sid_tgt).unsqueeze(-1)
+    z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g_src)
+    z_p = self.flow(z, y_mask, g=g_src)
+    z_hat = self.flow(z_p, y_mask, g=g_tgt, reverse=True)
+    o_hat = self.dec(z_hat * y_mask, g=g_tgt)
+    return o_hat, y_mask, (z, z_p, z_hat)
+

tts-gateway/sidecar/models/modules.py

@@ -0,0 +1,390 @@
+import copy
+import math
+import numpy as np
+import scipy
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+from torch.nn import Conv1d, ConvTranspose1d, AvgPool1d, Conv2d
+from torch.nn.utils import weight_norm, remove_weight_norm
+
+import commons
+from commons import init_weights, get_padding
+from models.transforms import piecewise_rational_quadratic_transform
+
+
+LRELU_SLOPE = 0.1
+
+
+class LayerNorm(nn.Module):
+  def __init__(self, channels, eps=1e-5):
+    super().__init__()
+    self.channels = channels
+    self.eps = eps
+
+    self.gamma = nn.Parameter(torch.ones(channels))
+    self.beta = nn.Parameter(torch.zeros(channels))
+
+  def forward(self, x):
+    x = x.transpose(1, -1)
+    x = F.layer_norm(x, (self.channels,), self.gamma, self.beta, self.eps)
+    return x.transpose(1, -1)
+
+ 
+class ConvReluNorm(nn.Module):
+  def __init__(self, in_channels, hidden_channels, out_channels, kernel_size, n_layers, p_dropout):
+    super().__init__()
+    self.in_channels = in_channels
+    self.hidden_channels = hidden_channels
+    self.out_channels = out_channels
+    self.kernel_size = kernel_size
+    self.n_layers = n_layers
+    self.p_dropout = p_dropout
+    assert n_layers > 1, "Number of layers should be larger than 0."
+
+    self.conv_layers = nn.ModuleList()
+    self.norm_layers = nn.ModuleList()
+    self.conv_layers.append(nn.Conv1d(in_channels, hidden_channels, kernel_size, padding=kernel_size//2))
+    self.norm_layers.append(LayerNorm(hidden_channels))
+    self.relu_drop = nn.Sequential(
+        nn.ReLU(),
+        nn.Dropout(p_dropout))
+    for _ in range(n_layers-1):
+      self.conv_layers.append(nn.Conv1d(hidden_channels, hidden_channels, kernel_size, padding=kernel_size//2))
+      self.norm_layers.append(LayerNorm(hidden_channels))
+    self.proj = nn.Conv1d(hidden_channels, out_channels, 1)
+    self.proj.weight.data.zero_()
+    self.proj.bias.data.zero_()
+
+  def forward(self, x, x_mask):
+    x_org = x
+    for i in range(self.n_layers):
+      x = self.conv_layers[i](x * x_mask)
+      x = self.norm_layers[i](x)
+      x = self.relu_drop(x)
+    x = x_org + self.proj(x)
+    return x * x_mask
+
+
+class DDSConv(nn.Module):
+  """
+  Dialted and Depth-Separable Convolution
+  """
+  def __init__(self, channels, kernel_size, n_layers, p_dropout=0.):
+    super().__init__()
+    self.channels = channels
+    self.kernel_size = kernel_size
+    self.n_layers = n_layers
+    self.p_dropout = p_dropout
+
+    self.drop = nn.Dropout(p_dropout)
+    self.convs_sep = nn.ModuleList()
+    self.convs_1x1 = nn.ModuleList()
+    self.norms_1 = nn.ModuleList()
+    self.norms_2 = nn.ModuleList()
+    for i in range(n_layers):
+      dilation = kernel_size ** i
+      padding = (kernel_size * dilation - dilation) // 2
+      self.convs_sep.append(nn.Conv1d(channels, channels, kernel_size, 
+          groups=channels, dilation=dilation, padding=padding
+      ))
+      self.convs_1x1.append(nn.Conv1d(channels, channels, 1))
+      self.norms_1.append(LayerNorm(channels))
+      self.norms_2.append(LayerNorm(channels))
+
+  def forward(self, x, x_mask, g=None):
+    if g is not None:
+      x = x + g
+    for i in range(self.n_layers):
+      y = self.convs_sep[i](x * x_mask)
+      y = self.norms_1[i](y)
+      y = F.gelu(y)
+      y = self.convs_1x1[i](y)
+      y = self.norms_2[i](y)
+      y = F.gelu(y)
+      y = self.drop(y)
+      x = x + y
+    return x * x_mask
+
+
+class WN(torch.nn.Module):
+  def __init__(self, hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=0, p_dropout=0):
+    super(WN, self).__init__()
+    assert(kernel_size % 2 == 1)
+    self.hidden_channels =hidden_channels
+    self.kernel_size = kernel_size,
+    self.dilation_rate = dilation_rate
+    self.n_layers = n_layers
+    self.gin_channels = gin_channels
+    self.p_dropout = p_dropout
+
+    self.in_layers = torch.nn.ModuleList()
+    self.res_skip_layers = torch.nn.ModuleList()
+    self.drop = nn.Dropout(p_dropout)
+
+    if gin_channels != 0:
+      cond_layer = torch.nn.Conv1d(gin_channels, 2*hidden_channels*n_layers, 1)
+      self.cond_layer = torch.nn.utils.weight_norm(cond_layer, name='weight')
+
+    for i in range(n_layers):
+      dilation = dilation_rate ** i
+      padding = int((kernel_size * dilation - dilation) / 2)
+      in_layer = torch.nn.Conv1d(hidden_channels, 2*hidden_channels, kernel_size,
+                                 dilation=dilation, padding=padding)
+      in_layer = torch.nn.utils.weight_norm(in_layer, name='weight')
+      self.in_layers.append(in_layer)
+
+      # last one is not necessary
+      if i < n_layers - 1:
+        res_skip_channels = 2 * hidden_channels
+      else:
+        res_skip_channels = hidden_channels
+
+      res_skip_layer = torch.nn.Conv1d(hidden_channels, res_skip_channels, 1)
+      res_skip_layer = torch.nn.utils.weight_norm(res_skip_layer, name='weight')
+      self.res_skip_layers.append(res_skip_layer)
+
+  def forward(self, x, x_mask, g=None, **kwargs):
+    output = torch.zeros_like(x)
+    n_channels_tensor = torch.IntTensor([self.hidden_channels])
+
+    if g is not None:
+      g = self.cond_layer(g)
+
+    for i in range(self.n_layers):
+      x_in = self.in_layers[i](x)
+      if g is not None:
+        cond_offset = i * 2 * self.hidden_channels
+        g_l = g[:,cond_offset:cond_offset+2*self.hidden_channels,:]
+      else:
+        g_l = torch.zeros_like(x_in)
+
+      acts = commons.fused_add_tanh_sigmoid_multiply(
+          x_in,
+          g_l,
+          n_channels_tensor)
+      acts = self.drop(acts)
+
+      res_skip_acts = self.res_skip_layers[i](acts)
+      if i < self.n_layers - 1:
+        res_acts = res_skip_acts[:,:self.hidden_channels,:]
+        x = (x + res_acts) * x_mask
+        output = output + res_skip_acts[:,self.hidden_channels:,:]
+      else:
+        output = output + res_skip_acts
+    return output * x_mask
+
+  def remove_weight_norm(self):
+    if self.gin_channels != 0:
+      torch.nn.utils.remove_weight_norm(self.cond_layer)
+    for l in self.in_layers:
+      torch.nn.utils.remove_weight_norm(l)
+    for l in self.res_skip_layers:
+     torch.nn.utils.remove_weight_norm(l)
+
+
+class ResBlock1(torch.nn.Module):
+    def __init__(self, channels, kernel_size=3, dilation=(1, 3, 5)):
+        super(ResBlock1, self).__init__()
+        self.convs1 = nn.ModuleList([
+            weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0],
+                               padding=get_padding(kernel_size, dilation[0]))),
+            weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1],
+                               padding=get_padding(kernel_size, dilation[1]))),
+            weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[2],
+                               padding=get_padding(kernel_size, dilation[2])))
+        ])
+        self.convs1.apply(init_weights)
+
+        self.convs2 = nn.ModuleList([
+            weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1,
+                               padding=get_padding(kernel_size, 1))),
+            weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1,
+                               padding=get_padding(kernel_size, 1))),
+            weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1,
+                               padding=get_padding(kernel_size, 1)))
+        ])
+        self.convs2.apply(init_weights)
+
+    def forward(self, x, x_mask=None):
+        for c1, c2 in zip(self.convs1, self.convs2):
+            xt = F.leaky_relu(x, LRELU_SLOPE)
+            if x_mask is not None:
+                xt = xt * x_mask
+            xt = c1(xt)
+            xt = F.leaky_relu(xt, LRELU_SLOPE)
+            if x_mask is not None:
+                xt = xt * x_mask
+            xt = c2(xt)
+            x = xt + x
+        if x_mask is not None:
+            x = x * x_mask
+        return x
+
+    def remove_weight_norm(self):
+        for l in self.convs1:
+            remove_weight_norm(l)
+        for l in self.convs2:
+            remove_weight_norm(l)
+
+
+class ResBlock2(torch.nn.Module):
+    def __init__(self, channels, kernel_size=3, dilation=(1, 3)):
+        super(ResBlock2, self).__init__()
+        self.convs = nn.ModuleList([
+            weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0],
+                               padding=get_padding(kernel_size, dilation[0]))),
+            weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1],
+                               padding=get_padding(kernel_size, dilation[1])))
+        ])
+        self.convs.apply(init_weights)
+
+    def forward(self, x, x_mask=None):
+        for c in self.convs:
+            xt = F.leaky_relu(x, LRELU_SLOPE)
+            if x_mask is not None:
+                xt = xt * x_mask
+            xt = c(xt)
+            x = xt + x
+        if x_mask is not None:
+            x = x * x_mask
+        return x
+
+    def remove_weight_norm(self):
+        for l in self.convs:
+            remove_weight_norm(l)
+
+
+class Log(nn.Module):
+  def forward(self, x, x_mask, reverse=False, **kwargs):
+    if not reverse:
+      y = torch.log(torch.clamp_min(x, 1e-5)) * x_mask
+      logdet = torch.sum(-y, [1, 2])
+      return y, logdet
+    else:
+      x = torch.exp(x) * x_mask
+      return x
+    
+
+class Flip(nn.Module):
+  def forward(self, x, *args, reverse=False, **kwargs):
+    x = torch.flip(x, [1])
+    if not reverse:
+      logdet = torch.zeros(x.size(0)).to(dtype=x.dtype, device=x.device)
+      return x, logdet
+    else:
+      return x
+
+
+class ElementwiseAffine(nn.Module):
+  def __init__(self, channels):
+    super().__init__()
+    self.channels = channels
+    self.m = nn.Parameter(torch.zeros(channels,1))
+    self.logs = nn.Parameter(torch.zeros(channels,1))
+
+  def forward(self, x, x_mask, reverse=False, **kwargs):
+    if not reverse:
+      y = self.m + torch.exp(self.logs) * x
+      y = y * x_mask
+      logdet = torch.sum(self.logs * x_mask, [1,2])
+      return y, logdet
+    else:
+      x = (x - self.m) * torch.exp(-self.logs) * x_mask
+      return x
+
+
+class ResidualCouplingLayer(nn.Module):
+  def __init__(self,
+      channels,
+      hidden_channels,
+      kernel_size,
+      dilation_rate,
+      n_layers,
+      p_dropout=0,
+      gin_channels=0,
+      mean_only=False):
+    assert channels % 2 == 0, "channels should be divisible by 2"
+    super().__init__()
+    self.channels = channels
+    self.hidden_channels = hidden_channels
+    self.kernel_size = kernel_size
+    self.dilation_rate = dilation_rate
+    self.n_layers = n_layers
+    self.half_channels = channels // 2
+    self.mean_only = mean_only
+
+    self.pre = nn.Conv1d(self.half_channels, hidden_channels, 1)
+    self.enc = WN(hidden_channels, kernel_size, dilation_rate, n_layers, p_dropout=p_dropout, gin_channels=gin_channels)
+    self.post = nn.Conv1d(hidden_channels, self.half_channels * (2 - mean_only), 1)
+    self.post.weight.data.zero_()
+    self.post.bias.data.zero_()
+
+  def forward(self, x, x_mask, g=None, reverse=False):
+    x0, x1 = torch.split(x, [self.half_channels]*2, 1)
+    h = self.pre(x0) * x_mask
+    h = self.enc(h, x_mask, g=g)
+    stats = self.post(h) * x_mask
+    if not self.mean_only:
+      m, logs = torch.split(stats, [self.half_channels]*2, 1)
+    else:
+      m = stats
+      logs = torch.zeros_like(m)
+
+    if not reverse:
+      x1 = m + x1 * torch.exp(logs) * x_mask
+      x = torch.cat([x0, x1], 1)
+      logdet = torch.sum(logs, [1,2])
+      return x, logdet
+    else:
+      x1 = (x1 - m) * torch.exp(-logs) * x_mask
+      x = torch.cat([x0, x1], 1)
+      return x
+
+
+class ConvFlow(nn.Module):
+  def __init__(self, in_channels, filter_channels, kernel_size, n_layers, num_bins=10, tail_bound=5.0):
+    super().__init__()
+    self.in_channels = in_channels
+    self.filter_channels = filter_channels
+    self.kernel_size = kernel_size
+    self.n_layers = n_layers
+    self.num_bins = num_bins
+    self.tail_bound = tail_bound
+    self.half_channels = in_channels // 2
+
+    self.pre = nn.Conv1d(self.half_channels, filter_channels, 1)
+    self.convs = DDSConv(filter_channels, kernel_size, n_layers, p_dropout=0.)
+    self.proj = nn.Conv1d(filter_channels, self.half_channels * (num_bins * 3 - 1), 1)
+    self.proj.weight.data.zero_()
+    self.proj.bias.data.zero_()
+
+  def forward(self, x, x_mask, g=None, reverse=False):
+    x0, x1 = torch.split(x, [self.half_channels]*2, 1)
+    h = self.pre(x0)
+    h = self.convs(h, x_mask, g=g)
+    h = self.proj(h) * x_mask
+
+    b, c, t = x0.shape
+    h = h.reshape(b, c, -1, t).permute(0, 1, 3, 2) # [b, cx?, t] -> [b, c, t, ?]
+
+    unnormalized_widths = h[..., :self.num_bins] / math.sqrt(self.filter_channels)
+    unnormalized_heights = h[..., self.num_bins:2*self.num_bins] / math.sqrt(self.filter_channels)
+    unnormalized_derivatives = h[..., 2 * self.num_bins:]
+
+    x1, logabsdet = piecewise_rational_quadratic_transform(x1,
+        unnormalized_widths,
+        unnormalized_heights,
+        unnormalized_derivatives,
+        inverse=reverse,
+        tails='linear',
+        tail_bound=self.tail_bound
+    )
+
+    x = torch.cat([x0, x1], 1) * x_mask
+    logdet = torch.sum(logabsdet * x_mask, [1,2])
+    if not reverse:
+        return x, logdet
+    else:
+        return x

tts-gateway/sidecar/models/transforms.py

@@ -0,0 +1,209 @@
+import torch
+from torch.nn import functional as F
+
+import numpy as np
+
+
+DEFAULT_MIN_BIN_WIDTH = 1e-3
+DEFAULT_MIN_BIN_HEIGHT = 1e-3
+DEFAULT_MIN_DERIVATIVE = 1e-3
+
+
+def piecewise_rational_quadratic_transform(inputs, 
+                                           unnormalized_widths,
+                                           unnormalized_heights,
+                                           unnormalized_derivatives,
+                                           inverse=False,
+                                           tails=None, 
+                                           tail_bound=1.,
+                                           min_bin_width=DEFAULT_MIN_BIN_WIDTH,
+                                           min_bin_height=DEFAULT_MIN_BIN_HEIGHT,
+                                           min_derivative=DEFAULT_MIN_DERIVATIVE):
+
+    if tails is None:
+        spline_fn = rational_quadratic_spline
+        spline_kwargs = {}
+    else:
+        spline_fn = unconstrained_rational_quadratic_spline
+        spline_kwargs = {
+            'tails': tails,
+            'tail_bound': tail_bound
+        }
+
+    outputs, logabsdet = spline_fn(
+            inputs=inputs,
+            unnormalized_widths=unnormalized_widths,
+            unnormalized_heights=unnormalized_heights,
+            unnormalized_derivatives=unnormalized_derivatives,
+            inverse=inverse,
+            min_bin_width=min_bin_width,
+            min_bin_height=min_bin_height,
+            min_derivative=min_derivative,
+            **spline_kwargs
+    )
+    return outputs, logabsdet
+
+
+def searchsorted(bin_locations, inputs, eps=1e-6):
+    bin_locations[..., -1] += eps
+    return torch.sum(
+        inputs[..., None] >= bin_locations,
+        dim=-1
+    ) - 1
+
+
+def unconstrained_rational_quadratic_spline(inputs,
+                                            unnormalized_widths,
+                                            unnormalized_heights,
+                                            unnormalized_derivatives,
+                                            inverse=False,
+                                            tails='linear',
+                                            tail_bound=1.,
+                                            min_bin_width=DEFAULT_MIN_BIN_WIDTH,
+                                            min_bin_height=DEFAULT_MIN_BIN_HEIGHT,
+                                            min_derivative=DEFAULT_MIN_DERIVATIVE):
+    inside_interval_mask = (inputs >= -tail_bound) & (inputs <= tail_bound)
+
+    if tails == 'linear':
+        unnormalized_derivatives = F.pad(unnormalized_derivatives, pad=(1, 1))
+        constant = np.log(np.exp(1 - min_derivative) - 1)
+        unnormalized_derivatives[..., 0] = constant
+        unnormalized_derivatives[..., -1] = constant
+    else:
+        raise RuntimeError('{} tails are not implemented.'.format(tails))
+
+    # ONNX-export-friendly rewrite (Phase 0, TTS_GATEWAY_PLAN.md): the original
+    # boolean-mask indexed assignment (`outputs[mask] = ...`) subselects a
+    # data-dependent number of elements, which torch.export's symbolic tracer
+    # can't handle (unbacked SymInts leaking into later Conv1d shape guards).
+    # Rewritten as a fixed-shape computation + torch.where select instead.
+    # Clamping is a no-op wherever inside_interval_mask is True (those inputs
+    # are already within bounds), so the spline output there is identical to
+    # the original masked version; where the mask is False, the spline's
+    # output for that clamped position is simply discarded by torch.where in
+    # favor of the identity passthrough - clamping there only exists to keep
+    # rational_quadratic_spline's internal `gather` calls in-bounds.
+    clamped_inputs = torch.clamp(inputs, -tail_bound, tail_bound)
+    spline_outputs, spline_logabsdet = rational_quadratic_spline(
+        inputs=clamped_inputs,
+        unnormalized_widths=unnormalized_widths,
+        unnormalized_heights=unnormalized_heights,
+        unnormalized_derivatives=unnormalized_derivatives,
+        inverse=inverse,
+        left=-tail_bound, right=tail_bound, bottom=-tail_bound, top=tail_bound,
+        min_bin_width=min_bin_width,
+        min_bin_height=min_bin_height,
+        min_derivative=min_derivative
+    )
+
+    outputs = torch.where(inside_interval_mask, spline_outputs, inputs)
+    logabsdet = torch.where(inside_interval_mask, spline_logabsdet, torch.zeros_like(inputs))
+
+    return outputs, logabsdet
+
+def rational_quadratic_spline(inputs,
+                              unnormalized_widths,
+                              unnormalized_heights,
+                              unnormalized_derivatives,
+                              inverse=False,
+                              left=0., right=1., bottom=0., top=1.,
+                              min_bin_width=DEFAULT_MIN_BIN_WIDTH,
+                              min_bin_height=DEFAULT_MIN_BIN_HEIGHT,
+                              min_derivative=DEFAULT_MIN_DERIVATIVE):
+    # Removed for ONNX export (Phase 0 of TTS_GATEWAY_PLAN.md): this is a pure
+    # bounds assertion on tensor values, not part of the computation - the
+    # caller (unconstrained_rational_quadratic_spline) already only invokes
+    # this on inputs[inside_interval_mask], which are guaranteed in [left,
+    # right] by construction. Left in as a data-dependent Python `if`, it
+    # breaks torch.export's symbolic tracer (GuardOnDataDependentSymNode).
+    # Behavior is unchanged either way; only affects traceability.
+
+    num_bins = unnormalized_widths.shape[-1]
+
+    if min_bin_width * num_bins > 1.0:
+        raise ValueError('Minimal bin width too large for the number of bins')
+    if min_bin_height * num_bins > 1.0:
+        raise ValueError('Minimal bin height too large for the number of bins')
+
+    widths = F.softmax(unnormalized_widths, dim=-1)
+    widths = min_bin_width + (1 - min_bin_width * num_bins) * widths
+    cumwidths = torch.cumsum(widths, dim=-1)
+    cumwidths = F.pad(cumwidths, pad=(1, 0), mode='constant', value=0.0)
+    cumwidths = (right - left) * cumwidths + left
+    cumwidths[..., 0] = left
+    cumwidths[..., -1] = right
+    widths = cumwidths[..., 1:] - cumwidths[..., :-1]
+
+    derivatives = min_derivative + F.softplus(unnormalized_derivatives)
+
+    heights = F.softmax(unnormalized_heights, dim=-1)
+    heights = min_bin_height + (1 - min_bin_height * num_bins) * heights
+    cumheights = torch.cumsum(heights, dim=-1)
+    cumheights = F.pad(cumheights, pad=(1, 0), mode='constant', value=0.0)
+    cumheights = (top - bottom) * cumheights + bottom
+    cumheights[..., 0] = bottom
+    cumheights[..., -1] = top
+    heights = cumheights[..., 1:] - cumheights[..., :-1]
+
+    if inverse:
+        bin_idx = searchsorted(cumheights, inputs)[..., None]
+    else:
+        bin_idx = searchsorted(cumwidths, inputs)[..., None]
+
+    input_cumwidths = cumwidths.gather(-1, bin_idx)[..., 0]
+    input_bin_widths = widths.gather(-1, bin_idx)[..., 0]
+
+    input_cumheights = cumheights.gather(-1, bin_idx)[..., 0]
+    delta = heights / widths
+    input_delta = delta.gather(-1, bin_idx)[..., 0]
+
+    input_derivatives = derivatives.gather(-1, bin_idx)[..., 0]
+    input_derivatives_plus_one = derivatives[..., 1:].gather(-1, bin_idx)[..., 0]
+
+    input_heights = heights.gather(-1, bin_idx)[..., 0]
+
+    if inverse:
+        a = (((inputs - input_cumheights) * (input_derivatives
+                                             + input_derivatives_plus_one
+                                             - 2 * input_delta)
+              + input_heights * (input_delta - input_derivatives)))
+        b = (input_heights * input_derivatives
+             - (inputs - input_cumheights) * (input_derivatives
+                                              + input_derivatives_plus_one
+                                              - 2 * input_delta))
+        c = - input_delta * (inputs - input_cumheights)
+
+        discriminant = b.pow(2) - 4 * a * c
+        # Removed for ONNX export (Phase 0 of TTS_GATEWAY_PLAN.md) - same
+        # reasoning as the bounds check above: a numerical-invariant sanity
+        # check on a tensor value, not part of the computation itself, and it
+        # breaks torch.export's symbolic tracer the same way.
+
+        root = (2 * c) / (-b - torch.sqrt(discriminant))
+        outputs = root * input_bin_widths + input_cumwidths
+
+        theta_one_minus_theta = root * (1 - root)
+        denominator = input_delta + ((input_derivatives + input_derivatives_plus_one - 2 * input_delta)
+                                     * theta_one_minus_theta)
+        derivative_numerator = input_delta.pow(2) * (input_derivatives_plus_one * root.pow(2)
+                                                     + 2 * input_delta * theta_one_minus_theta
+                                                     + input_derivatives * (1 - root).pow(2))
+        logabsdet = torch.log(derivative_numerator) - 2 * torch.log(denominator)
+
+        return outputs, -logabsdet
+    else:
+        theta = (inputs - input_cumwidths) / input_bin_widths
+        theta_one_minus_theta = theta * (1 - theta)
+
+        numerator = input_heights * (input_delta * theta.pow(2)
+                                     + input_derivatives * theta_one_minus_theta)
+        denominator = input_delta + ((input_derivatives + input_derivatives_plus_one - 2 * input_delta)
+                                     * theta_one_minus_theta)
+        outputs = input_cumheights + numerator / denominator
+
+        derivative_numerator = input_delta.pow(2) * (input_derivatives_plus_one * theta.pow(2)
+                                                     + 2 * input_delta * theta_one_minus_theta
+                                                     + input_derivatives * (1 - theta).pow(2))
+        logabsdet = torch.log(derivative_numerator) - 2 * torch.log(denominator)
+
+        return outputs, logabsdet

tts-gateway/sidecar/models/utils.py

@@ -0,0 +1,258 @@
+import os
+import glob
+import sys
+import argparse
+import logging
+import json
+import subprocess
+import numpy as np
+from scipy.io.wavfile import read
+import torch
+
+MATPLOTLIB_FLAG = False
+
+logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
+logger = logging
+
+
+def load_checkpoint(checkpoint_path, model, optimizer=None):
+  assert os.path.isfile(checkpoint_path)
+  checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
+  iteration = checkpoint_dict['iteration']
+  learning_rate = checkpoint_dict['learning_rate']
+  if optimizer is not None:
+    optimizer.load_state_dict(checkpoint_dict['optimizer'])
+  saved_state_dict = checkpoint_dict['model']
+  if hasattr(model, 'module'):
+    state_dict = model.module.state_dict()
+  else:
+    state_dict = model.state_dict()
+  new_state_dict= {}
+  for k, v in state_dict.items():
+    try:
+      new_state_dict[k] = saved_state_dict[k]
+    except:
+      logger.info("%s is not in the checkpoint" % k)
+      new_state_dict[k] = v
+  if hasattr(model, 'module'):
+    model.module.load_state_dict(new_state_dict)
+  else:
+    model.load_state_dict(new_state_dict)
+  logger.info("Loaded checkpoint '{}' (iteration {})" .format(
+    checkpoint_path, iteration))
+  return model, optimizer, learning_rate, iteration
+
+
+def save_checkpoint(model, optimizer, learning_rate, iteration, checkpoint_path):
+  logger.info("Saving model and optimizer state at iteration {} to {}".format(
+    iteration, checkpoint_path))
+  if hasattr(model, 'module'):
+    state_dict = model.module.state_dict()
+  else:
+    state_dict = model.state_dict()
+  torch.save({'model': state_dict,
+              'iteration': iteration,
+              'optimizer': optimizer.state_dict(),
+              'learning_rate': learning_rate}, checkpoint_path)
+
+
+def summarize(writer, global_step, scalars={}, histograms={}, images={}, audios={}, audio_sampling_rate=22050):
+  for k, v in scalars.items():
+    writer.add_scalar(k, v, global_step)
+  for k, v in histograms.items():
+    writer.add_histogram(k, v, global_step)
+  for k, v in images.items():
+    writer.add_image(k, v, global_step, dataformats='HWC')
+  for k, v in audios.items():
+    writer.add_audio(k, v, global_step, audio_sampling_rate)
+
+
+def latest_checkpoint_path(dir_path, regex="G_*.pth"):
+  f_list = glob.glob(os.path.join(dir_path, regex))
+  f_list.sort(key=lambda f: int("".join(filter(str.isdigit, f))))
+  x = f_list[-1]
+  print(x)
+  return x
+
+
+def plot_spectrogram_to_numpy(spectrogram):
+  global MATPLOTLIB_FLAG
+  if not MATPLOTLIB_FLAG:
+    import matplotlib
+    matplotlib.use("Agg")
+    MATPLOTLIB_FLAG = True
+    mpl_logger = logging.getLogger('matplotlib')
+    mpl_logger.setLevel(logging.WARNING)
+  import matplotlib.pylab as plt
+  import numpy as np
+  
+  fig, ax = plt.subplots(figsize=(10,2))
+  im = ax.imshow(spectrogram, aspect="auto", origin="lower",
+                  interpolation='none')
+  plt.colorbar(im, ax=ax)
+  plt.xlabel("Frames")
+  plt.ylabel("Channels")
+  plt.tight_layout()
+
+  fig.canvas.draw()
+  data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
+  data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,))
+  plt.close()
+  return data
+
+
+def plot_alignment_to_numpy(alignment, info=None):
+  global MATPLOTLIB_FLAG
+  if not MATPLOTLIB_FLAG:
+    import matplotlib
+    matplotlib.use("Agg")
+    MATPLOTLIB_FLAG = True
+    mpl_logger = logging.getLogger('matplotlib')
+    mpl_logger.setLevel(logging.WARNING)
+  import matplotlib.pylab as plt
+  import numpy as np
+
+  fig, ax = plt.subplots(figsize=(6, 4))
+  im = ax.imshow(alignment.transpose(), aspect='auto', origin='lower',
+                  interpolation='none')
+  fig.colorbar(im, ax=ax)
+  xlabel = 'Decoder timestep'
+  if info is not None:
+      xlabel += '\n\n' + info
+  plt.xlabel(xlabel)
+  plt.ylabel('Encoder timestep')
+  plt.tight_layout()
+
+  fig.canvas.draw()
+  data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
+  data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,))
+  plt.close()
+  return data
+
+
+def load_wav_to_torch(full_path):
+  sampling_rate, data = read(full_path)
+  return torch.FloatTensor(data.astype(np.float32)), sampling_rate
+
+
+def load_filepaths_and_text(filename, split="|"):
+  with open(filename, encoding='utf-8') as f:
+    filepaths_and_text = [line.strip().split(split) for line in f]
+  return filepaths_and_text
+
+
+def get_hparams(init=True):
+  parser = argparse.ArgumentParser()
+  parser.add_argument('-c', '--config', type=str, default="./configs/base.json",
+                      help='JSON file for configuration')
+  parser.add_argument('-m', '--model', type=str, required=True,
+                      help='Model name')
+  
+  args = parser.parse_args()
+  model_dir = os.path.join("./logs", args.model)
+
+  if not os.path.exists(model_dir):
+    os.makedirs(model_dir)
+
+  config_path = args.config
+  config_save_path = os.path.join(model_dir, "config.json")
+  if init:
+    with open(config_path, "r") as f:
+      data = f.read()
+    with open(config_save_path, "w") as f:
+      f.write(data)
+  else:
+    with open(config_save_path, "r") as f:
+      data = f.read()
+  config = json.loads(data)
+  
+  hparams = HParams(**config)
+  hparams.model_dir = model_dir
+  return hparams
+
+
+def get_hparams_from_dir(model_dir):
+  config_save_path = os.path.join(model_dir, "config.json")
+  with open(config_save_path, "r") as f:
+    data = f.read()
+  config = json.loads(data)
+
+  hparams =HParams(**config)
+  hparams.model_dir = model_dir
+  return hparams
+
+
+def get_hparams_from_file(config_path):
+  with open(config_path, "r") as f:
+    data = f.read()
+  config = json.loads(data)
+
+  hparams =HParams(**config)
+  return hparams
+
+
+def check_git_hash(model_dir):
+  source_dir = os.path.dirname(os.path.realpath(__file__))
+  if not os.path.exists(os.path.join(source_dir, ".git")):
+    logger.warn("{} is not a git repository, therefore hash value comparison will be ignored.".format(
+      source_dir
+    ))
+    return
+
+  cur_hash = subprocess.getoutput("git rev-parse HEAD")
+
+  path = os.path.join(model_dir, "githash")
+  if os.path.exists(path):
+    saved_hash = open(path).read()
+    if saved_hash != cur_hash:
+      logger.warn("git hash values are different. {}(saved) != {}(current)".format(
+        saved_hash[:8], cur_hash[:8]))
+  else:
+    open(path, "w").write(cur_hash)
+
+
+def get_logger(model_dir, filename="train.log"):
+  global logger
+  logger = logging.getLogger(os.path.basename(model_dir))
+  logger.setLevel(logging.DEBUG)
+  
+  formatter = logging.Formatter("%(asctime)s\t%(name)s\t%(levelname)s\t%(message)s")
+  if not os.path.exists(model_dir):
+    os.makedirs(model_dir)
+  h = logging.FileHandler(os.path.join(model_dir, filename))
+  h.setLevel(logging.DEBUG)
+  h.setFormatter(formatter)
+  logger.addHandler(h)
+  return logger
+
+
+class HParams():
+  def __init__(self, **kwargs):
+    for k, v in kwargs.items():
+      if type(v) == dict:
+        v = HParams(**v)
+      self[k] = v
+    
+  def keys(self):
+    return self.__dict__.keys()
+
+  def items(self):
+    return self.__dict__.items()
+
+  def values(self):
+    return self.__dict__.values()
+
+  def __len__(self):
+    return len(self.__dict__)
+
+  def __getitem__(self, key):
+    return getattr(self, key)
+
+  def __setitem__(self, key, value):
+    return setattr(self, key, value)
+
+  def __contains__(self, key):
+    return key in self.__dict__
+
+  def __repr__(self):
+    return self.__dict__.__repr__()

tts-gateway/sidecar/monotonic_align/__init__.py

@@ -0,0 +1,19 @@
+import numpy as np
+import torch
+from .monotonic_align.core import maximum_path_c
+
+
+def maximum_path(neg_cent, mask):
+  """ Cython optimized version.
+  neg_cent: [b, t_t, t_s]
+  mask: [b, t_t, t_s]
+  """
+  device = neg_cent.device
+  dtype = neg_cent.dtype
+  neg_cent = neg_cent.data.cpu().numpy().astype(np.float32)
+  path = np.zeros(neg_cent.shape, dtype=np.int32)
+
+  t_t_max = mask.sum(1)[:, 0].data.cpu().numpy().astype(np.int32)
+  t_s_max = mask.sum(2)[:, 0].data.cpu().numpy().astype(np.int32)
+  maximum_path_c(path, neg_cent, t_t_max, t_s_max)
+  return torch.from_numpy(path).to(device=device, dtype=dtype)

tts-gateway/sidecar/monotonic_align/core.pyx

@@ -0,0 +1,42 @@
+cimport cython
+from cython.parallel import prange
+
+
+@cython.boundscheck(False)
+@cython.wraparound(False)
+cdef void maximum_path_each(int[:,::1] path, float[:,::1] value, int t_y, int t_x, float max_neg_val=-1e9) nogil:
+  cdef int x
+  cdef int y
+  cdef float v_prev
+  cdef float v_cur
+  cdef float tmp
+  cdef int index = t_x - 1
+
+  for y in range(t_y):
+    for x in range(max(0, t_x + y - t_y), min(t_x, y + 1)):
+      if x == y:
+        v_cur = max_neg_val
+      else:
+        v_cur = value[y-1, x]
+      if x == 0:
+        if y == 0:
+          v_prev = 0.
+        else:
+          v_prev = max_neg_val
+      else:
+        v_prev = value[y-1, x-1]
+      value[y, x] += max(v_prev, v_cur)
+
+  for y in range(t_y - 1, -1, -1):
+    path[y, index] = 1
+    if index != 0 and (index == y or value[y-1, index] < value[y-1, index-1]):
+      index = index - 1
+
+
+@cython.boundscheck(False)
+@cython.wraparound(False)
+cpdef void maximum_path_c(int[:,:,::1] paths, float[:,:,::1] values, int[::1] t_ys, int[::1] t_xs) nogil:
+  cdef int b = paths.shape[0]
+  cdef int i
+  for i in prange(b, nogil=True):
+    maximum_path_each(paths[i], values[i], t_ys[i], t_xs[i])

tts-gateway/sidecar/monotonic_align/setup.py

@@ -0,0 +1,9 @@
+from distutils.core import setup
+from Cython.Build import cythonize
+import numpy
+
+setup(
+  name = 'monotonic_align',
+  ext_modules = cythonize("core.pyx"),
+  include_dirs=[numpy.get_include()]
+)

tts-gateway/sidecar/requirements.txt

@@ -0,0 +1,8 @@
+# Additive on top of the pytorch/pytorch base image's own CUDA-matched torch
+# build (see Dockerfile) - scipy is needed only because models/utils.py has a
+# top-level `from scipy.io.wavfile import read` import; Cython is needed only
+# to build the monotonic_align extension (imported at module load by
+# models/models.py even though this sidecar's infer()-only path never calls
+# it - see the comment in Dockerfile).
+scipy==1.16.3
+Cython>=3.0.0

tts-gateway/sidecar/server.py

@@ -0,0 +1,128 @@
+"""Minimal inference sidecar for tts-gateway (Phase 3, TTS_GATEWAY_PLAN.md).
+
+Runs only SynthesizerTrn.infer() - no Flask, no training code, no Japanese
+text handling (that's done entirely in Go by
+internal/adapters/secondary/jtalk; this process receives already-normalized,
+already-interspersed symbol IDs and just runs the model). Vendored model
+code (models/, commons.py) is copied from tmp/reference/uma-tts-api,
+including the transforms.py edits made during Phase 0's ONNX spike - those
+are behavior-preserving no-ops for this eager-mode infer() path too (see
+tmp/reference/uma-tts-api/spike/FINDINGS.md), so there was no reason to
+maintain two divergent copies.
+"""
+
+import json
+import logging
+import os
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+import torch
+
+from models import utils
+from models.models import SynthesizerTrn
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
+logger = logging.getLogger(__name__)
+
+CONFIG_PATH = os.environ.get("CONFIG_PATH", "/models/uma.json")
+CHECKPOINT_PATH = os.environ.get("CHECKPOINT_PATH", "/models/G_790000.pth")
+PORT = int(os.environ.get("PORT", "50054"))
+
+# len(text.symbols.symbols) from the reference repo: 1 (pad) + 8 (special) +
+# 22 (punctuation) + 62 (letters+digits) + 84 (dummy) = 177. Hardcoded rather
+# than importing text/symbols.py, since this sidecar deliberately carries no
+# text-handling code - Go's internal/core/domain/symbols.go is the source of
+# truth for the vocabulary now, and the two are verified to agree in
+# tts-gateway's tests.
+N_VOCAB = 177
+
+model_lock = threading.Lock()
+
+
+def load_model():
+    hps = utils.get_hparams_from_file(CONFIG_PATH)
+    net_g = SynthesizerTrn(
+        n_vocab=N_VOCAB,
+        spec_channels=hps.data.filter_length // 2 + 1,
+        segment_size=hps.train.segment_size // hps.data.hop_length,
+        n_speakers=hps.data.n_speakers,
+        **hps.model,
+    )
+    net_g.eval()
+    if torch.cuda.is_available():
+        net_g.cuda()
+    else:
+        logger.warning("CUDA not available, running on CPU")
+    utils.load_checkpoint(CHECKPOINT_PATH, net_g, None)
+    logger.info("model loaded from %s", CHECKPOINT_PATH)
+    return net_g, hps
+
+
+net_g, hps = load_model()
+
+
+def synthesize(symbol_ids, speaker_id, noise_scale, noise_scale_w, length_scale):
+    x = torch.LongTensor(symbol_ids).unsqueeze(0)
+    x_lengths = torch.LongTensor([len(symbol_ids)])
+    sid = torch.LongTensor([speaker_id])
+    if torch.cuda.is_available():
+        x, x_lengths, sid = x.cuda(), x_lengths.cuda(), sid.cuda()
+
+    with torch.no_grad(), model_lock:
+        audio, *_ = net_g.infer(
+            x, x_lengths, sid=sid,
+            noise_scale=noise_scale, noise_scale_w=noise_scale_w, length_scale=length_scale,
+        )
+    return audio[0, 0].data.cpu().float().numpy()
+
+
+class Handler(BaseHTTPRequestHandler):
+    def log_message(self, fmt, *args):
+        logger.info("%s - %s", self.address_string(), fmt % args)
+
+    def do_GET(self):
+        if self.path == "/health":
+            self._respond(200, b'{"status":"ok"}', "application/json")
+        else:
+            self._respond(404, b"not found", "text/plain")
+
+    def do_POST(self):
+        if self.path != "/synthesize":
+            self._respond(404, b"not found", "text/plain")
+            return
+        try:
+            length = int(self.headers.get("Content-Length", "0"))
+            body = json.loads(self.rfile.read(length))
+            pcm = synthesize(
+                body["symbol_ids"],
+                body["speaker_id"],
+                body.get("noise_scale", 0.37),
+                body.get("noise_scale_w", 0.46),
+                body.get("length_scale", 1.3),
+            )
+        except Exception as e:  # noqa: BLE001 - report any failure to the client
+            logger.exception("synthesize failed")
+            self._respond(400, str(e).encode("utf-8"), "text/plain")
+            return
+
+        payload = pcm.astype("<f4").tobytes()
+        self.send_response(200)
+        self.send_header("Content-Type", "application/octet-stream")
+        self.send_header("X-Sample-Rate", str(hps.data.sampling_rate))
+        self.send_header("Content-Length", str(len(payload)))
+        self.end_headers()
+        self.wfile.write(payload)
+
+    def _respond(self, code, body, content_type):
+        self.send_response(code)
+        self.send_header("Content-Type", content_type)
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+
+if __name__ == "__main__":
+    server = ThreadingHTTPServer(("0.0.0.0", PORT), Handler)
+    logger.info("inference sidecar listening on :%d", PORT)
+    server.serve_forever()