From 0d58e4674027dcc0939da5599cdaccb426772703 Mon Sep 17 00:00:00 2001 From: Nik Afiq Date: Sat, 25 Jul 2026 02:09:02 +0900 Subject: [PATCH] feat: add /speak command for TTS integration - Implemented the /speak command in Discord bot to synthesize speech using the TTS gateway. - Added voice handling logic to join voice channels and play synthesized audio. - Created tests for the new command and voice functionalities. - Introduced TTSGateway interface for TTS service communication. - Updated configuration to include TTS gateway address. - Documented the TTS gateway integration and model artifact distribution process. --- CLAUDE.md | 17 ++- TTS_GATEWAY_PLAN.md | 27 ++-- discord-bot/.env.example | 1 + discord-bot/Dockerfile | 10 +- discord-bot/cmd/bot/main.go | 20 ++- discord-bot/go.mod | 2 + discord-bot/go.sum | 4 + .../adapters/primary/discord/handler.go | 107 +++++++++++++- .../adapters/primary/discord/handler_test.go | 108 ++++++++++++++ .../adapters/primary/discord/register.go | 19 +++ .../adapters/primary/discord/voice.go | 131 +++++++++++++++++ .../adapters/primary/discord/voice_test.go | 73 ++++++++++ .../adapters/secondary/ttsgateway/client.go | 124 ++++++++++++++++ discord-bot/internal/app/command.go | 32 ++++- discord-bot/internal/app/command_test.go | 133 +++++++++++++++++- discord-bot/internal/config/config.go | 34 ++--- discord-bot/internal/core/ports/driven/tts.go | 9 ++ handoff.md | 90 ++++++++++++ tts-gateway/README.md | 39 ++++- tts-gateway/model/Dockerfile | 7 + 20 files changed, 937 insertions(+), 50 deletions(-) create mode 100644 discord-bot/internal/adapters/primary/discord/handler_test.go create mode 100644 discord-bot/internal/adapters/primary/discord/voice.go create mode 100644 discord-bot/internal/adapters/primary/discord/voice_test.go create mode 100644 discord-bot/internal/adapters/secondary/ttsgateway/client.go create mode 100644 discord-bot/internal/core/ports/driven/tts.go create mode 100644 handoff.md create mode 100644 tts-gateway/model/Dockerfile diff --git a/CLAUDE.md b/CLAUDE.md index 60cb07e..f41e9c1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,12 +17,14 @@ ai-gateway ------> Ollama | v ha-gateway + +discord-bot -----> tts-gateway -----> tts-sidecar (GPU inference) ``` - **ha-gateway** (port `50051`) — gRPC boundary for Home Assistant. Talks to HA's REST API; implements entity state, light control/discovery, switch control/discovery, and climate (HVAC) control/discovery; also relays SwitchBot Cloud remote commands (`RemoteApp`) when `SWITCHBOT_TOKEN`/`SWITCHBOT_SECRET` are set. Event streaming is stubbed. - **ai-gateway** (port `50052`) — gRPC service that turns free-form text into home actions. Calls Ollama for intent extraction, resolves intents against a cached light list from `ha-gateway`, and calls `ha-gateway` to execute approved actions. -- **discord-bot** — registers `/light`, `/switch`, `/ac`, `/ai` slash commands and calls `ha-gateway`/`ai-gateway` via gRPC clients. -- **tts-gateway** (port `50053`) — gRPC text-to-speech service (VITS voice model, 92 Umamusume voices), paired with a Python/libtorch inference sidecar (`tts-gateway/sidecar/`, deployed as a second container in the same pod) that needs an Nvidia GPU — see `TTS_GATEWAY_PLAN.md` for why. Not yet wired into the diagram above; nothing calls it yet (a discord-bot `/speak` command is the planned follow-up). See `tts-gateway/README.md` for its API/config/local-run details. +- **discord-bot** — registers `/light`, `/switch`, `/ac`, `/ai`, `/speak` slash commands and calls `ha-gateway`/`ai-gateway`/`tts-gateway` via gRPC clients. `/speak` synthesizes speech via `tts-gateway` (which returns AAC) and transcodes it to Opus locally (via `ffmpeg`/`github.com/jonas747/dca`) to stream into the invoking user's current voice channel — see `internal/adapters/primary/discord/voice.go`. +- **tts-gateway** (port `50053`) — gRPC text-to-speech service (VITS voice model, 92 Umamusume voices), paired with a Python/libtorch inference sidecar (`tts-gateway/sidecar/`, deployed as a second container in the same pod) that needs an Nvidia GPU — see `TTS_GATEWAY_PLAN.md` for why. Called by `discord-bot`'s `/speak` command. See `tts-gateway/README.md` for its API/config/local-run details. Each service is a separate Go module (own `go.mod`) joined by `go.work` at the root, plus a `gen` module for shared generated code. Module paths are `gitea.nik4nao.com/nik/home-services/{ha-gateway,ai-gateway,discord-bot,tts-gateway,gen}`. @@ -172,10 +174,13 @@ rather than trusting this description to stay accurate): `nvidia.com/gpu: 1` request/limit on the sidecar container. `strategy: Recreate` instead of the default `RollingUpdate` — nik-gpu only has one allocatable GPU, so a rolling update would deadlock waiting for a GPU still held by the pod it's replacing. -- Model artifact distribution (checkpoint + hparams) is a `hostPath` volume at - `/data/tts-gateway` on the nik-gpu node, not baked into the image or automated — someone has to - manually place `G_790000.pth`/`uma.json` there. This is a known-open gap (see `handoff.md`), - not a deliberate final design. +- Model artifact distribution (checkpoint + hparams) is an `emptyDir` volume populated at pod + start by an `initContainers` entry (`model-init`) that copies from a small, versioned image + (`gitea.nik4nao.com/nik/tts-model:`, built from `tts-gateway/model/Dockerfile`) — not a + raw `hostPath` into the node's disk anymore. That model image still can't be built by CI (the + checkpoint isn't committed to git, ~455MB, and only ever existed on nik-gpu's local disk) — it's + built and pushed manually, on nik-gpu, whenever the checkpoint/config change. See + `tts-gateway/README.md` for the exact commands. - mTLS (`TLS_DIR`) is currently commented out on `tts-gateway`, for plaintext `grpcurl` testing from outside the cluster during initial rollout — a live TODO, not a permanent decision, unlike every other service here which assumes mTLS-or-trusted-network as its only access boundary (see diff --git a/TTS_GATEWAY_PLAN.md b/TTS_GATEWAY_PLAN.md index b52d9d2..e4d0420 100644 --- a/TTS_GATEWAY_PLAN.md +++ b/TTS_GATEWAY_PLAN.md @@ -170,16 +170,18 @@ repo's hand-written-mock testing convention rather than golden-byte comparison. WAV writer, `ffmpeg` transcode adapter, full `Synthesize` RPC wiring, speaker-not-found → `INVALID_ARGUMENT` (mirroring `app.py`'s current 400 behavior), `ListSpeakers` RPC. -**Phase 5 — Containerize & deploy to nik-gpu** -See [GPU/deployment notes](#gpu-and-deployment-notes) below — this phase needs a Dockerfile that -diverges from the other three services' pattern, plus a decision on how the model artifact -reaches the container. +**Phase 5 — Containerize & deploy to nik-gpu** — **done.** Deployed and confirmed working in +production; see `handoff.md` for status. Model artifact distribution is resolved as a small +versioned image (`tts-gateway/model/Dockerfile`) copied into a shared volume by a k8s +`initContainer`, built manually on nik-gpu (the checkpoint never existed anywhere CI can reach) — +see `tts-gateway/README.md`'s "Model artifact distribution (production)" section. -**Phase 6 — Client integration (optional, separate follow-up)** -A `/speak` (or similar) Discord command in `discord-bot` calling this gateway, following the -same secondary-adapter + gRPC-client pattern already used for its `ha-gateway`/`ai-gateway` -clients. Not assumed in scope here — a deliberate follow-on decision once the service itself -works. +**Phase 6 — Client integration** — **done.** `discord-bot` registers a `/speak` command calling +this gateway via the same secondary-adapter + gRPC-client pattern used for its +`ha-gateway`/`ai-gateway` clients (`discord-bot/internal/adapters/secondary/ttsgateway/`), then +transcodes the AAC response to Opus locally (`ffmpeg` + `github.com/jonas747/dca`) and streams it +into the invoking user's current voice channel +(`discord-bot/internal/adapters/primary/discord/voice.go`). ## GPU and deployment notes @@ -226,9 +228,10 @@ nik-gpu's host state untouched and the setup reproducible from the Dockerfile al ## Open questions (product/design, not technical blockers) -- **Output codec:** keep AAC (parity with today) or switch to Opus? If the eventual consumer is - `discord-bot`, Opus is Discord's native voice codec — worth deciding once Phase 6 is actually - on the table, not before. +- **Output codec:** ~~keep AAC (parity with today) or switch to Opus?~~ **Resolved: kept AAC.** + `tts-gateway` itself is unchanged; `discord-bot` transcodes AAC → Opus locally for voice + playback (see Phase 6 above), rather than pushing an Opus-specific output format onto every + other current/future consumer of this gateway. - **Concurrency:** the reference implementation serializes all inference behind one lock. Same starting point is reasonable here — revisit only if real latency/throughput needs show up. - **Auth:** consistent with every other service in this repo, no app-layer authorization — diff --git a/discord-bot/.env.example b/discord-bot/.env.example index bd58be6..3e8ca9a 100644 --- a/discord-bot/.env.example +++ b/discord-bot/.env.example @@ -2,6 +2,7 @@ DISCORD_TOKEN=your-bot-token-here GUILD_ID=your-guild-id-here HA_GATEWAY_ADDR=localhost:50051 AI_GATEWAY_ADDR=localhost:50052 +TTS_GATEWAY_ADDR=localhost:50053 OTEL_ENDPOINT= LOG_LEVEL=info LOG_FORMAT=text diff --git a/discord-bot/Dockerfile b/discord-bot/Dockerfile index 52ec1de..ae623b2 100644 --- a/discord-bot/Dockerfile +++ b/discord-bot/Dockerfile @@ -27,6 +27,14 @@ RUN CGO_ENABLED=0 GOOS=linux go build \ -ldflags="-s -w -X main.version=${VERSION}" \ -o /discord-bot ./cmd/bot -FROM gcr.io/distroless/static:nonroot +# /speak needs `ffmpeg` at runtime (github.com/jonas747/dca shells out to it to transcode +# tts-gateway's AAC output to Opus for Discord voice), which rules out a distroless base - +# same reasoning as tts-gateway/Dockerfile. +FROM ubuntu:22.04 +RUN apt-get update && apt-get install -y --no-install-recommends \ + ffmpeg \ + ca-certificates \ + && rm -rf /var/lib/apt/lists/* + COPY --from=builder /discord-bot /discord-bot ENTRYPOINT ["/discord-bot"] diff --git a/discord-bot/cmd/bot/main.go b/discord-bot/cmd/bot/main.go index acef256..c328ed4 100644 --- a/discord-bot/cmd/bot/main.go +++ b/discord-bot/cmd/bot/main.go @@ -14,6 +14,7 @@ import ( discordadapter "gitea.nik4nao.com/nik/home-services/discord-bot/internal/adapters/primary/discord" "gitea.nik4nao.com/nik/home-services/discord-bot/internal/adapters/secondary/aigateway" "gitea.nik4nao.com/nik/home-services/discord-bot/internal/adapters/secondary/gateway" + "gitea.nik4nao.com/nik/home-services/discord-bot/internal/adapters/secondary/ttsgateway" "gitea.nik4nao.com/nik/home-services/discord-bot/internal/app" "gitea.nik4nao.com/nik/home-services/discord-bot/internal/config" "gitea.nik4nao.com/nik/home-services/discord-bot/internal/logger" @@ -42,6 +43,7 @@ func main() { "version", version, "ha_gateway_addr", cfg.HAGatewayAddr, "ai_gateway_addr", cfg.AIGatewayAddr, + "tts_gateway_addr", cfg.TTSGatewayAddr, "discord_token", redactToken(cfg.DiscordToken), "tls_dir", cfg.TLSDir, "log_level", cfg.LogLevel, @@ -86,9 +88,20 @@ func main() { } }() + ttsClient, err := ttsgateway.New(ctx, cfg.TTSGatewayAddr, cfg.TLSDir, log) + if err != nil { + log.Error("tts-gateway client setup failed", "err", err) + os.Exit(1) + } + defer func() { + if err := ttsClient.Close(); err != nil { + log.Error("tts-gateway client close failed", "err", err) + } + }() + modelStore := modelstore.New() validator := modelvalidator.New(aiClient, 30*time.Second) - commandApp := app.NewCommandApp(haClient, aiClient, modelStore, validator) + commandApp := app.NewCommandApp(haClient, aiClient, modelStore, validator, ttsClient) tracker := discordadapter.NewTracker(context.Background()) // Discord-specific wiring stays at the edge so the app layer remains transport-agnostic. @@ -97,7 +110,9 @@ func main() { log.Error("create discord session failed", "err", err) os.Exit(1) } - session.Identify.Intents = discordgo.IntentsGuilds + // GuildVoiceStates is needed to resolve which voice channel the invoking + // user is currently in for /speak (see internal/adapters/primary/discord/voice.go). + session.Identify.Intents = discordgo.IntentsGuilds | discordgo.IntentsGuildVoiceStates handler := discordadapter.NewHandler(commandApp, tracker) handler.Register(session) @@ -121,6 +136,7 @@ func main() { "command_scope", scope, "ha_gateway_addr", cfg.HAGatewayAddr, "ai_gateway_addr", cfg.AIGatewayAddr, + "tts_gateway_addr", cfg.TTSGatewayAddr, "version", version, ) diff --git a/discord-bot/go.mod b/discord-bot/go.mod index 3ce4ca2..3bee9d7 100644 --- a/discord-bot/go.mod +++ b/discord-bot/go.mod @@ -6,6 +6,7 @@ require ( gitea.nik4nao.com/nik/home-services/gen v0.0.0 github.com/bwmarrin/discordgo v0.29.0 github.com/joho/godotenv v1.5.1 + github.com/jonas747/dca v0.0.0-20210930103944-155f5e5f0cc7 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.60.0 go.opentelemetry.io/otel v1.39.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.35.0 @@ -25,6 +26,7 @@ require ( github.com/google/uuid v1.6.0 // indirect github.com/gorilla/websocket v1.4.2 // indirect github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 // indirect + github.com/jonas747/ogg v0.0.0-20161220051205-b4f6f4cf3757 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.35.0 // indirect go.opentelemetry.io/proto/otlp v1.5.0 // indirect diff --git a/discord-bot/go.sum b/discord-bot/go.sum index b5765dd..15f9dc7 100644 --- a/discord-bot/go.sum +++ b/discord-bot/go.sum @@ -23,6 +23,10 @@ github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 h1:e9Rjr40Z98/clHv5Yg79Is0Ntos github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1/go.mod h1:tIxuGz/9mpox++sgp9fJjHO0+q1X9/UOWd798aAm22M= github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0= github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4= +github.com/jonas747/dca v0.0.0-20210930103944-155f5e5f0cc7 h1:Iw1pZVDiq4tY8hO7Wt1D1EqDC0BmMw/bFO/Rdt4gs9Q= +github.com/jonas747/dca v0.0.0-20210930103944-155f5e5f0cc7/go.mod h1:rxjYX9OJU81unMxQDHChU/lAiOhlY9MV+faPX/NmwLk= +github.com/jonas747/ogg v0.0.0-20161220051205-b4f6f4cf3757 h1:Kyv+zTfWIGRNaz/4+lS+CxvuKVZSKFz/6G8E3BKKBRs= +github.com/jonas747/ogg v0.0.0-20161220051205-b4f6f4cf3757/go.mod h1:cZnNmdLiLpihzgIVqiaQppi9Ts3D4qF/M45//yW35nI= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= diff --git a/discord-bot/internal/adapters/primary/discord/handler.go b/discord-bot/internal/adapters/primary/discord/handler.go index 42e2451..6b60bde 100644 --- a/discord-bot/internal/adapters/primary/discord/handler.go +++ b/discord-bot/internal/adapters/primary/discord/handler.go @@ -38,10 +38,12 @@ type commandHandler interface { HandleAIModelSet(ctx context.Context, name string) (string, error) HandleAIModelGet(ctx context.Context) (string, error) HandleAIModelList(ctx context.Context) (string, error) + HandleSpeak(ctx context.Context, speakerName, text string) (audio []byte, mimeType string, err error) AutocompleteLights(ctx context.Context) ([]apppkg.Choice, error) AutocompleteSwitches(ctx context.Context) ([]apppkg.Choice, error) AutocompleteClimates(ctx context.Context) ([]apppkg.Choice, error) AutocompleteAIModels(ctx context.Context) ([]apppkg.Choice, error) + AutocompleteSpeakers(ctx context.Context) ([]apppkg.Choice, error) } // Handler adapts Discord interactions to the command application layer. @@ -84,7 +86,7 @@ func (h *Handler) handleApplicationCommand(ctx context.Context, s *discordgo.Ses data := i.ApplicationCommandData() command := data.Name - if len(data.Options) > 0 { + if len(data.Options) > 0 && isSubcommandOption(data.Options[0]) { command += "." + data.Options[0].Name if len(data.Options[0].Options) > 0 && data.Options[0].Type == discordgo.ApplicationCommandOptionSubCommandGroup { command += "." + data.Options[0].Options[0].Name @@ -104,6 +106,15 @@ func (h *Handler) handleApplicationCommand(ctx context.Context, s *discordgo.Ses ), ) defer span.End() + + // speak is a flat command (speaker/text values directly under data.Options, no + // subcommand wrapper) unlike light/switch/ac/ai, so it's dispatched before the + // subcommand-assuming logic below rather than through commandPath's switch. + if data.Name == "speak" { + h.handleSpeak(ctx, s, i, data, start) + return + } + if len(data.Options) == 0 { h.respondError(ctx, s, i.Interaction, true, start, fmt.Errorf("missing subcommand")) return @@ -305,6 +316,57 @@ func (h *Handler) handleApplicationCommand(ctx context.Context, s *discordgo.Ses } } +// handleSpeak synthesizes speech via tts-gateway and plays it in the invoking +// user's current voice channel, following the same deferred-ack + background + +// followup shape as the ai.query case (synthesis + voice join + playback will +// exceed Discord's ~3s initial-ack window). +func (h *Handler) handleSpeak(ctx context.Context, s *discordgo.Session, i *discordgo.InteractionCreate, data discordgo.ApplicationCommandInteractionData, start time.Time) { + log := logger.FromContext(ctx) + if err := h.deferResponse(s, i.Interaction, true); err != nil { + log.Error("discord response failed", + "duration_ms", time.Since(start).Milliseconds(), + "error", err.Error(), + ) + return + } + + speakerName := stringOption(data.Options, "speaker") + text := stringOption(data.Options, "text") + guildID := i.GuildID + userID := "" + if i.Member != nil && i.Member.User != nil { + userID = i.Member.User.ID + } else if i.User != nil { + userID = i.User.ID + } + + interaction := i.Interaction + reqLog := log + h.tracker.Go(func(trackerCtx context.Context) { + asyncCtx := logger.WithLogger(trackerCtx, reqLog) + asyncCtx, cancel := context.WithTimeout(asyncCtx, 2*time.Minute) + defer cancel() + + audio, mimeType, err := h.app.HandleSpeak(asyncCtx, speakerName, text) + if err == nil { + err = speak(asyncCtx, s, guildID, userID, audio, mimeType) + } + + msg := fmt.Sprintf("Spoke as `%s`.", speakerName) + if err != nil { + reqLog.Error("speak command failed", "error", err.Error()) + msg = mapSpeakError(err) + } + if _, followErr := s.FollowupMessageCreate(interaction, true, &discordgo.WebhookParams{ + Content: msg, + Flags: discordgo.MessageFlagsEphemeral, + }); followErr != nil { + reqLog.Error("discord response failed", "error", followErr.Error()) + } + }) + log.Info("command handled", "duration_ms", time.Since(start).Milliseconds()) +} + // handleAutocomplete keeps autocomplete fast by filtering already-fetched choices // instead of requiring Discord-specific logic in the app layer. func (h *Handler) handleAutocomplete(ctx context.Context, s *discordgo.Session, i *discordgo.InteractionCreate) { @@ -326,6 +388,10 @@ func (h *Handler) handleAutocomplete(ctx context.Context, s *discordgo.Session, if focusedOptionName(data) == "name" { choices, err = h.app.AutocompleteAIModels(ctx) } + case "speak": + if focusedOptionName(data) == "speaker" { + choices, err = h.app.AutocompleteSpeakers(ctx) + } default: choices = nil } @@ -419,6 +485,24 @@ func (h *Handler) followup(ctx context.Context, s *discordgo.Session, interactio } } +func mapSpeakError(err error) string { + if errors.Is(err, ErrNotInVoiceChannel) { + return "Join a voice channel first." + } + if errors.Is(err, context.DeadlineExceeded) { + return "Speaking took too long. Try again later." + } + if status, ok := grpcstatus.FromError(err); ok { + switch status.Code() { + case codes.InvalidArgument: + return "Unknown speaker. Try /speak again with a different name." + case codes.Unavailable: + return "The TTS service is unreachable right now. Try again in a moment." + } + } + return "Sorry, something went wrong speaking that." +} + func mapAIError(err error) string { if errors.Is(err, context.DeadlineExceeded) { return "The AI took too long to respond. Try again later." @@ -448,7 +532,14 @@ func interactionLogger(ctx context.Context, i *discordgo.InteractionCreate) *slo } func requiredStringOption(sub *discordgo.ApplicationCommandInteractionDataOption, name string) string { - for _, opt := range sub.Options { + return stringOption(sub.Options, name) +} + +// stringOption looks up a named string-valued option in a flat options slice - +// used directly for flat commands like /speak, and via requiredStringOption for +// commands nested one level under a subcommand. +func stringOption(opts []*discordgo.ApplicationCommandInteractionDataOption, name string) string { + for _, opt := range opts { if opt.Name == name { return opt.StringValue() } @@ -456,6 +547,12 @@ func requiredStringOption(sub *discordgo.ApplicationCommandInteractionDataOption return "" } +// isSubcommandOption reports whether opt represents a subcommand/subcommand-group +// node, as opposed to a plain value option (e.g. a flat command's own arguments). +func isSubcommandOption(opt *discordgo.ApplicationCommandInteractionDataOption) bool { + return opt.Type == discordgo.ApplicationCommandOptionSubCommand || opt.Type == discordgo.ApplicationCommandOptionSubCommandGroup +} + func optionalUint32Option(sub *discordgo.ApplicationCommandInteractionDataOption, name string) *uint32 { for _, opt := range sub.Options { if opt.Name == name { @@ -468,6 +565,9 @@ func optionalUint32Option(sub *discordgo.ApplicationCommandInteractionDataOption func focusedOptionValue(data discordgo.ApplicationCommandInteractionData) string { for _, sub := range data.Options { + if sub.Focused { + return sub.StringValue() + } if sub.Type == discordgo.ApplicationCommandOptionSubCommandGroup { for _, nested := range sub.Options { for _, opt := range nested.Options { @@ -488,6 +588,9 @@ func focusedOptionValue(data discordgo.ApplicationCommandInteractionData) string func focusedOptionName(data discordgo.ApplicationCommandInteractionData) string { for _, sub := range data.Options { + if sub.Focused { + return sub.Name + } if sub.Type == discordgo.ApplicationCommandOptionSubCommandGroup { for _, nested := range sub.Options { for _, opt := range nested.Options { diff --git a/discord-bot/internal/adapters/primary/discord/handler_test.go b/discord-bot/internal/adapters/primary/discord/handler_test.go new file mode 100644 index 0000000..6fb19eb --- /dev/null +++ b/discord-bot/internal/adapters/primary/discord/handler_test.go @@ -0,0 +1,108 @@ +package discord + +import ( + "testing" + + "github.com/bwmarrin/discordgo" +) + +func TestStringOption(t *testing.T) { + opts := []*discordgo.ApplicationCommandInteractionDataOption{ + {Name: "speaker", Type: discordgo.ApplicationCommandOptionString, Value: "Rice Shower"}, + {Name: "text", Type: discordgo.ApplicationCommandOptionString, Value: "hello"}, + } + + if got := stringOption(opts, "speaker"); got != "Rice Shower" { + t.Fatalf("stringOption(speaker) = %q, want %q", got, "Rice Shower") + } + if got := stringOption(opts, "missing"); got != "" { + t.Fatalf("stringOption(missing) = %q, want empty", got) + } +} + +func TestIsSubcommandOption(t *testing.T) { + tests := []struct { + name string + typ discordgo.ApplicationCommandOptionType + want bool + }{ + {name: "subcommand", typ: discordgo.ApplicationCommandOptionSubCommand, want: true}, + {name: "subcommand group", typ: discordgo.ApplicationCommandOptionSubCommandGroup, want: true}, + {name: "plain string option", typ: discordgo.ApplicationCommandOptionString, want: false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := isSubcommandOption(&discordgo.ApplicationCommandInteractionDataOption{Type: tt.typ}) + if got != tt.want { + t.Fatalf("isSubcommandOption(%v) = %v, want %v", tt.typ, got, tt.want) + } + }) + } +} + +func TestFocusedOptionNameAndValue(t *testing.T) { + tests := []struct { + name string + data discordgo.ApplicationCommandInteractionData + wantName string + wantValue string + }{ + { + name: "flat command - top-level option focused", + data: discordgo.ApplicationCommandInteractionData{ + Name: "speak", + Options: []*discordgo.ApplicationCommandInteractionDataOption{ + {Name: "speaker", Type: discordgo.ApplicationCommandOptionString, Value: "Rice", Focused: true}, + {Name: "text", Type: discordgo.ApplicationCommandOptionString, Value: "hi"}, + }, + }, + wantName: "speaker", + wantValue: "Rice", + }, + { + name: "nested subcommand option focused (regression)", + data: discordgo.ApplicationCommandInteractionData{ + Name: "ai", + Options: []*discordgo.ApplicationCommandInteractionDataOption{ + { + Name: "model", + Type: discordgo.ApplicationCommandOptionSubCommandGroup, + Options: []*discordgo.ApplicationCommandInteractionDataOption{ + { + Name: "set", + Type: discordgo.ApplicationCommandOptionSubCommand, + Options: []*discordgo.ApplicationCommandInteractionDataOption{ + {Name: "name", Type: discordgo.ApplicationCommandOptionString, Value: "llama", Focused: true}, + }, + }, + }, + }, + }, + }, + wantName: "name", + wantValue: "llama", + }, + { + name: "nothing focused", + data: discordgo.ApplicationCommandInteractionData{ + Name: "speak", + Options: []*discordgo.ApplicationCommandInteractionDataOption{ + {Name: "speaker", Type: discordgo.ApplicationCommandOptionString, Value: "Rice"}, + }, + }, + wantName: "", + wantValue: "", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := focusedOptionName(tt.data); got != tt.wantName { + t.Fatalf("focusedOptionName() = %q, want %q", got, tt.wantName) + } + if got := focusedOptionValue(tt.data); got != tt.wantValue { + t.Fatalf("focusedOptionValue() = %q, want %q", got, tt.wantValue) + } + }) + } +} diff --git a/discord-bot/internal/adapters/primary/discord/register.go b/discord-bot/internal/adapters/primary/discord/register.go index 440fc00..067393a 100644 --- a/discord-bot/internal/adapters/primary/discord/register.go +++ b/discord-bot/internal/adapters/primary/discord/register.go @@ -207,6 +207,25 @@ func RegisterCommands(s *discordgo.Session, guildID string) error { }, }, }, + { + Name: "speak", + Description: "Speak text in your current voice channel", + Options: []*discordgo.ApplicationCommandOption{ + { + Type: discordgo.ApplicationCommandOptionString, + Name: "speaker", + Description: "Speaker voice", + Required: true, + Autocomplete: true, + }, + { + Type: discordgo.ApplicationCommandOptionString, + Name: "text", + Description: "Text to speak", + Required: true, + }, + }, + }, } if _, err := s.ApplicationCommandBulkOverwrite(appID, guildID, commands); err != nil { diff --git a/discord-bot/internal/adapters/primary/discord/voice.go b/discord-bot/internal/adapters/primary/discord/voice.go new file mode 100644 index 0000000..ce8916c --- /dev/null +++ b/discord-bot/internal/adapters/primary/discord/voice.go @@ -0,0 +1,131 @@ +package discord + +import ( + "context" + "errors" + "fmt" + "io" + "os" + "time" + + "gitea.nik4nao.com/nik/home-services/discord-bot/internal/logger" + "github.com/bwmarrin/discordgo" + "github.com/jonas747/dca" +) + +// ErrNotInVoiceChannel is returned when the invoking user isn't in a voice channel. +var ErrNotInVoiceChannel = errors.New("join a voice channel first") + +// resolveVoiceChannel returns the voice channel the given guild member currently +// occupies, or ErrNotInVoiceChannel if they aren't in one. +func resolveVoiceChannel(s *discordgo.Session, guildID, userID string) (string, error) { + vs, err := s.State.VoiceState(guildID, userID) + if err != nil { + return "", ErrNotInVoiceChannel + } + return vs.ChannelID, nil +} + +// speak joins the invoking user's current voice channel, transcodes audio (as +// returned by tts-gateway, e.g. AAC) to Opus via ffmpeg/dca, streams it, then +// leaves. This is Discord-specific playback plumbing that the transport-agnostic +// app layer has no business knowing about - see internal/app/command.go's +// HandleSpeak for the RPC half of this feature. +func speak(ctx context.Context, s *discordgo.Session, guildID, userID string, audio []byte, mimeType string) error { + log := logger.FromContext(ctx).With("mime_type", mimeType) + + channelID, err := resolveVoiceChannel(s, guildID, userID) + if err != nil { + return err + } + + vc, err := s.ChannelVoiceJoin(guildID, channelID, false, true) + if err != nil { + return fmt.Errorf("join voice channel: %w", err) + } + defer func() { + if err := vc.Disconnect(); err != nil { + log.Error("voice disconnect failed", "error", err.Error()) + } + }() + + if err := waitForVoiceReady(ctx, vc); err != nil { + return err + } + + // dca shells out to ffmpeg, which needs a seekable input for containers like the + // AAC/M4A tts-gateway returns - its own moov atom sits at the end of the file (see + // tts-gateway/internal/adapters/secondary/ffmpeg/ffmpeg.go), so a non-seekable pipe + // input would fail. Write to a temp file first, mirroring that same adapter's idiom + // on the output side. + tmpPath, err := writeTempAudio(audio) + if err != nil { + return err + } + defer os.Remove(tmpPath) + + encodeSession, err := dca.EncodeFile(tmpPath, dca.StdEncodeOptions) + if err != nil { + return fmt.Errorf("encode audio to opus: %w", err) + } + defer encodeSession.Cleanup() + + if err := vc.Speaking(true); err != nil { + return fmt.Errorf("set speaking state: %w", err) + } + defer func() { + if err := vc.Speaking(false); err != nil { + log.Error("clear speaking state failed", "error", err.Error()) + } + }() + + done := make(chan error, 1) + dca.NewStream(encodeSession, vc, done) + + select { + case err := <-done: + if err != nil && err != io.EOF { + return fmt.Errorf("stream audio: %w", err) + } + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +// writeTempAudio persists synthesized audio to a temp file for ffmpeg to read from. +func writeTempAudio(audio []byte) (string, error) { + tmpFile, err := os.CreateTemp("", "discord-bot-speak-*") + if err != nil { + return "", fmt.Errorf("create temp audio file: %w", err) + } + tmpPath := tmpFile.Name() + + if _, err := tmpFile.Write(audio); err != nil { + _ = tmpFile.Close() + os.Remove(tmpPath) + return "", fmt.Errorf("write temp audio file: %w", err) + } + if err := tmpFile.Close(); err != nil { + os.Remove(tmpPath) + return "", fmt.Errorf("close temp audio file: %w", err) + } + return tmpPath, nil +} + +// waitForVoiceReady polls until the voice connection's UDP socket is ready to +// send, or ctx is done - discordgo has no blocking/channel-based signal for this. +func waitForVoiceReady(ctx context.Context, vc *discordgo.VoiceConnection) error { + ticker := time.NewTicker(50 * time.Millisecond) + defer ticker.Stop() + for { + if vc.Ready { + return nil + } + select { + case <-ticker.C: + case <-ctx.Done(): + return fmt.Errorf("wait for voice connection ready: %w", ctx.Err()) + } + } +} diff --git a/discord-bot/internal/adapters/primary/discord/voice_test.go b/discord-bot/internal/adapters/primary/discord/voice_test.go new file mode 100644 index 0000000..2089401 --- /dev/null +++ b/discord-bot/internal/adapters/primary/discord/voice_test.go @@ -0,0 +1,73 @@ +package discord + +import ( + "errors" + "testing" + + "github.com/bwmarrin/discordgo" +) + +func newTestSession(t *testing.T, guildID string, voiceStates []*discordgo.VoiceState) *discordgo.Session { + t.Helper() + state := discordgo.NewState() + if err := state.GuildAdd(&discordgo.Guild{ID: guildID, VoiceStates: voiceStates}); err != nil { + t.Fatalf("GuildAdd() error = %v", err) + } + return &discordgo.Session{State: state} +} + +func TestResolveVoiceChannel(t *testing.T) { + const guildID = "guild-1" + const userID = "user-1" + + tests := []struct { + name string + voiceStates []*discordgo.VoiceState + userID string + wantChannel string + wantErr error + }{ + { + name: "user in a voice channel", + voiceStates: []*discordgo.VoiceState{ + {GuildID: guildID, UserID: userID, ChannelID: "channel-1"}, + }, + userID: userID, + wantChannel: "channel-1", + }, + { + name: "user not in any voice channel", + voiceStates: nil, + userID: userID, + wantErr: ErrNotInVoiceChannel, + }, + { + name: "user in a different guild's voice channel is not found", + voiceStates: []*discordgo.VoiceState{ + {GuildID: guildID, UserID: "someone-else", ChannelID: "channel-1"}, + }, + userID: userID, + wantErr: ErrNotInVoiceChannel, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + session := newTestSession(t, guildID, tt.voiceStates) + + got, err := resolveVoiceChannel(session, guildID, tt.userID) + if tt.wantErr != nil { + if !errors.Is(err, tt.wantErr) { + t.Fatalf("resolveVoiceChannel() error = %v, want %v", err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("resolveVoiceChannel() error = %v", err) + } + if got != tt.wantChannel { + t.Fatalf("resolveVoiceChannel() = %q, want %q", got, tt.wantChannel) + } + }) + } +} diff --git a/discord-bot/internal/adapters/secondary/ttsgateway/client.go b/discord-bot/internal/adapters/secondary/ttsgateway/client.go new file mode 100644 index 0000000..8f0fe58 --- /dev/null +++ b/discord-bot/internal/adapters/secondary/ttsgateway/client.go @@ -0,0 +1,124 @@ +package ttsgateway + +import ( + "context" + "crypto/tls" + "crypto/x509" + "fmt" + "log/slog" + "os" + "path/filepath" + "time" + + "gitea.nik4nao.com/nik/home-services/discord-bot/internal/logger" + ttsv1 "gitea.nik4nao.com/nik/home-services/gen/tts/v1" + "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials" + "google.golang.org/grpc/credentials/insecure" +) + +// Client implements the app's TTS driven port over gRPC. +type Client struct { + conn *grpc.ClientConn + client ttsv1.TTSServiceClient + log *slog.Logger +} + +// New constructs a gRPC client for the internal tts-gateway service. +func New(ctx context.Context, addr, tlsDir string, log *slog.Logger) (*Client, error) { + transportCreds := insecure.NewCredentials() + if tlsDir != "" { + creds, err := loadTransportCredentials(tlsDir) + if err != nil { + return nil, fmt.Errorf("load mTLS credentials: %w", err) + } + transportCreds = creds + } + + conn, err := grpc.NewClient( + addr, + grpc.WithTransportCredentials(transportCreds), + grpc.WithStatsHandler(otelgrpc.NewClientHandler()), + ) + if err != nil { + return nil, fmt.Errorf("dial tts-gateway: %w", err) + } + + return &Client{ + conn: conn, + client: ttsv1.NewTTSServiceClient(conn), + log: log, + }, nil +} + +// Close closes the underlying gRPC connection. +func (c *Client) Close() error { + if err := c.conn.Close(); err != nil { + return fmt.Errorf("close tts-gateway client: %w", err) + } + return nil +} + +// Synthesize forwards a speech synthesis request to tts-gateway. noise_scale/ +// noise_scale_w/length_scale are left unset so tts-gateway applies its own defaults. +func (c *Client) Synthesize(ctx context.Context, speakerName, text string) ([]byte, string, error) { + start := time.Now() + log := logger.FromContext(ctx).With("grpc.method", "TTSService/Synthesize") + resp, err := c.client.Synthesize(ctx, &ttsv1.SynthesizeRequest{ + SpeakerName: speakerName, + Text: text, + }) + if err != nil { + log.Error("grpc call failed", + "duration_ms", time.Since(start).Milliseconds(), + "error", err.Error(), + ) + return nil, "", fmt.Errorf("synthesize speech: %w", err) + } + log.Debug("grpc call completed", "duration_ms", time.Since(start).Milliseconds()) + return resp.GetAudio(), resp.GetMimeType(), nil +} + +// ListSpeakers returns the speaker roster from tts-gateway, optionally filtered by search. +func (c *Client) ListSpeakers(ctx context.Context, search string) ([]string, error) { + start := time.Now() + log := logger.FromContext(ctx).With("grpc.method", "TTSService/ListSpeakers") + resp, err := c.client.ListSpeakers(ctx, &ttsv1.ListSpeakersRequest{Search: search}) + if err != nil { + log.Error("grpc call failed", + "duration_ms", time.Since(start).Milliseconds(), + "error", err.Error(), + ) + return nil, fmt.Errorf("list tts-gateway speakers: %w", err) + } + log.Debug("grpc call completed", "duration_ms", time.Since(start).Milliseconds()) + return append([]string(nil), resp.GetSpeakerNames()...), nil +} + +func loadTransportCredentials(tlsDir string) (credentials.TransportCredentials, error) { + cert, err := tls.LoadX509KeyPair( + filepath.Join(tlsDir, "tls.crt"), + filepath.Join(tlsDir, "tls.key"), + ) + if err != nil { + return nil, fmt.Errorf("load client key pair: %w", err) + } + + caPEM, err := os.ReadFile(filepath.Join(tlsDir, "ca.crt")) + if err != nil { + return nil, fmt.Errorf("read server CA: %w", err) + } + + rootCAs := x509.NewCertPool() + if !rootCAs.AppendCertsFromPEM(caPEM) { + return nil, fmt.Errorf("append server CA: invalid PEM") + } + + return credentials.NewTLS(&tls.Config{ + Certificates: []tls.Certificate{cert}, + RootCAs: rootCAs, + ServerName: "tts-gateway.home-services.svc.cluster.local", + MinVersion: tls.VersionTLS13, + }), nil +} diff --git a/discord-bot/internal/app/command.go b/discord-bot/internal/app/command.go index b6ebb55..f5e7c21 100644 --- a/discord-bot/internal/app/command.go +++ b/discord-bot/internal/app/command.go @@ -23,11 +23,12 @@ type CommandApp struct { ai driven.AIGateway models *modelstore.Store validator *modelvalidator.Validator + tts driven.TTSGateway } // NewCommandApp constructs the Discord command application service. -func NewCommandApp(ha driven.HAGateway, ai driven.AIGateway, models *modelstore.Store, validator *modelvalidator.Validator) *CommandApp { - return &CommandApp{ha: ha, ai: ai, models: models, validator: validator} +func NewCommandApp(ha driven.HAGateway, ai driven.AIGateway, models *modelstore.Store, validator *modelvalidator.Validator, tts driven.TTSGateway) *CommandApp { + return &CommandApp{ha: ha, ai: ai, models: models, validator: validator, tts: tts} } // HandleLightList formats discovered lights into a monospace-friendly response. @@ -279,6 +280,33 @@ func (a *CommandApp) HandleAIModelList(ctx context.Context) (string, error) { return strings.Join(lines, "\n"), nil } +// HandleSpeak synthesizes speech via tts-gateway for the invoking Discord adapter +// to play back in a voice channel; the app layer stays transport-agnostic and +// leaves voice-channel playback itself to the Discord adapter. +func (a *CommandApp) HandleSpeak(ctx context.Context, speakerName, text string) ([]byte, string, error) { + audio, mimeType, err := a.tts.Synthesize(ctx, speakerName, text) + if err != nil { + return nil, "", fmt.Errorf("handle speak: %w", err) + } + return audio, mimeType, nil +} + +// AutocompleteSpeakers returns the full tts-gateway speaker roster for the /speak +// command; handleAutocomplete's existing substring filter narrows it client-side, +// same as the light/switch/ac autocompletes above. +func (a *CommandApp) AutocompleteSpeakers(ctx context.Context) ([]Choice, error) { + speakers, err := a.tts.ListSpeakers(ctx, "") + if err != nil { + return nil, fmt.Errorf("autocomplete speakers: %w", err) + } + + choices := make([]Choice, 0, len(speakers)) + for _, speaker := range speakers { + choices = append(choices, Choice{Label: speaker, Value: speaker}) + } + return choices, nil +} + // AutocompleteAIModels returns model names for the /ai model set command. func (a *CommandApp) AutocompleteAIModels(ctx context.Context) ([]Choice, error) { models, err := a.validator.Known(ctx) diff --git a/discord-bot/internal/app/command_test.go b/discord-bot/internal/app/command_test.go index 6721884..b95d821 100644 --- a/discord-bot/internal/app/command_test.go +++ b/discord-bot/internal/app/command_test.go @@ -35,6 +35,11 @@ type mockAIGateway struct { listModelsFunc func(ctx context.Context) ([]string, error) } +type mockTTSGateway struct { + synthesizeFunc func(ctx context.Context, speakerName, text string) ([]byte, string, error) + listSpeakersFunc func(ctx context.Context, search string) ([]string, error) +} + func (m *mockHAGateway) ListLights(ctx context.Context) ([]driven.Light, error) { if m.listLightsFunc == nil { return nil, nil @@ -147,8 +152,22 @@ func (m *mockAIGateway) ListModels(ctx context.Context) ([]string, error) { return m.listModelsFunc(ctx) } +func (m *mockTTSGateway) Synthesize(ctx context.Context, speakerName, text string) ([]byte, string, error) { + if m.synthesizeFunc == nil { + return nil, "", nil + } + return m.synthesizeFunc(ctx, speakerName, text) +} + +func (m *mockTTSGateway) ListSpeakers(ctx context.Context, search string) ([]string, error) { + if m.listSpeakersFunc == nil { + return nil, nil + } + return m.listSpeakersFunc(ctx, search) +} + func newTestCommandApp(ha *mockHAGateway, ai *mockAIGateway) *CommandApp { - return NewCommandApp(ha, ai, modelstore.New(), modelvalidator.New(ai, time.Minute)) + return NewCommandApp(ha, ai, modelstore.New(), modelvalidator.New(ai, time.Minute), &mockTTSGateway{}) } func TestCommandAppHandleLightList(t *testing.T) { @@ -1081,7 +1100,7 @@ func TestCommandAppHandleAIQuery(t *testing.T) { } return "Turning on Kitchen.", "llama3:latest", nil }, - }, store, modelvalidator.New(&mockAIGateway{}, time.Minute)) + }, store, modelvalidator.New(&mockAIGateway{}, time.Minute), &mockTTSGateway{}) got, err := app.HandleAIQuery(context.Background(), "turn on kitchen") if err != nil { @@ -1092,6 +1111,116 @@ func TestCommandAppHandleAIQuery(t *testing.T) { } } +func TestCommandAppHandleSpeak(t *testing.T) { + tests := []struct { + name string + speakerName string + text string + audio []byte + mimeType string + synthErr error + wantErr string + }{ + { + name: "synthesize success", + speakerName: "Rice Shower", + text: "おはようございます", + audio: []byte{0x00, 0x01, 0x02}, + mimeType: "audio/aac", + }, + { + name: "synthesize error propagated", + speakerName: "Unknown Speaker", + text: "hello", + synthErr: errors.New("boom"), + wantErr: "handle speak: boom", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var gotSpeaker, gotText string + app := NewCommandApp(&mockHAGateway{}, &mockAIGateway{}, modelstore.New(), modelvalidator.New(&mockAIGateway{}, time.Minute), &mockTTSGateway{ + synthesizeFunc: func(ctx context.Context, speakerName, text string) ([]byte, string, error) { + gotSpeaker, gotText = speakerName, text + if tt.synthErr != nil { + return nil, "", tt.synthErr + } + return tt.audio, tt.mimeType, nil + }, + }) + + audio, mimeType, err := app.HandleSpeak(context.Background(), tt.speakerName, tt.text) + if tt.wantErr != "" { + if err == nil || err.Error() != tt.wantErr { + t.Fatalf("HandleSpeak() error = %v, want %q", err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("HandleSpeak() error = %v", err) + } + if gotSpeaker != tt.speakerName || gotText != tt.text { + t.Fatalf("Synthesize() called with (%q, %q), want (%q, %q)", gotSpeaker, gotText, tt.speakerName, tt.text) + } + if !reflect.DeepEqual(audio, tt.audio) || mimeType != tt.mimeType { + t.Fatalf("HandleSpeak() = (%#v, %q), want (%#v, %q)", audio, mimeType, tt.audio, tt.mimeType) + } + }) + } +} + +func TestCommandAppAutocompleteSpeakers(t *testing.T) { + tests := []struct { + name string + speakers []string + listErr error + want []Choice + wantErr string + }{ + { + name: "maps speaker names to choices", + speakers: []string{"Rice Shower", "Special Week"}, + want: []Choice{ + {Label: "Rice Shower", Value: "Rice Shower"}, + {Label: "Special Week", Value: "Special Week"}, + }, + }, + { + name: "ListSpeakers error", + listErr: errors.New("boom"), + wantErr: "autocomplete speakers: boom", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + app := NewCommandApp(&mockHAGateway{}, &mockAIGateway{}, modelstore.New(), modelvalidator.New(&mockAIGateway{}, time.Minute), &mockTTSGateway{ + listSpeakersFunc: func(ctx context.Context, search string) ([]string, error) { + if tt.listErr != nil { + return nil, tt.listErr + } + return tt.speakers, nil + }, + }) + + got, err := app.AutocompleteSpeakers(context.Background()) + if tt.wantErr != "" { + if err == nil || err.Error() != tt.wantErr { + t.Fatalf("AutocompleteSpeakers() error = %v, want %q", err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("AutocompleteSpeakers() error = %v", err) + } + if !reflect.DeepEqual(got, tt.want) { + t.Fatalf("AutocompleteSpeakers() = %#v, want %#v", got, tt.want) + } + }) + } +} + func TestCommandAppHandleAIModelSet(t *testing.T) { app := newTestCommandApp(&mockHAGateway{}, &mockAIGateway{ listModelsFunc: func(ctx context.Context) ([]string, error) { diff --git a/discord-bot/internal/config/config.go b/discord-bot/internal/config/config.go index a3b9891..4292762 100644 --- a/discord-bot/internal/config/config.go +++ b/discord-bot/internal/config/config.go @@ -7,14 +7,15 @@ import ( // Config holds runtime configuration for the Discord bot process. type Config struct { - DiscordToken string - GuildID string - HAGatewayAddr string - AIGatewayAddr string - TLSDir string - OTELEndpoint string - LogLevel string - LogFormat string + DiscordToken string + GuildID string + HAGatewayAddr string + AIGatewayAddr string + TTSGatewayAddr string + TLSDir string + OTELEndpoint string + LogLevel string + LogFormat string } // Load reads Discord bot configuration from environment variables. @@ -30,14 +31,15 @@ func Load() (*Config, error) { } return &Config{ - DiscordToken: token, - GuildID: os.Getenv("GUILD_ID"), - HAGatewayAddr: addr, - AIGatewayAddr: getenvDefault("AI_GATEWAY_ADDR", "ai-gateway.home-services.svc.cluster.local:50052"), - TLSDir: os.Getenv("TLS_DIR"), - OTELEndpoint: os.Getenv("OTEL_ENDPOINT"), - LogLevel: getenvDefault("LOG_LEVEL", "info"), - LogFormat: getenvDefault("LOG_FORMAT", "json"), + DiscordToken: token, + GuildID: os.Getenv("GUILD_ID"), + HAGatewayAddr: addr, + AIGatewayAddr: getenvDefault("AI_GATEWAY_ADDR", "ai-gateway.home-services.svc.cluster.local:50052"), + TTSGatewayAddr: getenvDefault("TTS_GATEWAY_ADDR", "tts-gateway.home-services.svc.cluster.local:50053"), + TLSDir: os.Getenv("TLS_DIR"), + OTELEndpoint: os.Getenv("OTEL_ENDPOINT"), + LogLevel: getenvDefault("LOG_LEVEL", "info"), + LogFormat: getenvDefault("LOG_FORMAT", "json"), }, nil } diff --git a/discord-bot/internal/core/ports/driven/tts.go b/discord-bot/internal/core/ports/driven/tts.go new file mode 100644 index 0000000..1c74838 --- /dev/null +++ b/discord-bot/internal/core/ports/driven/tts.go @@ -0,0 +1,9 @@ +package driven + +import "context" + +// TTSGateway exposes the text-to-speech synthesis API used by the Discord bot. +type TTSGateway interface { + Synthesize(ctx context.Context, speakerName, text string) (audio []byte, mimeType string, err error) + ListSpeakers(ctx context.Context, search string) ([]string, error) +} diff --git a/handoff.md b/handoff.md new file mode 100644 index 0000000..c6930d5 --- /dev/null +++ b/handoff.md @@ -0,0 +1,90 @@ +# tts-gateway Handoff + +Status as of 2026-07-25: Phases 0–6 of `TTS_GATEWAY_PLAN.md` are done, and the deployed service is +confirmed working in production (real `grpcurl` call through `kubectl port-forward`, real GPU +inference, real playable audio). This file is a punch list for picking the remaining work up in a +fresh session — it doesn't re-explain things that are already documented elsewhere; it points to +where. + +**Update (later same day):** items 1 and 3 below are now done — model artifact distribution is a +versioned image + k8s `initContainer` (see `tts-gateway/README.md`), and `discord-bot` has a +working `/speak` command (see `TTS_GATEWAY_PLAN.md` Phase 6). Only item 2 (mTLS) remains, left +untouched per an explicit decision to verify these two first. The two "what's left" entries below +are kept as-written for their historical reasoning/decision trail rather than rewritten in place. + +## Where things stand + +- `tts-gateway` (Go) + `tts-sidecar` (Python/libtorch) are both built, containerized, and running + in Kubernetes (`home-services` namespace). +- CI (`.gitea/workflows/ci.yaml`) builds and pushes both images on every push to `main`, gated by + path-filtered per-service rebuilds and using registry-based Docker layer caching. +- Full history — the Phase 0 ONNX-export/g2p investigation, why there's a Python sidecar instead + of pure Go, the exact `japanese_cleaners` pipeline verification — is in `TTS_GATEWAY_PLAN.md` + and `tmp/reference/uma-tts-api/spike/FINDINGS.md`. Read those before re-deriving anything; the + ONNX export failure in particular took several iterations to characterize correctly and isn't + worth re-investigating from scratch. + +## What's left + +1. **Model artifact distribution isn't automated.** The k8s Deployment + (`~/repo/homelab/manifests/home-services/tts-gateway.yaml` — separate repo, see below) + `hostPath`-mounts `/data/tts-gateway` on the `nik-gpu` node into the sidecar container. + Someone has to manually place `G_790000.pth` + `uma.json` there before the pod goes Ready — no + init-container download, no PVC, nothing automated. `TTS_GATEWAY_PLAN.md`'s Phase 5 flagged + this as an open decision; it's still open, just worked around. If asked to fix it: options are + a PVC populated by an init-container download step, baking the checkpoint into a private image + layer, or similar — weigh against the checkpoint being ~455MB and not something to casually put + in a git-tracked Dockerfile context. + +2. **mTLS is off.** `tts-gateway.yaml` comments out the `TLS_DIR` env var and its volume mount, + for plaintext `grpcurl` testing from outside the cluster during initial rollout — a live TODO, + not a permanent decision. Every other service in this repo assumes internal-network-or-mTLS as + its only access boundary (`CLAUDE.md`'s "Configuration Notes"). Re-enabling: uncomment the env + var + mount in that manifest, and set up a `tts-gateway-tls` secret the same way + `ha-gateway-sealed.yaml`/`ha-gateway-secret.sh` do it for ha-gateway (tts-gateway doesn't + currently have its own `-secret.sh`/`-sealed.yaml` pair — it'll need one, since unlike + ha-gateway/discord-bot it has no other secrets today, so this would be its first). + +3. **Phase 6 (discord-bot integration) not started.** A `/speak`-style Discord slash command + calling `tts-gateway`, following the existing pattern — `discord-bot` already has gRPC clients + for `ha-gateway` and `ai-gateway` at `discord-bot/internal/adapters/secondary/{gateway,aigateway}`; + a new `internal/adapters/secondary/ttsgateway` client would follow the same shape. Needs a + product decision first: keep AAC (parity with what's already implemented) or switch to Opus + (Discord's native voice codec) — deliberately deferred in `TTS_GATEWAY_PLAN.md`'s open + questions until this exact moment, not yet decided. + +## Things worth knowing before touching this + +- **`tts-gateway` and `tts-sidecar` share one Kubernetes Pod**, not two separate Deployments — the + Go gateway reaches the sidecar over `localhost:50054`. This mirrors the `--network host` setup + documented in `tts-gateway/README.md` for local Docker testing. Don't split them into separate + Deployments/Services without also rethinking `INFERENCE_SIDECAR_ADDR` and the networking model. +- **The k8s manifests live in a different repo**: `~/repo/homelab/manifests/home-services/` — not + in this repo at all. `CLAUDE.md` now has an "Infrastructure / Deployment" section documenting + this; check there (and that repo directly) before assuming something isn't deployed just because + this repo has no manifests for it. +- **`open_jtalk`'s tokenization has an intentional bug that must be preserved.** The Go text + normalizer (`tts-gateway/internal/adapters/secondary/jtalk`) maps the cleaned phoneme string to + symbol IDs character-by-character, not phoneme-by-phoneme, because that's what the checkpoint + was actually trained on (`text/symbols.py`'s "wrong tokens" comment in the original reference + implementation). Don't "fix" this without retraining the model — see the comments in `jtalk.go` + and `tts-gateway/README.md`. +- **ONNX export doesn't work for this checkpoint, and it's not a quick fix.** Read + `tmp/reference/uma-tts-api/spike/FINDINGS.md` before re-attempting it — the model's + data-dependent output length (predicted phoneme durations determine audio length at runtime) + defeats `torch.export`'s guard system in a way that would need an unknown number of per-layer + `torch._check` hints to resolve, not a single targeted change. +- **CI's build cache needs one "priming" run per image.** If a build job still looks slow after + the caching change in `ci.yaml`, check whether a `:buildcache` tag actually exists yet in the + registry for that specific image — the first run after the cache was added has nothing to pull + from and builds fully fresh. + +## Where to look for more detail + +- `TTS_GATEWAY_PLAN.md` (repo root) — original plan, feasibility analysis, phase breakdown, open + product questions (codec, concurrency, auth). +- `tmp/reference/uma-tts-api/spike/FINDINGS.md` — Phase 0 spike results with full reasoning. +- `tts-gateway/README.md` — service-level docs: gRPC API, config reference, how to run/test + locally and on nik-gpu. +- `~/repo/homelab/manifests/home-services/tts-gateway.yaml` — actual production config (separate + repo, maintained independently — check it directly rather than trusting a stale summary). diff --git a/tts-gateway/README.md b/tts-gateway/README.md index 394ad75..63f0609 100644 --- a/tts-gateway/README.md +++ b/tts-gateway/README.md @@ -67,8 +67,33 @@ The server also registers gRPC health checks and reflection. | `PORT` | `50054` | HTTP listen port | Neither the checkpoint nor the hparams file is committed to git - both must -be supplied at runtime (see below). Model artifact distribution (bind mount -vs. build-time download vs. registry) is still an open decision. +be supplied at runtime (see below). In production, they're baked into a +small versioned image (`tts-gateway/model/Dockerfile`) that a k8s +`initContainer` copies into a shared volume before `tts-sidecar` starts - +see "Model artifact distribution (production)" below. For local/manual +testing on nik-gpu, a plain bind mount (as shown below) is simpler. + +### Model artifact distribution (production) + +CI can never build the model image - the checkpoint (~455MB) and hparams +aren't committed to git, and nik-gpu's local disk at `/data/tts-gateway` is +the only place they exist. Build and push it manually, directly on nik-gpu +(not via `docker --context nik-gpu` from elsewhere - the build context is +gathered client-side, so it would look for `/data/tts-gateway` on the wrong +machine), whenever the checkpoint or config change: + +```bash +ssh nik-gpu +docker build -f ~/repo/home-service/tts-gateway/model/Dockerfile \ + -t gitea.nik4nao.com/nik/tts-model:g790000-v1 /data/tts-gateway +docker push gitea.nik4nao.com/nik/tts-model:g790000-v1 +``` + +Bump the tag (`v2`, `v3`, ...) rather than overwriting one - non-`:latest` +tags default to `imagePullPolicy: IfNotPresent`, so a node that already +pulled a tag won't re-pull an overwritten one. The k8s Deployment +(`~/repo/homelab/manifests/home-services/tts-gateway.yaml`) references the +tag explicitly in its `model-init` init container. ## Running And Testing On nik-gpu @@ -188,8 +213,8 @@ sidecar/ # Python/libtorch inference servic - Concurrency is unbounded on the Go side, but the sidecar's model forward pass is inherently single-threaded per GPU; heavy concurrent load will just queue at the sidecar. -- Model artifact distribution isn't finalized - the sidecar currently - expects a bind-mounted checkpoint/config, not something the image ships - with. -- No Discord-bot integration yet (a `/speak`-style command calling this - service is a deliberate follow-up, not yet built). +- Model artifact distribution to production is a manually-built-and-pushed + versioned image (see above), not something CI can automate - the + checkpoint only ever existed on nik-gpu's local disk. +- `discord-bot`'s `/speak` command calls this service and plays the result + in a Discord voice channel (`discord-bot/internal/adapters/primary/discord/voice.go`). diff --git a/tts-gateway/model/Dockerfile b/tts-gateway/model/Dockerfile new file mode 100644 index 0000000..7167621 --- /dev/null +++ b/tts-gateway/model/Dockerfile @@ -0,0 +1,7 @@ +# One-time (or only-when-the-checkpoint-changes) manually built and pushed by a human on +# nik-gpu - CI can never build this: the checkpoint and hparams aren't committed to git +# (~455MB, and nik-gpu's local disk at /data/tts-gateway is the only place they exist). +# Build CONTEXT must be that directory, not a checkout of this repo - see +# tts-gateway/README.md for the exact commands. +FROM busybox:1.36 +COPY G_790000.pth uma.json /models/