[feature request] enhance consistency of TTS

[FNsi]

Currently if there's a long text input, qwen3 tts will simply process it all, with empty output or noise after 2:30 minutes; but if we simply split the text to multiple short pieces, the voice consistency will be broken;

I think add function to auto split text inside ctx, and use the last sentence or few words as voice clone, starting the next piece of text, thus the voice consistency be maintained, the computation speed up, then we can easily create long audios.

[LostRuins]

I have added this feature in the rolling release, please try https://github.com/LostRuins/koboldcpp/releases/tag/rolling

[FNsi]

I have added this feature in the rolling release, please try https://github.com/LostRuins/koboldcpp/releases/tag/rolling

Nice! Didn't know there's a rolling version yet 😂😂

[LostRuins]

rolling version is my alpha channel, it's auto uploaded and new changes will appear there first. Consider it the unstable version.

[FNsi]

rolling version is my alpha channel, it's auto uploaded and new changes will appear there first. Consider it the unstable version.

Thanks, I will try it soon, also another question, did it already fix vulkan core dump issue in Qwen tts?

[FNsi]

😂i guess I.. need another issue

[Request] parsed json (28 fields)
[Pipeline] T=3426, S=1713
[Pipeline] seed=916613, steps=8, guidance=1.0, shift=3.0, duration=137.0s
[Pipeline] caption: 74 tokens, lyrics: 202 tokens
[Encode] TextEncoder (74 tokens): 134.9 ms
[Encode] Lyric vocab lookup (202 tokens): 0.7 ms
[CondEnc] Lyric sliding mask: 202x202, window=128
[CondEnc] Timbre sliding mask: 750x750, window=128
[Encode] Packed: lyric=202 + timbre=1 + text=74 = 277 tokens
[Encode] ConditionEncoder: 326.8 ms, enc_S=277
[Context Batch0] noise seed=916613
[DiT] Starting: T=3426, S=1713, enc_S=277, steps=8, batch=1
[DiT] Batch N=1, T=3426, S=1713, enc_S=277
[DiT] Graph: 2129 nodes
[DiT] step 1/8 t=1.000
[DiT] step 2/8 t=0.955
[DiT] step 3/8 t=0.900
[DiT] step 4/8 t=0.833
[DiT] step 5/8 t=0.750
[DiT] step 6/8 t=0.643
[DiT] step 7/8 t=0.500
[DiT] step 8/8 t=0.300
[DiT] Total generation: 23371.2 ms (23371.2 ms/sample)
[VAE] Tiled decode: 27 tiles (chunk=256, overlap=64, stride=128)
[VAE] Graph: 474 nodes, T_latent=192
[VAE] Upsample factor: 1920.00 (expected ~1920)
[VAE] Graph: 474 nodes, T_latent=256
radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery.
terminate called after throwing an instance of 'vk::DeviceLostError'
  what():  vk::Queue::submit: ErrorDeviceLost
Aborted (core dumped)

Vulkan is breaking in music vae😂 fixed by move vae to cpu; tts fixed by downsize the chunk from 1024 to 256

[rtcvb32]

I wonder if instructions can be given in the middle and split the TTS generation at those points so it can be more robust.

I had several examples of voice outputs given to see how TTS sounds, only for it to take the first one, and ignore all others. (was using the Q3TTS Voice Design tts in the 1.111.2 release)

[happy male] "Hey there!" (listened to and goes to 'hey there')
[sad female] "Oh hi..." (ignored, says 'sad female' in the original tone).

[FNsi]

[happy male] "Hey there!" (listened to and goes to 'hey there')
[sad female] "Oh hi..." (ignored, says 'sad female' in the original tone).

i think that need to apply grammar check, also something to store different voice in that case...

[LostRuins]

No, each TTS sample can only handle 1 instruction. If you add more, only the first 1 will be used. If you need 2 different voices, you'll have to send 2 requests.

[FNsi]

I have added this feature in the rolling.

I try with ttsmaxtoken 4096 with a long prompt in with an initial reference audio, now it okay to generate long audio but the consistency is broken still...voice changes, end with long empty noise part after all text processed.

[rtcvb32]

No, each TTS sample can only handle 1 instruction. If you add more, only the first 1 will be used. If you need 2 different voices, you'll have to send 2 requests.

Were i to program this to do otherwise, i'd probably use a regex to find the bracket/instructions and split it to separate requests which can likely be run in parallel, then appended for a final result. Alternatively have a running 'last/default instruction' and replace it at each newline and process each line/sentence separately.