latency - how to get sub 200ms ultra low latency as mentioned
#14
by saketfractal - opened
Firstly, thanks for open-sourcing. I loaded the model on single A40. Thereafter, time taken to run model.generate() was around 9.5 secs for a 12 sec output audio file. The description mentions it can achieve sub 200ms latency. is that for the priced model only? or the stats based on streaming based generation?
The sub 200ms latency is for their paid service
you need to run it in streaming mode, 200ms is the time to first token(TTFT) on more powerful CPU
you need to run it in streaming mode, 200ms is the time to first token(TTFT) on more powerful CPU
how to implement this specially chatterbox doesn't have 'generate_stream()' function