# TTS | A Deep Learning Toolkit For Text-to-Speech

**URL:** <https://onehack.st/t/tts-a-deep-learning-toolkit-for-text-to-speech/289463>\
**Category:** Tools & Scripts\
**Tags:** tools, programming, freebies, audio, ai\
**Created:** [August 20, 2024, 8:41pm UTC](https://onehack.st/t/tts-a-deep-learning-toolkit-for-text-to-speech/289463 "2024-08-20T20:41:20Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sarah](https://onehack.st/user_avatar/onehack.st/sarah/32/166974_2.png) [@Sarah](https://onehack.st/u/Sarah)\
**Post date:** [August 20, 2024, 8:41pm UTC](https://onehack.st/t/tts-a-deep-learning-toolkit-for-text-to-speech/289463/1 "2024-08-20T20:41:21Z")

</div>

## 🐸Coqui.ai News

- 📣 ⓍTTSv2 is here with 16 languages and better performance across the board.
- 📣 ⓍTTS fine-tuning code is out. Check the [example recipes](https://github.com/coqui-ai/TTS/tree/dev/recipes/ljspeech).
- 📣 ⓍTTS can now stream with \<200ms latency.
- 📣 ⓍTTS, our production TTS model that can speak 13 languages, is released [Blog Post](https://coqui.ai/blog/tts/open_xtts), [Demo](https://huggingface.co/spaces/coqui/xtts), [Docs](https://tts.readthedocs.io/en/dev/models/xtts.html)
- 📣 [🐶Bark](https://github.com/suno-ai/bark) is now available for inference with unconstrained voice cloning. [Docs](https://tts.readthedocs.io/en/dev/models/bark.html)
- 📣 You can use [~1100 Fairseq models](https://github.com/facebookresearch/fairseq/tree/main/examples/mms) with 🐸TTS.
- 📣 🐸TTS now supports 🐢Tortoise with faster inference. [Docs](https://tts.readthedocs.io/en/dev/models/tortoise.html)

[![](https://camo.githubusercontent.com/008cc60554e5ddd7f235b1a11d6872c37a91216fd012894b33bc8f7af4200132/68747470733a2f2f7374617469632e73636172662e73682f612e706e673f782d707869643d63663331376665372d323138382d343732312d626330312d313234626235643564626232)](https://camo.githubusercontent.com/008cc60554e5ddd7f235b1a11d6872c37a91216fd012894b33bc8f7af4200132/68747470733a2f2f7374617469632e73636172662e73682f612e706e673f782d707869643d63663331376665372d323138382d343732312d626330312d313234626235643564626232)

## [![](https://raw.githubusercontent.com/coqui-ai/TTS/main/images/coqui-log-green-TTS.png)](https://raw.githubusercontent.com/coqui-ai/TTS/main/images/coqui-log-green-TTS.png)

**🐸TTS is a library for advanced Text-to-Speech generation.**

🚀 Pretrained models in +1100 languages.

🛠 Tools for training new models and fine-tuning existing models in any language.

📚 Utilities for dataset analysis and curation.

* * *

[![Discord](https://camo.githubusercontent.com/d1f1a5d178fa3719aa4ce55c393ecd0564d7252eb615a194630896459484dfa7/68747470733a2f2f696d672e736869656c64732e696f2f646973636f72642f313033373332363635383830373533333632383f636f6c6f723d253233394235394236266c6162656c3d636861742532306f6e253230646973636f7264)](https://discord.gg/5eXr5seRrv) [![License](https://camo.githubusercontent.com/8e45ec257613c2b03f48c9f04c310fc48c74136ad81bc238905b4bc2e60c4248/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e73652d4d504c253230322e302d627269676874677265656e2e737667)](https://opensource.org/licenses/MPL-2.0) [![PyPI version](https://camo.githubusercontent.com/62fc752641761ad097b758a36c06fc7735df420fad7f2fd199bfa164f3b7f774/68747470733a2f2f62616467652e667572792e696f2f70792f5454532e737667)](https://badge.fury.io/py/TTS) [![Covenant](https://camo.githubusercontent.com/7d620efaa3eac1c5b060ece5d6aacfcc8b81a74a04d05cd0398689c01c4463bb/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f436f6e7472696275746f72253230436f76656e616e742d76322e3025323061646f707465642d6666363962342e737667)](https://github.com/coqui-ai/TTS/blob/master/CODE_OF_CONDUCT.md) [![Downloads](https://camo.githubusercontent.com/7264dacaa214649805b2a50d41b2ef3e894de669e7b9c7b1d554f826e3255f5f/68747470733a2f2f706570792e746563682f62616467652f747473)](https://pepy.tech/project/tts) [![DOI](https://camo.githubusercontent.com/9cdf739fbbd2ad6667a03e8f756f1ad7740e37fa571b11da3247a0dd4fd2d5d1/68747470733a2f2f7a656e6f646f2e6f72672f62616467652f3236353631323434302e737667)](https://zenodo.org/badge/latestdoi/265612440)

[![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/aux_tests.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/aux_tests.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/data_tests.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/data_tests.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/docker.yaml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/docker.yaml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/inference_tests.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/inference_tests.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/style_check.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/style_check.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/text_tests.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/text_tests.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/tts_tests.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/tts_tests.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/vocoder_tests.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/vocoder_tests.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/zoo_tests0.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/zoo_tests0.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/zoo_tests1.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/zoo_tests1.yml/badge.svg) [![GithubActions](https://github.com/coqui-ai/TTS/actions/workflows/zoo_tests2.yml/badge.svg)](https://github.com/coqui-ai/TTS/actions/workflows/zoo_tests2.yml/badge.svg) [![Docs](https://camo.githubusercontent.com/4e52ce5fd9d1961dfc90ca265c3f78dd42064cb4a9728863f2de351cd091aa57/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f7474732f62616467652f3f76657273696f6e3d6c6174657374267374796c653d706c6173746963)](https://tts.readthedocs.io/en/latest/)

* * *

## 💬 Where to ask questions

Please use our dedicated channels for questions and discussion. Help is much more valuable if it’s shared publicly so that more people can benefit from it.

| Type | Platforms |
| --- | --- |
| 🚨 **Bug Reports** | [GitHub Issue Tracker](https://github.com/coqui-ai/tts/issues) |
| 🎁 **Feature Requests & Ideas** | [GitHub Issue Tracker](https://github.com/coqui-ai/tts/issues) |
| 👩‍💻 **Usage Questions** | [GitHub Discussions](https://github.com/coqui-ai/TTS/discussions) |
| 🗯 **General Discussion** | [GitHub Discussions](https://github.com/coqui-ai/TTS/discussions) or [Discord](https://discord.gg/5eXr5seRrv) |

## 🔗 Links and Resources

| Type | Links |
| --- | --- |
| 💼 **Documentation** | [ReadTheDocs](https://tts.readthedocs.io/en/latest/) |
| 💾 **Installation** | [TTS/README.md](https://github.com/coqui-ai/TTS/tree/dev#installation) |
| 👩‍💻 **Contributing** | [CONTRIBUTING.md](https://github.com/coqui-ai/TTS/blob/main/CONTRIBUTING.md) |
| 📌 **Road Map** | [Main Development Plans](https://github.com/coqui-ai/TTS/issues/378) |
| 🚀 **Released Models** | [TTS Releases](https://github.com/coqui-ai/TTS/releases) and [Experimental Models](https://github.com/coqui-ai/TTS/wiki/Experimental-Released-Models) |
| 📰 **Papers** | [TTS Papers](https://github.com/erogol/TTS-papers) |

## 🥇 TTS Performance

[![](https://raw.githubusercontent.com/coqui-ai/TTS/main/images/TTS-performance.png)](https://raw.githubusercontent.com/coqui-ai/TTS/main/images/TTS-performance.png)

Underlined “TTS\*” and “Judy\*” are **internal** 🐸TTS models that are not released open-source. They are here to show the potential. Models prefixed with a dot (.Jofish .Abe and .Janice) are real human voices.

## Features

- High-performance Deep Learning models for Text2Speech tasks.
  - Text2Spec models (Tacotron, Tacotron2, Glow-TTS, SpeedySpeech).
  - Speaker Encoder to compute speaker embeddings efficiently.
  - Vocoder models (MelGAN, Multiband-MelGAN, GAN-TTS, ParallelWaveGAN, WaveGrad, WaveRNN)

- Fast and efficient model training.
- Detailed training logs on the terminal and Tensorboard.
- Support for Multi-speaker TTS.
- Efficient, flexible, lightweight but feature complete `Trainer API`.
- Released and ready-to-use models.
- Tools to curate Text2Speech datasets under`dataset_analysis`.
- Utilities to use and test your models.
- Modular (but not too much) code base enabling easy implementation of new ideas.

## Model Implementations

### Spectrogram models

- Tacotron: [paper](https://arxiv.org/abs/1703.10135)
- Tacotron2: [paper](https://arxiv.org/abs/1712.05884)
- Glow-TTS: [paper](https://arxiv.org/abs/2005.11129)
- Speedy-Speech: [paper](https://arxiv.org/abs/2008.03802)
- Align-TTS: [paper](https://arxiv.org/abs/2003.01950)
- FastPitch: [paper](https://arxiv.org/pdf/2006.06873.pdf)
- FastSpeech: [paper](https://arxiv.org/abs/1905.09263)
- FastSpeech2: [paper](https://arxiv.org/abs/2006.04558)
- SC-GlowTTS: [paper](https://arxiv.org/abs/2104.05557)
- Capacitron: [paper](https://arxiv.org/abs/1906.03402)
- OverFlow: [paper](https://arxiv.org/abs/2211.06892)
- Neural HMM TTS: [paper](https://arxiv.org/abs/2108.13320)
- Delightful TTS: [paper](https://arxiv.org/abs/2110.12612)

### End-to-End Models

- ⓍTTS: [blog](https://coqui.ai/blog/tts/open_xtts)
- VITS: [paper](https://arxiv.org/pdf/2106.06103)
- 🐸 YourTTS: [paper](https://arxiv.org/abs/2112.02418)
- 🐢 Tortoise: [orig. repo](https://github.com/neonbjb/tortoise-tts)
- 🐶 Bark: [orig. repo](https://github.com/suno-ai/bark)

### Attention Methods

- Guided Attention: [paper](https://arxiv.org/abs/1710.08969)
- Forward Backward Decoding: [paper](https://arxiv.org/abs/1907.09006)
- Graves Attention: [paper](https://arxiv.org/abs/1910.10288)
- Double Decoder Consistency: [blog](https://erogol.com/solving-attention-problems-of-tts-models-with-double-decoder-consistency/)
- Dynamic Convolutional Attention: [paper](https://arxiv.org/pdf/1910.10288.pdf)
- Alignment Network: [paper](https://arxiv.org/abs/2108.10447)

### Speaker Encoder

- GE2E: [paper](https://arxiv.org/abs/1710.10467)
- Angular Loss: [paper](https://arxiv.org/pdf/2003.11982.pdf)

### Vocoders

- MelGAN: [paper](https://arxiv.org/abs/1910.06711)
- MultiBandMelGAN: [paper](https://arxiv.org/abs/2005.05106)
- ParallelWaveGAN: [paper](https://arxiv.org/abs/1910.11480)
- GAN-TTS discriminators: [paper](https://arxiv.org/abs/1909.11646)
- WaveRNN: [origin](https://github.com/fatchord/WaveRNN/)
- WaveGrad: [paper](https://arxiv.org/abs/2009.00713)
- HiFiGAN: [paper](https://arxiv.org/abs/2010.05646)
- UnivNet: [paper](https://arxiv.org/abs/2106.07889)

### Voice Conversion

- FreeVC: [paper](https://arxiv.org/abs/2210.15418)

You can also help us implement more models.

## Installation

🐸TTS is tested on Ubuntu 18.04 with **python \>= 3.9, \< 3.12.**.

If you are only interested in [synthesizing speech](https://tts.readthedocs.io/en/latest/inference.html) with the released 🐸TTS models, installing from PyPI is the easiest option.

`pip install TTS`

If you plan to code or train models, clone 🐸TTS and install it locally.

`git clone https://github.com/coqui-ai/TTS pip install -e .[all,dev,notebooks] # Select the relevant extras`

If you are on Ubuntu (Debian), you can also run following commands for installation.

```auto
$ make system-deps # intended to be used on Ubuntu (Debian). Let us know if you have a different OS.
$ make install

```

If you are on Windows, 👑@GuyPaddock wrote installation instructions [here](https://stackoverflow.com/questions/66726331/how-can-i-run-mozilla-tts-coqui-tts-training-with-cuda-on-a-windows-system).

## Docker Image

You can also try TTS without install with the docker image. Simply run the following command and you will be able to run TTS without installing it.

```auto
docker run --rm -it -p 5002:5002 --entrypoint /bin/bash ghcr.io/coqui-ai/tts-cpu
python3 TTS/server/server.py --list_models #To get the list of available models
python3 TTS/server/server.py --model_name tts_models/en/vctk/vits # To start a server

```

You can then enjoy the TTS server here More details about the docker images (like GPU support) can be found [here](https://tts.readthedocs.io/en/latest/docker_images.html)

## Synthesizing speech by 🐸TTS

### 🐍 Python API

#### Running a multi-speaker and multi-lingual model

```auto
import torch
from TTS.api import TTS

# Get device
device = "cuda" if torch.cuda.is_available() else "cpu"

# List available 🐸TTS models
print(TTS().list_models())

# Init TTS
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)

# Run TTS
# ❗ Since this model is multi-lingual voice cloning model, we must set the target speaker_wav and language
# Text to speech list of amplitude values as output
wav = tts.tts(text="Hello world!", speaker_wav="my/cloning/audio.wav", language="en")
# Text to speech to a file
tts.tts_to_file(text="Hello world!", speaker_wav="my/cloning/audio.wav", language="en", file_path="output.wav")

```

#### Running a single speaker model

```auto
# Init TTS with the target model name
tts = TTS(model_name="tts_models/de/thorsten/tacotron2-DDC", progress_bar=False).to(device)

# Run TTS
tts.tts_to_file(text="Ich bin eine Testnachricht.", file_path=OUTPUT_PATH)

# Example voice cloning with YourTTS in English, French and Portuguese
tts = TTS(model_name="tts_models/multilingual/multi-dataset/your_tts", progress_bar=False).to(device)
tts.tts_to_file("This is voice cloning.", speaker_wav="my/cloning/audio.wav", language="en", file_path="output.wav")
tts.tts_to_file("C'est le clonage de la voix.", speaker_wav="my/cloning/audio.wav", language="fr-fr", file_path="output.wav")
tts.tts_to_file("Isso é clonagem de voz.", speaker_wav="my/cloning/audio.wav", language="pt-br", file_path="output.wav")

```

#### Example voice conversion

Converting the voice in `source_wav` to the voice of `target_wav`

`tts = TTS(model_name="voice_conversion_models/multilingual/vctk/freevc24", progress_bar=False).to("cuda") tts.voice_conversion_to_file(source_wav="my/source.wav", target_wav="my/target.wav", file_path="output.wav")`

#### Example voice cloning together with the voice conversion model.

This way, you can clone voices by using any model in 🐸TTS.

`tts = TTS("tts_models/de/thorsten/tacotron2-DDC") tts.tts_with_vc_to_file( "Wie sage ich auf Italienisch, dass ich dich liebe?", speaker_wav="target/speaker.wav", file_path="output.wav" )`

#### Example text to speech using **Fairseq models in ~1100 languages** 🤯.

For Fairseq models, use the following name format: `tts_models/<lang-iso_code>/fairseq/vits`. You can find the language ISO codes [here](https://dl.fbaipublicfiles.com/mms/tts/all-tts-languages.html) and learn about the Fairseq models [here](https://github.com/facebookresearch/fairseq/tree/main/examples/mms).

`# TTS with on the fly voice conversion api = TTS("tts_models/deu/fairseq/vits") api.tts_with_vc_to_file( "Wie sage ich auf Italienisch, dass ich dich liebe?", speaker_wav="target/speaker.wav", file_path="output.wav" )`

### Command-line `tts`

Synthesize speech on command line.

You can either use your trained model or choose a model from the provided list.

If you don’t specify any models, then it uses LJSpeech based English model.

#### Single Speaker Models

- List provided models:

```auto
$ tts --list_models

```

- Get model info (for both tts\_models and vocoder\_models):
  - Query by type/name: The model\_info\_by\_name uses the name as it from the --list\_models.

```auto
$ tts --model_info_by_name "<model_type>/<language>/<dataset>/<model_name>"

```

For example:

```auto
$ tts --model_info_by_name tts_models/tr/common-voice/glow-tts
$ tts --model_info_by_name vocoder_models/en/ljspeech/hifigan_v2

```

- Query by type/idx: The model\_query\_idx uses the corresponding idx from --list\_models.

```auto
$ tts --model_info_by_idx "<model_type>/<model_query_idx>"

```

For example:

```auto
$ tts --model_info_by_idx tts_models/3

```

- Query info for model info by full name:

```auto
$ tts --model_info_by_name "<model_type>/<language>/<dataset>/<model_name>"

```

- Run TTS with default models:

```auto
$ tts --text "Text for TTS" --out_path output/path/speech.wav

```

- Run TTS and pipe out the generated TTS wav file data:

```auto
$ tts --text "Text for TTS" --pipe_out --out_path output/path/speech.wav | aplay

```

- Run a TTS model with its default vocoder model:

```auto
$ tts --text "Text for TTS" --model_name "<model_type>/<language>/<dataset>/<model_name>" --out_path output/path/speech.wav

```

For example:

```auto
$ tts --text "Text for TTS" --model_name "tts_models/en/ljspeech/glow-tts" --out_path output/path/speech.wav

```

- Run with specific TTS and vocoder models from the list:

```auto
$ tts --text "Text for TTS" --model_name "<model_type>/<language>/<dataset>/<model_name>" --vocoder_name "<model_type>/<language>/<dataset>/<model_name>" --out_path output/path/speech.wav

```

For example:

```auto
$ tts --text "Text for TTS" --model_name "tts_models/en/ljspeech/glow-tts" --vocoder_name "vocoder_models/en/ljspeech/univnet" --out_path output/path/speech.wav

```

- Run your own TTS model (Using Griffin-Lim Vocoder):

```auto
$ tts --text "Text for TTS" --model_path path/to/model.pth --config_path path/to/config.json --out_path output/path/speech.wav

```

- Run your own TTS and Vocoder models:

```auto
$ tts --text "Text for TTS" --model_path path/to/model.pth --config_path path/to/config.json --out_path output/path/speech.wav
    --vocoder_path path/to/vocoder.pth --vocoder_config_path path/to/vocoder_config.json

```

#### Multi-speaker Models

- List the available speakers and choose a \<speaker\_id\> among them:

```auto
$ tts --model_name "<language>/<dataset>/<model_name>" --list_speaker_idxs

```

- Run the multi-speaker TTS model with the target speaker ID:

```auto
$ tts --text "Text for TTS." --out_path output/path/speech.wav --model_name "<language>/<dataset>/<model_name>" --speaker_idx <speaker_id>

```

- Run your own multi-speaker TTS model:

```auto
$ tts --text "Text for TTS" --out_path output/path/speech.wav --model_path path/to/model.pth --config_path path/to/config.json --speakers_file_path path/to/speaker.json --speaker_idx <speaker_id>

```

### Voice Conversion Models

```auto
$ tts --out_path output/path/speech.wav --model_name "<language>/<dataset>/<model_name>" --source_wav <path/to/speaker/wav> --target_wav <path/to/reference/wav>

```

## Directory Structure

```auto
|- notebooks/ (Jupyter Notebooks for model evaluation, parameter selection and data analysis.)
|- utils/ (common utilities.)
|- TTS
    |- bin/ (folder for all the executables.)
      |- train*.py (train your target model.)
      |- ...
    |- tts/ (text to speech models)
        |- layers/ (model layer definitions)
        |- models/ (model definitions)
        |- utils/ (model specific utilities.)
    |- speaker_encoder/ (Speaker Encoder models.)
        |- (same)
    |- vocoder/ (Vocoder models.)
        |- (same)

```

## GitHub:

[https://github.com/coqui-ai/TTS](https://github.com/coqui-ai/TTS)