🎙️ Build a Free Voice-Cloning Monster — One Offline Pipeline, 27 Repos

Tutorials & Methods
, ,
1

:test_tube: The Voice-Cloning Pipeline Pros Don’t Explain — yt-dlp → UVR → DeepFilterNet → Voicebox → seed-vc, All Free, All Local

Voicebox clones any voice for free. The combos below turn it into a studio that out-muscles the paid cloud apps — all on your machine.

The hub clones voices, types what you say, gives your AI a mouth. Bolt on the right free repos → clean-clip prep → near-perfect cloning → timbre re-skin → full video dubbing. Zero subscriptions, nothing leaves your PC. Tap a chapter :backhand_index_pointing_down:

🎤 New here? — what Voicebox even is (3 free tricks)

Free, open-source, 28k+ GitHub stars. Replaces two paid apps (ElevenLabs + WisprFlow) on its own. Three tricks:

  • Clones voices → feed it 3 seconds of anyone talking, it reads anything you type in their voice. No clip? 50+ presets built in.
  • You talk, it types → hold a key, speak, let go → words drop into whatever’s open. Fancy word: dictation.
  • Your AI talks back → coding helpers (Claude Code, Cursor) speak answers out loud in a voice you cloned. One command: voicebox.speak.

:light_bulb: TTS = text-to-speech, the robot that reads text aloud. Local = everything stays on YOUR machine, nothing uploaded. That’s the whole point.

The 7 engines — pick your fighter:

Engine Langs Best for
Qwen3-TTS 10 Top cloning + “speak slowly / whisper” control
Qwen CustomVoice 10 9 built-in voices, no clip needed
LuxTTS EN Tiny + fast (150x realtime on CPU)
Chatterbox Multilingual 23 The polyglot — Arabic, Hindi, Polish + more
Chatterbox Turbo EN Does laughs, sighs, gasps
HumeAI TADA 10 Mood-aware, voices that feel
Kokoro 8 82M featherweight, runs on a potato

:light_bulb: Old PC? Kokoro / LuxTTS. Custom clone? Qwen3-TTS. Emotion? Chatterbox Turbo.

🎬 LEVEL 0 — clone a voice in 3 seconds (no code)
  1. Sample → drag an audio file, record your mic, or grab audio playing onscreen
  2. 3 seconds is enough → the clone learns fast
  3. Type, hit generate → it reads back in that voice

:light_bulb: The #1 secret: a clone is only as good as the clip. Dry, clean, one speaker, no music = scary-real. Noisy garbage = robot. Which is exactly why Level 1 exists.

🧪 LEVEL 1 — feed it surgically-clean clips (the prep chain)

Clone from a noisy rip → robot in a tin can. Run the source through a cleanup pipeline first: rip the voice out of the music, scrub the noise, slice a perfect 3–10 second sample. All free, all offline.

Tool Link What it does
yt-dlp github.com/yt-dlp/yt-dlp Rips audio from 1000+ sites — the front door of every clone
Ultimate Vocal Remover github.com/Anjok07/ultimatevocalremovergui Pulls the voice out of music/noise (click-and-go GUI)
Demucs github.com/facebookresearch/demucs Same job, command-line one-liner
DeepFilterNet github.com/Rikorose/DeepFilterNet Nukes background hiss + room echo, no Python
resemble-enhance github.com/resemble-ai/resemble-enhance Rescues trashy phone/stream audio
audio-slicer github.com/openvpi/audio-slicer Auto-chops a long clip into clean pieces

Grab the audio, rip the vocal out, scrub the noise — three commands:

yt-dlp -f bestaudio -x --audio-format wav "URL"
demucs --two-stems=vocals input.wav
deep-filter vocals.wav

:light_bulb: Trick: aim for a dry clip — no reverb, no music bed. Denoise before you slice. A crisp 5-second sample beats a noisy 2-minute one every time.

🎭 LEVEL 2 — re-skin the timbre (the secret weapon)

The move nobody tells beginners: let Voicebox generate the words, then run the output through a voice-conversion tool to nail the exact timbre. TTS gets the script + a rough voice; these (voice conversion = repaint one voice as another) lock the character tight.

Tool Link What it does
seed-vc github.com/Plachtaa/seed-vc Zero-shot re-skin from a 1–30s clip, no training. Start here.
Applio github.com/IAHispano/Applio Friendliest RVC fork — installer, batch convert, downloader
RVC github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI The OG — trains a tight voice model from ~10 min audio
so-vits-svc (fork) github.com/voicepaw/so-vits-svc-fork Makes a cloned voice sing — TTS can’t, this can
w-okada voice-changer github.com/w-okada/voice-changer Real-time changer for streaming/gaming
GPT-SoVITS github.com/RVC-Boss/GPT-SoVITS Alt cloning engine to A/B — its prep UI also preps clips

:light_bulb: Trick: no audio to train on? seed-vc converts instantly off one clip. Got data + want perfect? Train Applio. Want it to sing? so-vits-svc.

🌍 LEVEL 3 — auto-dub any video into YOUR voice

These already do the full chain — download → transcribe → translate → speak. Point their voice step at Voicebox = dub a foreign video into your own cloned voice, fully offline.

Tool Link What it does
VideoLingo github.com/Huanshere/VideoLingo One-box dubbing + Netflix-style subtitles
pyvideotrans github.com/jianchang512/pyvideotrans Multi-speaker dub, auto-splits who-said-what
SoniTranslate github.com/R3gm/SoniTranslate Synced multi-voice dub, wired for voice conversion
Auto-Synced-Translated-Dubs github.com/ThioJoe/Auto-Synced-Translated-Dubs Times each dubbed line to the original subtitles

:light_bulb: Trick: they run on Whisper (the ear that turns speech into text) — the same ear Voicebox uses — so the pieces snap together.

🤖 LEVEL 4 — wire it into your AI + automate

Voicebox listens on a local address and speaks MCP, so script it or plug it into agents.

Hook it into Claude Code so your AI talks back — one line:

claude mcp add voicebox --transport http --url http://127.0.0.1:17493/mcp --header "X-Voicebox-Client-Id: claude-code"

Make it spit out a voice file from any script — one command:

curl -X POST http://127.0.0.1:17493/generate \
  -H "Content-Type: application/json" \
  -d '{"text":"hello world","profile_id":"YOUR_ID","engine":"qwen_custom_voice","instruct":"warm, slow, cinematic"}' \
  --output line.wav
Tool Link What it does
n8n github.com/n8n-io/n8n Drag-drop automation — “new email → summarize → speak it”
Ollama github.com/ollama/ollama Local AI brain that feeds text into Voicebox, offline
llama.cpp github.com/ggml-org/llama.cpp Same, squeezed onto weak hardware
awesome-mcp-servers github.com/appcypher/awesome-mcp-servers Giant list of tools to plug into the same agent

:light_bulb: Trick: the instruct field ("warm, slow, cinematic") is your no-code director — steers tone + pace, no markup to learn.

😂 Emotion tags + studio effects (built in + free add-ons)

Type cues straight into text — only Chatterbox Turbo (other engines read them out loud literally): [laugh] [sigh] [gasp] [cough] [whisper] [groan]

8 built-in effects (Spotify’s own audio kit): pitch shift, reverb, delay, chorus/flanger, compressor, high/low-pass filter. Save mixes as presets.

Free post-tool Link What it does
pedalboard github.com/spotify/pedalboard Script your own effect chains / load plugins
Audacity github.com/audacity/audacity Full free editor — fades, trims, assembly
ffmpeg-normalize github.com/slhck/ffmpeg-normalize Levels a whole folder to streaming-loud

Level every clip so nothing’s too quiet or blown out:

ffmpeg -i in.wav -af loudnorm=I=-16:TP=-1.5:LRA=11 out.wav
⚙️ Faster transcription + run it on a potato

The dictation side runs on Whisper. These make it faster or lighter:

Tool Link What it does
faster-whisper github.com/SYSTRAN/faster-whisper Up to 4x faster transcription, less memory
whisper.cpp github.com/ggerganov/whisper.cpp CPU-friendly, no Python — for old machines
WhisperX github.com/m-bain/whisperX Word-level timing + splits speakers apart

:light_bulb: Trick: WhisperX figures out who spoke when in a group recording — pull one person’s lines and make a Voicebox profile per speaker.

⬇️ Download + the FULL crazy chain (start to finish)
Machine How
Mac (M1/M2/M3) DMG → drag to Applications
Mac (Intel) Intel DMG
Windows MSI installer
Docker docker compose up
Linux Build from source (site)

:link: voicebox.sh · :link: github.com/jamiepine/voicebox

The whole stacked pipeline:

yt-dlp (grab) → UVR / Demucs (rip voice out) → DeepFilterNet (scrub) → audio-slicer (clean clip) → Voicebox (clone + words) → seed-vc / Applio (lock timbre) → so-vits-svc (sing, optional) → ffmpeg loudnorm (publish-loud) → mux onto video.

:light_bulb: First launch downloads the engines (chunky files, one-time wait). Built in Rust = light, not a fan-frying memory hog.

⚠️ Watch-outs (so you don't ragequit)
  • Clone what you’ve got rights to — your voice, the presets, clips you may use. RVC/so-vits-svc/w-okada ban impersonation in their terms. Keep it clean.
  • RVC OG repo is frozen — use Applio. so-vits-svc upstream is archived — use the fork linked above.
  • Whisper “thanks for watching” loops hit on quiet audio — turn on VAD (voice-detection filter) in external whisper tools.
  • Don’t confuse it with Stardog’s unrelated commercial “Voicebox.”

Simple-pimple: Voicebox is the engine. These 27 free repos are the turbo, the bodywork + the paint. Clean clip in → clone → re-skin the voice → dub a whole video → let your AI speak it. All offline, all yours.

Build the monster, then drop the chain you ran :backhand_index_pointing_down:

Tidied up by @SRZ — added a few free tools and extra resources on top of OP’s original find. :studio_microphone:

5 Likes