VibeVoice: Turn 60 Minutes of Audio into Who, When, and What

Speech AI is not only about converting audio into text. In a real meeting, interview, or customer-support workflow, the useful output must also identify who spoke, when each utterance happened, and how the conversation fits together. Long recordings make the problem harder: splitting an hour into short windows can lose context and make speaker tracking inconsistent.

Microsoft VibeVoice is an open-source family of voice AI models that addresses long-form speech recognition and synthesis. The most practical path in the repository today is VibeVoice-ASR. Its documentation describes single-pass processing for up to 60 minutes of continuous audio, structured transcription with speaker information and timestamps, customized hotwords, and support for more than 50 languages.

This article focuses on an engineering evaluation rather than presenting VibeVoice as a drop-in production service. It covers the repository layout, the official ASR setup, the two inference paths, the Realtime TTS design, and the limitations that should shape a safe deployment.

What is in the repository?

VibeVoice currently contains three main directions:

VibeVoice-ASR is built around the VibeVoice-ASR-7B model for long-form speech recognition with speaker information, timestamps, customized hotwords, and multilingual support.
VibeVoice-TTS was designed for long-form, multi-speaker synthesis of up to about 90 minutes with as many as four speakers. The current documentation explicitly says that installation and usage are disabled.
VibeVoice-Realtime is a deployment-oriented 0.5B text-to-speech variant. It is designed to start speaking while an LLM is still generating the rest of its response.

The distinction matters. A meeting transcription pipeline should start with the ASR documentation. A voice agent that speaks from an LLM token stream belongs to the Realtime TTS path. The models should not be treated as interchangeable simply because they live in one GitHub repository.

Why VibeVoice-ASR is interesting

Long-form context

The ASR documentation describes a 64K-token input context and a target of up to 60 minutes of continuous audio in one pass. Compared with a pipeline that cuts audio into many short clips, this gives the model more opportunity to maintain global context and consistent speaker tracking.

The claim is about the model and official inference path, not a guarantee that every 60-minute file will run comfortably on every GPU. Runtime and memory depend on the audio, hardware, configuration, and model loading overhead. A responsible evaluation should begin with short recordings and gradually increase duration.

Who, When, and What

Many speech-to-text services require separate diarization and timestamping components. VibeVoice-ASR positions these outputs together: who spoke, when they spoke, and what they said. That structure is useful for meeting notes, interview editing, legal review, and searchable support records.

Speaker labels remain model output, not unquestionable truth. Overlapping speech, background noise, distant microphones, and accents can affect both diarization and recognition. Production workflows should preserve the original audio, model version, inference settings, and a path for human review.

Customized hotwords

Product names, people, technical terms, and internal abbreviations are often harder to recognize than ordinary vocabulary. VibeVoice-ASR supports customized hotwords so a project can provide domain-specific terms or background information.

Hotwords should be treated as guidance rather than guaranteed dictionary replacement. A useful implementation maintains a vocabulary per customer or project, passes it into inference, and samples difficult segments against the original audio. This is especially valuable for technical meetings containing model names, frameworks, APIs, and internal terminology.

Official setup path

The official ASR guide recommends an NVIDIA Deep Learning Container for CUDA management and lists NVIDIA PyTorch Container versions from 24.07 through 25.12 as verified examples. A basic setup is:

sudo docker run --privileged --net=host --ipc=host \
  --ulimit memlock=-1:-1 --ulimit stack=-1:-1 \
  --gpus all --rm -it nvcr.io/nvidia/pytorch:25.12-py3
git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice
pip install -e .

The repository requires Python 3.10 or newer. Its project metadata includes PyTorch, Transformers, Diffusers, Librosa, Gradio, FastAPI, Uvicorn, and audio-processing dependencies. The ASR guide also notes that flash attention may need to be installed separately when it is not already available in the container.

Before a long run, verify CUDA compatibility, audio decoding through ffmpeg or av, and available disk space for model weights. A short smoke test is safer than sending sensitive one-hour recordings to a new environment.

Two official ASR inference paths

Gradio demo

The ASR documentation provides a Gradio demo for checking that the environment and model load correctly:

apt update && apt install ffmpeg -y
python demo/vibevoice_asr_gradio_demo.py \
  --model_path microsoft/VibeVoice-ASR --share

The --share option creates a public sharing endpoint. Do not use it casually with customer recordings, internal meetings, or personal data. Prefer a local or controlled network when testing sensitive audio.

Direct file inference

For a pipeline, the repository also provides a direct file-inference script:

python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --audio_files /path/to/meeting.wav

A maintainable service should put a small orchestration layer around this path. The service can validate uploads, normalize formats, create queue jobs, retain results, and retry failures while the GPU worker stays focused on inference. This separation also makes model upgrades and resource scheduling easier.

A maintainable speech pipeline

A meeting-transcription workflow can be split into these layers:

1. Accept the audio and create an unpredictable task ID. Store the source, uploader, hash, and retention policy.

2. Inspect the real media format, sample rate, channels, and duration instead of trusting the filename.

3. Load project-specific hotwords and record the model version, hardware, and inference parameters.

4. Send the job to a GPU worker and run VibeVoice-ASR through the official script or an equivalent Python integration.

5. Preserve the original structured result. Do not keep only the final summary; later search and review should be able to return to source segments.

6. Sample-check speaker labels, timestamps, and technical terms. Route low-quality jobs to human review.

7. Delete source audio after the retention period and protect transcripts and derived data with access controls.

The key design choice is to assume that model errors are normal. Once a transcript enters a knowledge base or automatic summarization flow, one recognition error can be amplified downstream. Traceable source data and metadata make those failures diagnosable.

Realtime TTS: letting an LLM speak earlier

VibeVoice-Realtime-0.5B targets a different problem. Its documentation describes streaming text input and an initial audible latency of roughly 200 milliseconds, subject to hardware and system conditions. The model is designed to begin playback while the LLM is still producing the complete answer.

Its interleaved, windowed design incrementally encodes incoming text while continuing diffusion-based acoustic latent generation from prior context. This is a useful direction for voice assistants, live narration, and streamed data. The documented limitations are equally important: it is single-speaker, primarily intended for English, does not handle background music or sound effects, and may behave unpredictably with code, formulas, or unusual symbols.

The official quick test is:

pip install -e .[streamingtts]
python demo/vibevoice_realtime_demo.py \
  --model_path microsoft/VibeVoice-Realtime-0.5B

The application still has to handle WebSocket transport, audio buffering, playback, and network delay. Model latency and the time a user actually hears audio are not the same measurement.

License, misuse, and deployment boundaries

The repository uses the MIT License, but code licensing does not automatically answer every question about model weights, data, or redistribution. Review the current model pages, reports, and repository documentation before adoption.

Synthetic speech also introduces impersonation, fraud, and disinformation risks. The project documentation warns that outputs can be biased, inaccurate, or unexpected, and does not recommend commercial or real-world use without further testing and development. Production systems should add provenance labels, content review, access control, abuse monitoring, and deletion mechanisms. Do not imitate a real person’s voice without consent. Treat ASR transcripts as data that needs validation, not as the sole basis for legal, medical, or financial decisions.

Who should evaluate VibeVoice?

VibeVoice is worth evaluating if you need searchable transcripts for long interviews or meetings, want to study single-pass long-context ASR, need domain terminology support, or want to connect an LLM token stream to low-latency speech. It should not be reduced to the claim that downloading the repository replaces every hosted speech API. GPU cost, model loading time, long-audio memory use, language coverage, output quality, and abuse prevention all require measurement.

Conclusion

VibeVoice is valuable because it presents a coherent open-source route for long-form speech systems. VibeVoice-ASR combines a 60-minute single-pass target with structured Who/When/What transcription, multilingual support, and hotwords. VibeVoice-Realtime explores how a smaller TTS model can connect LLM generation with earlier audio playback.

The most practical adoption path is to establish a benchmark with short audio and domain vocabulary first, then test duration, language, speaker count, and hardware. That process produces reproducible quality data and gives the team time to design privacy, licensing, review, and safety boundaries before sensitive voice data enters a production pipeline.

References

GitHub repository: https://github.com/microsoft/VibeVoice
VibeVoice-ASR documentation: https://github.com/microsoft/VibeVoice/blob/main/docs/vibevoice-asr.md
VibeVoice-Realtime documentation: https://github.com/microsoft/VibeVoice/blob/main/docs/vibevoice-realtime-0.5b.md
VibeVoice technical report: https://arxiv.org/pdf/2508.19205
VibeVoice-ASR report: https://arxiv.org/pdf/2601.18184