Does the traditional view of AI large models require them to be scaled up to a certain level to be useful? A recent open-source project that went viral on GitHub, MiniMind-O, provides a completely different answer. This full omni-modal model with only 0.1B (approximately 100 million) parameters gained over 3,000 stars within a week of launch and sparked heated discussions in the developer community. It can not only listen, speak, and see, but also has surprisingly low training costs—requiring only four RTX 3090 GPUs and four hours to train from scratch.
MiniMind Series: Starting from “Simplicity is the Ultimate Sophistication”
MiniMind-O didn’t emerge out of nowhere. Its developer, Jingyao Gong, has been building the MiniMind open-source series since 2024, with the core philosophy of “simplicity is the ultimate sophistication.” The earliest MiniMind language model had only 25.8M parameters (about 1/2700th of GPT-3), yet it can run the entire pipeline from pretraining to reinforcement learning on consumer-grade GPUs.
The newly released MiniMind-O extends this minimalist philosophy to the full multimodal domain, integrating automatic speech recognition (ASR), text-to-speech synthesis (TTS), and visual understanding into a model with just 0.1B parameters.
Technical Architecture: Thinker + Talker Dual-Module Design
According to what was officially published on arXivTechnical ReportThe architecture of MiniMind-O is quite clever. It adopts a two-stage design:
ThinkerUsing the complete MiniMind Transformer as the backbone to receive and understand inputs from three modalities: text, speech, and images. The speech modality uses a frozen SenseVoice encoder, while images are processed through the SigLIP2 visual encoder, with both mapped to a unified latent space through an MLP projector.
SpeakerA standalone component comprising four MiniMind modules that implements eight-layer streaming speech generation based on Mimi codec. The intermediate output states from Thinker are fused with Mimi codec’s historical information, then processed by Talker to produce smooth speech output.
This modular design enables the model to handle multi-modal inputs and outputs simultaneously with minimal parameters, without requiring the massive unified architecture of traditional full-modal models.
Training requirements: Four RTX 3090 GPUs, four hours
What impresses most about MiniMind-O is its training cost. According to the project documentation, training the complete 0.1B multimodal model from scratch requires only four NVIDIA RTX 3090 GPUs and takes approximately four hours.
This means that general developers, researchers, and even CS students can reproduce and fine-tune a full multimodal model with voice and image capabilities on personal devices or rented cloud GPUs. Compared to those massive models that require hundreds or even thousands of GPUs and weeks of training, MiniMind-O has completely lowered the barrier to multimodal AI research by several levels.
He also shared the entire content on Bilibili. Friends who are interested can check it out, and there’s also…online versionAvailable for experience:
Thinker + Talker Collaboration Workflow
Specifically, the operation process of MiniMind-O is as follows:
1. Users input voice, text, or images
2. Audio is encoded via SenseVoice, images are encoded via SigLIP2, and then projected into MiniMind’s latent space using an MLP.
3. The Thinker processes this multimodal information, generating intermediate representations
4. Talker (Speaker) generates streaming voice replies based on Thinker’s output through Mimi codec
This approach enables MiniMind-O to think before responding like a real person, while also supporting various tasks including ASR (speech-to-text), TTS (text-to-speech), and image understanding—all within just 0.1B parameters.
Open Source Scope: Weights, Code, and Dataset All Publicly Available
Unlike some projects that brand themselves as open source but only release partial components, MiniMind-O achieves true and complete open source:
• model weightsThe trained checkpoints can be directly downloaded and used.
• Training codeComplete PyTorch Implementation: From Data Preprocessing to Model Training
• Training datasetContains three core Parquet format datasets: T2A (text-to-speech), I2T (image-to-text), and A2A (speech-to-speech).
• Technical ReportA paper detailing the architecture design and experimental results
This transparency allows researchers not only to use the model, but also to gain a deeper understanding of its internal workings, and even to make improvements upon it.
MiniMind-O GitHub URL
The Potential of Edge Deployment
The 0.1B parameter scale means MiniMind-O has great potential for edge deployment. Compared to multi-billion parameter models that require cloud computing, this ultra-lightweight all-modal model can run directly locally on mobile phones, IoT devices, embedded systems, or wearable devices.
For applications such as offline voice assistants, smart home control, and real-time translation devices, these lightweight models capable of handling “listening, speaking, and seeing” tasks locally have tremendous potential. More importantly, fully local computation means user data doesn’t need to be uploaded to the cloud, making it more privacy-friendly.
Currently, developers on GitHub have started experimenting with porting MiniMind-O to different hardware platforms, and teams are also exploring ways to further reduce model size while maintaining performance. For readers interested in open-source AI and edge computing, this project is worth following closely.
Source: KOCPC Chinese