Soothly
An offline baby-cry classifier I built as a new dad.
The app records audio, resamples it to 16 kHz mono, extracts MFCC, chroma, mel-spectrogram, spectral contrast and tonnetz features into a 194-value vector, and runs an ONNX model that predicts the cry type with a confidence score. Everything happens on the phone. Built on @siteed/audio-studio, with custom feature implementations where mobile libraries fell short. Next: the same models on embedded baby-monitor hardware.
As a new dad, I wanted a small weekend project to help with parenting. What started as a simple idea quickly turned into a deep dive down the rabbit hole of analyzing different baby cry datasets and building a complete training pipeline to make everything work on-device. Classic scope creep, but worth every minute!
In the future, I plan to implement these same models on embedded devices with baby monitoring video/audio systems. Stay tuned!
Demo
Soothly in action, analyzing baby cries and providing insights
Preview
Soothly app interface showing cry classification and history tracking
Tech Stack
Key Features
- AI-Powered Analysis: The machine learning model analyzes baby cries in real-time to identify their needs - no more guesswork at 3 AM.
- Quick Recognition: Get instant insights into whether your baby is hungry, tired, uncomfortable, or needs a diaper change.
- History Tracking: Keep track of crying patterns and needs over time to better understand your baby's unique communication style.
- Privacy-First: All processing happens locally on your device - no data ever leaves your phone.
- Real-Time Performance: Optimized for speed to deliver near-instant cry detection, even on older devices.
How It Works
- Audio Capture & Preprocessing: The app records audio using the device's microphone, resamples to 16kHz, and converts to mono for consistent processing.
- Feature Extraction: I extract rich audio features including MFCC, Chroma, Mel Spectrogram, Spectral Contrast, and Tonnetz to capture the unique characteristics of the baby's cry.
- Feature Aggregation: Features are calculated over small frames of audio and averaged to create a compact 194-element feature vector.
- Model Inference: The feature vector feeds into an ONNX model that predicts the cry type along with a confidence score.
Tech Challenges
- Mobile Optimization: Implementing efficient audio processing on mobile devices with limited resources was a real pain
- Model Efficiency: Had to optimize the machine learning model for on-device inference while maintaining accuracy
- Feature Pipeline: Creating a feature extraction pipeline that matches the training environment took several iterations
- Battery Usage: Balancing performance with battery consumption for real-time analysis required careful optimization
Technical Implementation
- Audio Processing: Feature extraction leverages my own package @siteed/audio-studio (formerly @siteed/expo-audio-stream) for all the audio processing and feature extraction
- Custom Features: Built custom implementations for features like spectral contrast that weren't available in mobile libraries
- Cross-Platform: Used ONNX model format for cross-platform compatibility and performance
- Implementation: Implemented in React Native with native modules for performance-heavy tasks