@siteed/expo-audio-studio
A comprehensive audio studio library for React Native and Expo with recording, analysis, visualization, and streaming capabilities across iOS, Android, and web platforms.
Why?
I needed a robust solution for handling audio in cross-platform mobile apps. Existing libraries didn't quite fit my needs, especially for real-time audio analysis and feature extraction. So I built this to fill the gap and later expanded it into a complete audio processing toolkit.
This package was formerly known as @siteed/expo-audio-stream but has been renamed to better reflect its expanded capabilities beyond just audio streaming. The old package still works as a compatibility wrapper, but all new features are being added to expo-audio-studio.
Preview
@siteed/expo-audio-studio in action on iOS and Android.
Key Features
- Real-time Audio Streaming: Stream audio data across iOS, Android, and web platforms
- Dual-stream Recording: Simultaneous raw PCM and compressed audio recording with configurable formats and bitrates
- Intelligent Interruption Handling: Automatic pause/resume during phone calls with configurable resumption
- Background Recording: Keep recording active while app is in background with rich notification system
- Advanced Audio Analysis: Extract MFCC, Chroma, Mel Spectrogram, and other audio features for machine learning and visualization
- Precision Audio Manipulation: Trim audio files with support for multiple segments in a single operation
- Complete Ecosystem: Works with @siteed/expo-audio-ui for ready-to-use visualization components
Audio Analysis Features
The library provides powerful audio feature extraction capabilities that I've optimized for mobile devices:
- Basic Analysis: RMS, energy, amplitude range, zero-crossing rate
- Spectral Features: Spectral centroid, flatness, rolloff, bandwidth
- Advanced Analysis: MFCC, Chromagram, Mel Spectrogram, harmonics-to-noise ratio, tempo estimation, pitch detection
Tech Challenges
Building this library was no walk in the park. The biggest challenges were:
- Getting consistent audio feature extraction values across all platforms - had to implement custom native modules and carefully match the processing pipeline on each platform
- Optimizing for real-time processing without killing battery life or performance
- Implementing a dual-stream approach where one stream is optimized for analysis and another for playback
- Handling platform-specific audio interruptions (phone calls, notifications, etc.) gracefully
- Creating a consistent API that works across web, iOS, and Android despite their very different audio subsystems
API Design
I designed the API to be intuitive and flexible, with specialized functions for different audio processing needs:
- Recording Hooks: useAudioRecorder, AudioRecorderProvider, useSharedAudioRecorder
- Analysis Functions: extractAudioAnalysis, extractPreview, extractAudioData, extractRawWavAnalysis
- Specialized Processing: extractMelSpectrogram, trimAudio
- Utility Functions: convertPCMToFloat32, getWavFileInfo, writeWavHeader
Companion Libraries
I've also created companion libraries to make working with audio even easier:
- Audio Playground: A full-featured demo application showcasing the library's capabilities
- @siteed/expo-audio-ui: Ready-to-use UI components for audio applications including waveform visualizers, recording controls, and more