Audio, Web and Worklets
The Web is the ultimate inner platform. Over the years, it has laboriously assimilated more and more operating system features. Regardless of whether you believe that makes sense, the Web has enormous momentum: its reach and accessibility are compelling, and so here we are, building signal processors in the web browser.
Audio as an Afterthought
My earliest recollection of audio on the web is of General MIDI tunes in an
<embed> tag. Back then the Web was not really an app platform: there was no intention of programmatic control over creation or playback for such "embedded multimedia objects".
Gradually the platform got more capable. HTML5 brought the HTMLMediaElement family, including the
Coinciding with the Web platform moving beyond hypermedia, a new audio api more suited for procedural audio, the Web Audio API, was introduced in 2011. Like many audio stacks in modern operating systems, it was fashioned as a graph of nodes. Each browser would implement a bunch of audio processing primitives, like oscillators, envelopes and filters, and web apps could build and manipulate graphs of those primitives on the go.
But what would one do when none of the built-in nodes was quite what was needed?
Consider graphics: maybe you were able to religiously maintain 60 frames per second by capping anything else at 16 milliseconds, which is not easy on the Web platform. For audio, that is an okay-ish but not great latency. In addition, any time you spend processing audio is subtracted from your frame budget. Prepare for the occasional major garbage collection or document reflow. In graphics, you drop a frame or two. For audio, it's snap crackle pop time.
WebAssembly was the next major advance in the quest for performant low-latency work on the Web platform. It takes the idea of asm.js to its logical conclusion. Wasm is a compactly encoded binary instruction format that uses linear memory and typed primitives.
It is the ideal target for audio code on the Web, if only we could somehow avoid running in the main event loop...
This is a great time to introduce support in Veneer; you can try it in the latest snapshot by heading over to
Project > Global Preferences and selecting the new audio threading model. Let me know if things break!
Threading in Veneer
Veneer uses multiple concurrent processes for better performance. While Kronos is fairly quick at compiling, it is still too slow to use synchronously from the UI. The native compiler runs in a dedicated Web Worker. Veneer sends code via a message queue, and receives binary Wasm blobs asynchronously in response. The browser prepares these for execution in another asynchronous process, which is the work queue for WebAssembly compilation. The resulting executable instance is connected to Web Audio via
The situation is slightly more complicated with
The main thread sends our new WebAssembly Module to the AudioWorklet context over the message queue. The module is instantiated worklet-side and associated with the AudioWorkletProcessor. Audio IO and the link between our AudioWorkletNode and AudioWorkletProcessor is handled by the Web Audio behind the scenes.
For interactive control, we send any parameter changes from the main thread over to the AudioWorkletProcessor over a message queue, and receive waveform peak data and any readout values in the other direction.
Bare Virtual Metal
There is one remaining twist, and it is not adult entertainment even if the heading made you think so.
In the case of ScriptProcessorNode we can hitch a ride on the C runtime library: there is already a small WebAssembly image present in the main thread for parsing Kronos expressions. It is created by Emscripten, which gracefully provides us with
free for memory management.
I hope Veneer's new audio capabilities work well for you! Let me know how it turns out.