Audio Anywhere

ADC, November 2020

Benedict R. Gaster / @cuberoo_
my pronouns are he/him/they or xe/xim/xir

Material related to this talk

• Slides can be downloaded here: https://bgaster.github.io/presentations_reveal/adc_11_2020/
• Project home page
• Source code is at https://github.com/bgaster
• Up and comming paper: Gaster, Benedict R. and Cole, Max. (2020) Audio Anywhere with Faust. To appear in Proceedings of the 2ndInternational Faust Conference (IFC-20), December 2020

quick demo

Audio Anywhere

An experimental system for real-time audio built with WebAssembly

(But not for the Web)

What is WebAssembly?

						
[] == ![];

NaN === NaN; // false

x = 100;

while(true) {
  if (x == 999) {
    x = "OMG";
    break;
  }
  x = x + 1;
}

log.console(x);
						
					

In 2011, Alon Zakai asked the question

What if you could compile code for the Browser?

But why stop there?

A Simple Module in WebAssembly Text Format

					
(module
 (import "math" "callback" (func $callback))

 (export "add" (func $add))
 (export "subtract" (func $substract))

 (func $add (param $a i32) (param $b i32) (result i32)
  local.get $a
  local.get $b
  i32.add
 )

 (func $subtract (param $a i32) (param $b i32) (result i32)
  local.get $a
  local.get $b
  i32.sub
 )
)
					
				

Wasm is not limited to the Browser

A number of projects can run Wasm natively

• Wasmtime
• Wasmer
• Life
• wasm2c

demo

Wasm could imply, compile once run anywhere

Alon Zakai, 2020

Things would be different one day. But you had to start small, like oak trees.

Tiffany Aching

AA Module

Hosting App Loads Bundle

						
{
	"wasm": [String],
	"gui": String,
	"info": {
		...
		"inputs": int, 
		"outputs": int, 
		...
	}
}
						
					

AA Bundle

• URLs: module.wasm(s), each one implements the AA Wasm API for nodes in audio graph;
• URL: gui.html, implementing the Javascript API for UI plugins; and
• Set of configuration parameters, which are defined as part of the module bundle itself and describe its capabilities, parameter mappings, and initialization constants

Audio Anywhere Wasm API

A C Interface

AA Wasm API (1)

						
// initialize module 
#[no_mangle]
pub fn init(sample_rate: f64) -> u64 // returns handle to module

// module meta data 
#[no_mangle]
pub fn get_sample_rate(module: u64) -> f64 
#[no_mangle]
pub fn get_inputs(module: u64) -> u32 
#[no_mangle]
pub fn get_outputs(module: u64) -> u32 
#[no_mangle]
pub fn get_voices(module: u64) -> i32
						
					

AA Wasm API (2)

						
// parameters
#[no_mangle]
pub fn get_param_index(module: u64, length: i32) -> i32 
#[no_mangle]
pub fn get_num_params_float(module: u64) -> u32 
#[no_mangle]
pub fn set_param_float(module: u64, index: u32, v: f32) 
#[no_mangle]
pub fn get_param_float(module: u64, index: u32) -> f32

// MIDI 
#[no_mangle]
pub fn handle_note_on(module: u64, mn: i32, vel: f32) 
#[no_mangle]
pub fn handle_note_off(module: u64, mn: i32, vel: f32)
						
					

AA Wasm API (3)

						
// compute audio 
#[no_mangle]
pub fn compute(module: u64, frames: u32)

// input and output buffer management 
#[no_mangle]
pub fn get_input(module: u64, index: u32) -> u32 
#[no_mangle]
pub fn get_output(module: u64, index: u32) -> u32 
#[no_mangle]
pub fn set_input(module: u64, index: u32, offset: u32) 
#[no_mangle]
pub fn set_output(module: u64, index: u32, offset: u32)

// extern functions provided by AA Host
extern "C" pub fn set(module: u64, index: u32, v: f32)
						
					

Writing the audio component

FAUST (Functional AUdio STream) is a domain-specific purely functional programming language for implementing signal processing algorithms in the form of libraries, audio plug-ins, or standalone applications.

AA Faust Example

						
declare aavoices "2"; 

import("stdfaust.lib"); 

process = vgroup("voices", par(n, 2, vgroup("aavoice%n", voice))) ;
voice = hgroup("midi", osc(freq)) 
with {
 freq = hslider("freq",200,50,1000,0.01); 
 gain = hslider("gain",0.5,0,1,0.01); gate = button("gate");
 envelope = en.adsr(0.01,0.01,0.8,0.1,gate)*gain;
 osc(freq) = os.sawtooth(freq)*envelope; 
};
						
					

Implementation

Compilation flow for AA DSP Code

AA module static memory layout

Compile Faust to Wasm via Rust

• Why Rust?
• Rust is a particularly close friend to Wasm, but
• It also provides a root to auto-vectorization and Wasm-128 SIMD
• All the supporting libraries are written in Rust

Compile Faust to Rust

						
					faust2audioanywhere synth.dsp
						
					

Generated Rust for Faust .dsp

						
struct mydsp {
  ...
}

impl mydsp {
  pub fn get_voices(&self) -> i32 { ... }
  pub fn get_input(&self, index: u32) -> u32 { ... }
  pub fn get_output(&self, index: u32) -> u32 { ... }
  pub fn get_num_outputs(&self) -> i32 { ... }
  pub fn handle_note_on(&mut self, mn: Note, vel: f32) { ... }	

  unsafe fn compute(
	&mut self, count: i32, inputs: &[T], outputs: &mut [&mut [T];2]) { ... }
	
  pub fn compute_external(&mut self, count: i32) { ... }
}
						
					

Generated compute_external function

						
pub fn compute_external(&mut self, count: i32) {
  unsafe {
    let (output0, output1) = 
      (::std::slice::from_raw_parts_mut(OUTPUTS[0], count as usize), 
       ::std::slice::from_raw_parts_mut(OUTPUTS[1], count as usize));

    self.compute(count, &[], &mut [output0, output1]);
  }
}	
						
					

generated compute function

						
#[target_feature(enable = "simd128")] 
unsafe fn compute(
  &mut self, count: i32, inputs: &[T], outputs: &mut [&mut [T];2]) {
  let [outputs0, outputs1] = outputs; 
  let (outputs0, outputs1) = {
    let outputs0 = outputs0[..count as usize].iter_mut(); 
    let outputs1 = outputs1[..count as usize].iter_mut();
    (outputs0, outputs1)
  };
  ...

  let zipped_iterators = outputs0.zip(outputs1);
  for (output0, output1) in zipped_iterators {
    ...
    *output0 = ... ...
    *output1 = ... ...
  } 
}
						
					

But how does it run in practice?

Faust copy processor - C++ vs Rust

• Original Faust to Rust compiler, 20 times slower than C++
• Optimized Faust to Rust compiler, slightly faster than C++

Null function call

Sine OSC

Copy 1 channel in/out

Next steps

Wasm lacks relocatable code

Questions?