Csound with HTML5

22 April 2020

Michael Gogins

Irreducible Productions

http://michaelgogins.tumblr.com

• What this talk is about:
  • Demonstration pieces for Csound with HTML5.
  • Implementing the interface of Csound for HTML5.
  • Best coding practices for using Csound with HTML5.
• These are online slides that you can view from any Web browser.
• Example pieces run in the browser.

Csound

• Csound is a software sound synthesis system, or sound programming language, written by Barry Vercoe in 1985.
• It runs on Linux, OS X, Windows, Android, iOS, Raspberry Pi, Web browsers, and more.
• It has APIs in C, C++, Java, JavaScript, Python, Lua, Lisp, and more.
• Precisely because Csound is so old, it has a vast library of features, instruments, examples, etc.

HTML5

• HTML5 is the name for standards defining HTML, JavaScript, and many other things on the Web.
• It is notionally an an open standard, but in reality follows the lead of Google, Apple, Mozilla, etc.
• JavaScript is the programming language of the Web, enabling different objects in the browser to be programmed, interact, etc.
• HTML5 is highly secure due to the browser sandbox.
• It is versionless: everything from the old days still runs, and in the future will continue to run.

Csound and HTML5

• Several environments run Csound in the JavaScript context of a Web page:
  • csound.node
  • Csound for WebAssembly
  • Csound for Android

• Csound in each of these environments has a very similar interface; many pieces play in all environments.
• Algorithmic compositions can be written in JavaScript.
• My Silencio library provides a number of compositional algorithms.
• Check out Gibber for another HTML5 music playground!

Simple Example

Oblivion, Astor Piazzola

The essential Csound API

• new Csound() (WebAssembly only)
• Csound.SetMessageCallback(callback)
• Csound.SetOption(option)
• Csound.CompileCsdText(csd)
• Csound.Start()
• Csound.Perform()
• Csound.SetChannel(name, value) real-time control
• Csound.ReadScore(sco) live coding
• Csound.CompileOrc(orc) live coding
• Csound.Message(text) print message
• Csound.Stop()
• Csound.Cleanup()
• Csound.Reset()

Csound+NW.js+HTML5

• Csound compiled as a C++ addon runs in NW.js on desktop operating systems.
• Csound runs as native code, multi-threaded, with native audio.
• This is the most performant option.
• It also has the simplest implementation: one source file and one build file.

Implementation

• Csound is built as a "native addon" for Node.js, even though used primarily in NW.js.
• The Napi namespace abstracts the unstable v8 API.
• Regular node-gyp is used to build csound.node.
• Defining an interface: Napi::Number CompileCsdText(const Napi::CallbackInfo &info) { Napi::Env env = info.Env(); std::string csd = info[0].As<Napi::String>().Utf8Value(); int result = csound_.CompileCsdText(csd.c_str()); return Napi::Number::New(env, result); }
• Exposing that interface: Napi::Object Initialize(Napi::Env env, Napi::Object exports) { std::fprintf(stderr, "Initializing csound.node...\n"); uv_async_init(uv_default_loop(), &uv_csound_message_async, uv_csound_message_callback); std::atexit(&on_exit); csound_.SetMessageCallback(csoundMessageCallback_); ... exports.Set(Napi::String::New(env, "CompileCsdText"), Napi::Function::New(env, CompileCsdText)); ... std::fprintf(stderr, "csound.node initialized.\n"); return exports; } NODE_API_MODULE(csound, Initialize)

WebAssembly

• WebAssembly is a new Web standard for running compiled code on the Web.
• JavaScript is fast, but WebAssembly can run about as fast as C.
• WebAssembly can be compiled from source code written in C, C++, Rust, and more.
• WebAssembly modules with JavaScript APIs can, of course, call each other.
• WebAssembly might turn out to be a new universal computing platform.

Csound+WebAssembly+HTML5

• Csound compiled for WebAssembly runs in all major browsers and NW.js, and on desktop and mobile operating systems.
• Audio performance with Web Audio is now decent with the AudioWorklet/AudioWorkletProcessor design pattern.

Implementation

• Emscripten's LLVM-based toolchain compiles Csound sources to WebAssembly.
• Emscripten's Embind C++ template class defines a JavaScript interface to Csound's C++ class.
• CsoundAudioNode makes Csound a regular AudioWorklet Node in the WebAudio infrastructure.
• CsoundAudioNode creates a CsoundAudioProcessor to run Csound in a separate context.
• Worklet and Processor communicate via asynchronous MessagePorts.
• For proper sequencing of Csound calls, async/await makes selected async calls act synchronous.

Embind

• Embind is a C++ template library that automates the definition of JavaScript interfaces to existing C++ code.
• The Embind Csound class overrides selected methods of the base Csound C++ class to work with WebAudio.
• The Embind CsoundEmbind class derives from Csound and exposes all other required API calls.

Embind Declarations

Csound+Android+HTML5

• The Csound for Android app runs as native code for Android and includes HTML5 integration via WebKit.
• Source code at csound-android.

Implementation

• The CsoundOboe C++ class derives from Csound's C++ class and uses Google's Oboe library for native audio.
• SWIG generates a Java interface to CsoundOboe.
• SWIG annotations expose Java methods to JavaScript. %javamethodmodifiers CsoundOboe::CompileCsdText(const char *text) %{@JavascriptInterface public %} virtual int CompileCsdText(const char *csd_text){ return CsoundThreaded::CompileCsdText(csd_text); }
• CsoundOboe is then added to WebKit's JavaScript context. csound_oboe = new CsoundOboe(); String driver_value = PreferenceManager .getDefaultSharedPreferences(this).getString("audioDriver", "0"); int driver_index = Integer.parseInt(driver_value); csound_oboe.setOboeApi(driver_index); oboe_callback_wrapper = new CsoundCallbackWrapper(csound_oboe.getCsound()) { @Override public void MessageCallback(int attr, String msg) { Log.d("CsoundOboe:", msg); postMessage(msg); } }; oboe_callback_wrapper.SetMessageCallback(); html_tab.addJavascriptInterface(csound_oboe, "csound");

Complex Example

Scrims, Michael Gogins (WebAssembly)

• Interactive visual music: an animated voyage through variations on the hopalong fractal.
• Real-time controls:
  • Parameters of the hopalong fractal.
  • Mapping of fractal points to musical notes.
  • Neo-Riemannian transformations of notes to generate harmonic progessions.
  • Parameters of Csound instruments/effects.
• The exact same code runs in NW.js with csound.node, in browsers with WebAssembly, on Android with CsoundOboe.

Best Coding Practices

• Use only essential features of the Csound API (listed above).
• Use csound_loader.js to safely load Csound on all platforms.
• Code for running and stopping Csound should be in async functions or expressions so that async/await constructs can serialize Csound API calls: function Play() { (async () => { csound = get_csound(csound_message_callback); if (csound == null) { return; } await csound.CompileCsdText(document.getElementById('csd').value); await csound.Start(); is_playing = true; await csound.Perform(); })(); }

Best Coding Practices

• Use sr = 48000, ksmps = 128, nchnls = 2, nchnls_i = 1 on all platforms.
• On Linux, use ALSA -odac:plughw, this will also work (be ignored, rather) for WebAssembly and Android.
• Do not call other methods of the Csound object until Csound.Start() has been called.
• Keep your Csound orchestra code in your HTML file as a TextArea or string literal.
• Use current JavaScript idioms: import, let, class, etc.
• As with any other code, keep it simple, make it clear, use standard libraries: jquery.js, three.js, dat.gui.js.

Best Coding Practices

• Simplify your life: Use exactly the same name for the HTML input id, the Csound channel name, and the name of a global variable in Csound to hold the value of a slider, e.g. "gk_Reverb_feedback".
• Sending slider values to Csound with JQuery: $('input').on('input', function(event) { if (is_playing === false) { return; } let slider_value = parseFloat(event.target.value); csound.SetControlChannel(event.target.id, slider_value); let output_selector = '#' + event.target.id + '_output'; $(output_selector).val(slider_value); });
• Receiving slider values in Csound: instr Controls gk_Reverb_feedback chnget "gk_Reverb_feedback" endin