SuperSonic: SuperCollider in the browser

1

This project recently appeared on GitHub, by Sam Aaron:

10 Likes
2

The title is a bit misleading since it is about scsynth in the browser and not SuperCollider (which is probably the combination of sclang+scsynth).

scsynth in the browser (aka wasm) is already a thing since 2021 by the awesome work of Hanns Holger Rutz and has since been updated. But since we lack people who review code this hasnā€™t been merged yet (which I personally think is really sad and also adressed this in the SC symposium, but still no review) . Both PRs/efforts are also credited in the new project.

It would be interesting to know why this gets now developed outside of SuperCollider instead of getting the existing PR(s) merged, b/c I think it would be great to join efforts on it since I really like wasm :slight_smile:

4 Likes
3

@dscheiba is supersonic a better implementation? It seems to use the audioworklets rather than the old deprecated way.

4

Yes - I always wanted to work on that but it didnā€™t made sense for me to work on it since even the easy implementation wasnā€™t getting reviewed. I also wanted to make buffers work in wasm but w/o review it didnā€™t made any sense for me to me start working on that :confused:

5

So this issue of not enough reviewers is something I continually run into (e.g., Bytecodes, nan slot). It is annoying. Iā€™ve found you can eventually get people to just push it through by continually adding it to the dev meeting list until something happens!!! Had that PR been a bit more on my radar I think I would have approved it, even though web assembly isnā€™t something Iā€™m that familiar with because it is clearly very useful and a huge benefit.

Perhaps we should discuss a nicer organisational way to go about this as having to advocate for your work constantly is a little annoying, demeaning, and feels like begging? It could just be a simple tag that say we want this, and then it is mentioned briefly at the beginning of every dev meeting, just as a way to keep it fresh in peopleā€™s minds, before some deadline passes and it is merged?

ā€¦ as of today however, since an implementation exist that uses the audioworklets, I think your PR should be updated first. Not know much about web assembly, I wouldnā€™t feel comfortable approving a PR using a deprecated interface when a modern approach already exists.

6

Hello.
I came here because I was wondering what people here think about SuperSonic.
After having read this thread I am now even more curious.

How does this make everyone feel?
Letā€™s discuss.

From my perspective what Sam Aaron has done really is like the historic milestone achievement in aviation of breaking the Sound Barrier in 1947.

I feel this is the biggest deal in a long time. SuperCollider could now potentially go anywhere and do anything! No limits! Infinite deployment of generative music to all web sites everywhere! Ad astra, per civitas!

4 Likes
7

The idea of SC in the browser is a dream come true. However, this appears to be Sonic Pi, not sclang.

2 Likes
8

Yup, I agree. It just seems like Sam sort of beat us to the moon here, you know?

This thread here shows that there has been a pending pull request to add the wasm build of scsynth for quite some time:

I am not a dev myself, so I am not able to understand what it would mean to bring this into the main repo, but I am very curious.

I wonder if it would be feasible to have sclang also living on the web in some capacity. I think dealing with files just generally is tricky from what I can understand, but still, synthesis is cool. A lot can be done with synthesis.

1 Like
9

Landing on the moon is wonderful. As someone who is neither a dev, I am naĆÆve to what is involved and the potential this may have enabled. I am increasingly convinced that web capability would unshackle generative music created in SC; it is an anti-affordance to create music in a dynamic, generative way only to share that music as fixed recordings. It makes far more sense to share through the web. I am grateful to anyone who works on this and happy to support them. It would represent a great leap for Supercollider.

10

@dscheiba i would suggest to simply reach out to Sam and ask him if he would be willing to collaborate. He could make a PR and you could review it, or vice versa.

2 Likes
11

Hiya, just thought it would be nice to say hello and give a bit of background and motivation for my recent work on SuperSonicā€™s scsynth port.

Firstly I wanted to say thanks to Hanns Holger Rutz for his incredible work with the first port of scsynth to wasm . I had been watching this work on the sidelines since it started and was really excited about the possibilities of running scsynth in a browser.

I was therefore rather sad to see it languish but my interest was rejuvinated both by the fact I had started working on web-based audio tools (https://bleep.sheffield.ac.uk/artist/seclusion) so saw real value for it in my own work and also thanks to Dennis Scheibaā€™s mighty effort of getting the PR back up-to-date and ready to merge.

I therefore started looking at it in earnest - especially when there was really positive talk about merging it into main. However, many, many months passed and nothing seemed to be happening, but by that point I really had set my sights on integrating it into my latest work (https://tau5.live). I started looking into it deeply and discovered that the wasm port in the PR ran on the main browser thread (so would potentially suffer from xruns under load), used a deprecated API (that may be removed from browsers any point in the future) and didnā€™t have support for audio buffers.

Given that the new PR was stalled and that even if it was merged it wasnā€™t really suitable for my use-case (due to the reasons above) - I thought Iā€™d have a go at getting it working inside an AudioWorklet which would fix the two major issues. Naively I assumed this wouldnā€™t be too much work - perhaps just calling some different APIs and tweaking some cmake flags. Then I could easily contribute these changes back which might help create some further motivation to get it merged.

How wrong I was.

The more I worked on it, the more I realised that AudioWorklets have huge constraints that fundamentally clash with scsynthā€™s architecture including (but not limited to):

  • No thread spawning (scsynth is multi-threaded by design)
  • No IO (scsynth writes to stdout and reads/writes from disk and the network)
  • No main() entry point (scsynth assumes a main() entry point and also assumes that the C++ runtime intiialisers are automatically called on boot to set critical constants used to populate lookup tables and other things)
  • No malloc (scsynth dynamically allocates memory for a variety of different reasons)

This meant that in order to get scsynth running inside a web AudioWorklet, I had to chop scsynth up, turn it inside out, run it in non-realtime mode (which turns out to be single-threaded by design), implement whole new IO components, re-implement memory allocation functionality, statically initialise constants, and a whole lot more.

All the while I was working on it, I had a strong feeling I was trying to do something impossible - but the fact I saw such huge benefit of it existing drove me on. I had risked 2 months full-time working on it before I heard the first (broken) sounds coming out of the speakers. By this time it had evolved into something radically different to the original PR. I also had not even considered backwards compatibility with the original scsynth code-base as my main goal was just to see if I could get it working at all.

Iā€™m still not confident that the architecture Iā€™ve built is solid and reliable beyond simple use-cases but I am confident itā€™s full of bugs and issues to iron out even if the architecture does turn out to be sound.

The fact that itā€™s not out-of-the-box backwards compatible and super easy to merge into scsynth main, was a radically different system to scsynth (kind of like SuperNova is a radically different) - it made sense to me to release it as a separate project. This means that it doesnā€™t (yet) have to live up to the very high standards of scsynth main and that I would be free to continue to explore and experiment with the architecture.

I therefore donā€™t recommend merging it back into main at this stage. Instead I think it makes more sense to wait and see if what Iā€™ve built is actually decent and reliable. If that turns out to be the case, we can start discussing if and how we get an equivalent implementation into scsynth main.

This may end up being a ground-up rewrite using SuperSonic as a template, or it could be copying a bunch of my work. I honestly have no idea how palatable the design decisions I have made would be to the review team.

I described much of this in my announcement post (https://www.patreon.com/posts/introducing-in-141953467) and mentioned that it would be a real honour if my work helps get an AudioWorklet compatible scsynth into main.

I hope this makes sense and Iā€™m happy to answer any questions anyone may have.

Iā€™d highly encourage people to have a play with whatā€™s there. Iā€™ve got synthdefs triggering, and audio buffers working too - so Iā€™m really far along with the port and itā€™s already at a fun and useful stage. I plan to start integrating it into Tau5 and properly battle-testing it with earnest.

4 Likes
12

Totally fair - I actually fixed that a while back - I think it was only the title that was misleading - the main content made it clear that it was a port of scsynth only.

13

Itā€™s definitely not Sonic Pi in any way. Itā€™s just pure scsynth and has the standard scsynth OSC API.

However, the confusion might be because the demo does make use of Sonic Piā€™s synths. This is because scsynth itself has no means of describing and compiling synthdefs - it can only receive the binary format in an OSC message and load it into memory.

Note that the npm releases Iā€™ve created include Sonic Piā€™s synthdefs and samples as optional extras to help people get started (which is 120 synthdefs and 36mb of CC0 licensed samples):

1 Like
14

Hi Sam, thanks for chiming in!

The fact that itā€™s not out-of-the-box backwards compatible and super easy to merge into scsynth main, was a radically different system to scsynth (kind of like SuperNova is a radically different) - it made sense to me to release it as a separate project.

This totally makes sense! To be clear: I donā€™t think you did anything wrong! You have prominently credited the previous work of Hanns Holger and Dennis and you even linked to your project in our wasm PR (Add wasm build of scsynth by capital-G Ā· Pull Request #6569 Ā· supercollider/supercollider Ā· GitHub), which I really appreciated!

Nobody is required to contribute their changes back to the main SC repository, even more so if the architectural decisions have significantly diverged.

Itā€™s fair to say that your project gave us some momentum. @dscheiba has now picked up work on the wasm branch and hopefully we can finally merge it!

I actually read your Patreon announcement post, where you have already laid out the design, and hereā€™s something I donā€™t understand:

scsynthā€™s audio callback itself is fully RT-safe, i.e. it does none of the things listed above! Therefore I donā€™t quite understand why it couldnā€™t run in an AudioWorklet and why you would have to repurpose NRT synthesis (which, as the name says, is not NRT-safe by design).

I would have thought we would only have to rearrange the code so that non-realtime-safe members are not visible when compiling the AudioWorklet. Iā€™m not really familiar with audio web APIs, though, so forgive my ignorance :slight_smile:

15

That very well could be the case! This is precisely the reason why I made it a separate project - I have no real confidence that the approach I took was the cleanest - I just know that itā€™s working so far :slight_smile:

My understanding of NRT was that it was single threaded and batch oriented - just read osc from disc and spat osc out - so that seemed like a good starting point - I replaced the disk with a ring buffer that js could see.

Also really glad to have injected some energy into this work. I think itā€™s really important.

1 Like
16

scsynthā€™s audio callback is also single threaded. It does the following things:

  1. read incoming OSC messages from a lockfree queue (no actual network I/O!) and dispatch commands
  2. copy audio input buffers to the input Busses
  3. render a block of audio
  4. writes output Busses to the audio output buffers

Thatā€™s really it.

Note that in NRT syntehsis all asynchronous commands, such as /b_alloc or /b_read, are dispatched synchronously in the current thread. Thatā€™s why itā€™s fundamentally not suited for realtime use. How do you handle Buffers (or asynchronous commands in general) in your project?

17

Thanks for digging into this - I think there might be some terminology confusion on my part thatā€™s worth clearing up.

When I said I was using ā€œNRT modeā€ I meant setting the mRealTime flag to false - not using the World_NonRealTimeSynthesis() function (the offline file rendering pathway). Iā€™m still calling World_Run() directly from the AudioWorklet callback - the same RT-safe audio rendering function that the standard audio drivers use.

My understanding of NRT mode (mRealTime = false) was that itā€™s single-threaded and batch-oriented. In RT mode, the audio driver spawns an NRT thread that processes notification FIFOs (triggers, node replies, etc.) asynchronously. With mRealTime = false, I just process those same FIFOs synchronously in the AudioWorklet thread:

  g_world->hw->mTriggers.Perform();
  g_world->hw->mNodeMsgs.Perform();
  g_world->hw->mNodeEnds.Perform();

Similarly, I process incoming OSC from a ring buffer (instead of a network socket) synchronously before calling World_Run().

For async commands like /b_allocRead, theyā€™d normally use CallNextStage() to bounce between threads. With mRealTime = false theyā€™d call CallEveryStage() and block. So I avoid them entirely:

  1. JavaScript intercepts buffer commands and doesnā€™t forward them to scsynth.
  2. Web Audio API handles file fetch/decode (async, non-blocking in JS)
  3. Buffer data is loaded into SharedArrayBuffer (buffer malloc/free is managed exclusively in JS)
  4. A new custom /b_allocPtr OSC command is sent to scsynth which just updates pointers (no blocking stages)
  5. New OSC return messages are created to communicate back to JS when a buffer is freed and when it has been allocated (so they can manage the memory)

Your suggestion about conditional compilation to hide non-RT-safe members while keeping mRealTime = true is really interesting - that might be cleaner semantically. I went with mRealTime = false to minimise scsynth changes, though I may have created terminology confusion in the process.

Iā€™m still learning how all these pieces fit together - let me know if Iā€™m still misunderstanding something!

18

Thanks, that clears up a lot for me!

With mRealTime = false, I just process those same FIFOs synchronously in the AudioWorklet thread:

g_world->hw->mTriggers.Perform();
g_world->hw->mNodeMsgs.Perform();
g_world->hw->mNodeEnds.Perform();

But these commands are not realtime safe. In scsynth they would send OSC messages to the Client. In the browser I guess they would call JS callbacks?

I guess my main question is: why canā€™t we have a proper NRT thread in JS?

I would imagine something like this:

  1. JS main thread:
  • forward OSC messages from the user to the Server via World_SendPacket ( ā†’ unrolls the message/bundle and pushes a command to the mOscPacketsToEngine FIFO)
  • periodically check the mTriggers, mNodeMsgs and mNodeEnds FIFOs (= Client replies) and forward messages to JS callbacks
  1. AudioWorklet:
  • pop messages from mOscPacketsToEngine FIFO and dispatch commands
  • fill input Busses
  • compute audio via World_Run
  • copy output Busses
  1. NRT Worker: run the NRT stages of async commands

But again, I never really worked with WebAudio, so these are just wild guesses. It might be relevant for @dscheiba, though.

Side question: did you implement OSC bundles with timestamps? Or do you only support immediate messages/bundles?

19

Regarding FIFO RT-safety, my implementations of the FIFOs (mTriggers, mNodeMsgs, mNodeEnds) are RT-safe - they just write messages into lock-free ring buffers in the AudioWorklet thread. No JavaScript callbacks are invoked during audio processing. JavaScript workers use Atomics.wait() to block until the AudioWorklet wakes them with Atomics.notify() - so thereā€™s no polling, just efficient sleeping until messages arrive. The workers then read from the ring buffers and handle the messages (logging, sending over network, etc) outside the audio thread.

Your suggestion about JS main thread + AudioWorklet + NRT Worker is interesting! I think thatā€™s architecturally similar to what I ended up with, just organised differently:

  • My ā€œNRT Workerā€ equivalent is JavaScript on the main thread (handling file I/O, network, etc)
  • AudioWorklet does the audio computation (calling World_Run())
  • Ring buffers coordinate between them

The main difference is I avoided spawning additional workers for the NRT tasks - just did them on the main thread. But I could see benefits to your approach if the NRT operations got heavy enough to warrant their own worker. Worth exploring!

Finally, timestamped bundles are definitely supported! I have a pre-scheduler thatā€™s in JS-land that parks timestamped OSC bundles until 50ms prior to their execution time, then pass them through the ring offer to WASM where they enter a priority queue scheduler (based on SC_CoreAudioā€™s implementation) that processes bundles with future timetags. Note that bundles with timetag 1 (or 0 as a special case) execute immediately with sample-accurate timing (including subsample offset for precision).

Iā€™m still ironing out edge cases but it all seems to work!

20

Thanks!

I see! But canā€™t we make the FIFOs themselves appropriate for AudioWorklets (using the Atomics API)? Then we wouldnā€™t need all the additional buffering.

Note that bundles with timetag 1 (or 0 as a special case) execute immediately with sample-accurate timing (including subsample offset for precision).

How can an immediate bundle/message be executed with sample accurate timing when it doesnā€™t have a timestamp?

Another question: how do you obtain/calculate the OSC time for each control block in the AudioWorklet?