Sc_same - general purpose audio autoencoder

Resources Libraries and Quarks
1

Hey,

Sorry - I know everyone is tiered of AI stuff, but I think this is maybe actually a neat thing.
I ported the SAME autoencoder to run in SuperCollider :slight_smile: This was already laying around some weeks, but I finally managed to get it build in CI [onnx runtime…] and write some docs around it.
Think of it as a general purpose RAVE.

This is currently only tested on MacBooks with Apple Silicon chips (aka M-chips)!

This is still considered in development, so expect code to break on new versions/releases.

SAME is a Semantically-Aligned Music Autoencoder.
This UGens provides the En- and Decoder for the latent space, which covers 256 dimensions and represents 4096 samples (around 85ms at 48 kHz).
Since SAME is built on chunked attention, it is necessary to provide at least 2 successive frames into the model, which results in a latency of at least 8192 samples, which is about 170ms at 48 kHz.

Tested on a M4 Max where you can spin up 40 auto-encoders w/o hitting a barrier :slight_smile:
I tried to get it RT friendly as possible, but there are still some bad practices involved… [thread spawning and joining are running in RT thread :confused: needs a queue at some point…]

Currently the onnx export code is not included because it is rather messy… First time working with onnx, but it is rather nice regarding CoreML support which is really impressive!

Once this training code is cleaned up, I’ll try to add a LoRa training on the model such that it can be tweaked to a corpora, which hopefully enables style transfer like in RAVE. We will have to see about that…

Here is a small snippet from the examples

(
Ndef(\sameDelayVector, {
    var oldSignal = LocalIn.ar(2);
    var sig = Splay.ar(4.collect({|i|
        var inSig = VarSaw.ar((LFDNoise3.kr(3.5).exprange(6, 15).floor**3), LFDNoise3.kr(0.5).exprange(0.1, 0.9));
        var env = Decay2.ar(Impulse.ar(LFDNoise3.kr(0.5).exprange(1, 3).floor), LFDNoise0.kr(0.3).range(0.0, 0.1), LFDNoise0.kr(0.4).exprange(0.1, 0.9));
        inSig * env * LFDNoise3.kr(3.5).exprange(0.2, 1.0) * \sigAmp.kr(0.5, spec: [0.0, 1.0]);
    })) + (oldSignal*\fb.kr(0.99, spec: [0.01, 1.1, \exp]));
    // turn fb to 1.0 for "sitting in a room like" loop after some time

    var latent = SAMEEncode.ar(sig, numFrames: 40);
    var nnSig = SAMEDecode.ar(latent, numFrames: 2);

    // mute the neural signal until there are engouh values
    nnSig = nnSig * (Trig.ar(1.0, 50 * 4096/48000) -1.0).abs.poll(1.0, \nnActive);

    LocalOut.ar(nnSig);

    nnSig = SelectX.ar(\wet.kr(0.5, spec: [0.0, 1.0]), [sig, nnSig.tanh]);
	
	nnSig = (nnSig * 4).tanh;
	nnSig;
}).play.gui;
)

Ndef(\sameDelayVector).set(\wet, 1.0, \fb, 1.0, \sigAmp, 0.0)

Ndef(\sameDelayVector).stop(fadeTime: 5.0);

using SAME as a neural delay for a simple synth sequence - which kicks in around 0:07 and is in a feedback loop w/ itself at around 0:25

If someone has found some cool tweaks/applications/sounds for this, I’d appreciate if you could share it in this thread. It works really well for remixing existing tracks by moshing the latent space with external signals.

Still trying to figure out what can be done with it, but currently mostly looking forward to the LoRa stuff… Not really interested in the prompting stuff where this also gets used for - see also the rather sloppy magenta realtime by google for this, which violates the SC license btw SuperCollider plugin license · Issue #41 · magenta/magenta-realtime · GitHub ^^ It seems that even google has lost its GPL scare and simply doesn’t care about software licenses anymore.

Please do a PR if you know how to get this running on Windows/Linux! Or report if it maybe already works^^ You need to build it yourself though, but shouldn’t be too hard, at least for linux.

4 Likes