iFFT Freeze/Scrub

The output file is not necessarily audio rate - it writes to an audio file, but that’s only because the only output format NRT supports is audio files. If you e.g. specify a control rate process, you’ll get an audio file w/ control rate data - likewise with outputting triggers via putTr. The quark will read these audio files back into sclang as arrays, so you don’t need to interact with the on-disk audio files at all unless you explicitly need to.

1 Like

it was the language: “Offlice Process expects audio rate output Ugen” in the help that made me wonder - the OP is hoping to store a PV chain - possible?

(and something like this should really be included in SC imo)

Here’s an example of capturing an FFT chain:

(
~o = OfflineProcess();
~o.putTrig(\fft, {
	|in|
	var fft, magsPhases;
	fft = FFT(LocalBuf(256), in);
	magsPhases = UnpackFFT(fft, 256);
	[fft, magsPhases];
});

fork {
	~run = ~o.process(Platform.resourceDir +/+ "sounds/a11wlk01.wav");
	~run.wait();
	
	"DONE".postln;
	~run.resultData(\fft).do(_.postln);
};
)

fft in your SynthDef is actually a trigger that is fired every time there’s a new FFT buffer available, so you can use it as the trigger for putTrig. One caveat here is that the above won’t naively work for FFT sizes larger than 256 - I think this might be a limitation of UnpackFFT? You may need to use multiple UnpackFFT's, or roll your own with an array of Unpack1FFT.

1 Like

I get

“Execution warning: Class ‘Deferred’ not found
ERROR: Message ‘new’ not understood.”

After compiling your Quark and trying that code

Deferred quark also on szctt github IIRC

The OfflineProcess looks interesting, but it wouldn’t result in a buffer that could be used with PV_PlayBuf. So, playing back may be difficult.

You can run a PV_RecordBuf synth in a NRT server.

p = Platform.resourceDir +/+ "sounds/a11wlk01.wav";
q = "~/pvtest.wav".standardizePath;

// get duration
f = SoundFile.openRead(p);
f.close;
f.duration  // 4.2832879818594

(
var resultbuf, inbuf;
var fftSize = 512;

z = Server(\nrt, NetAddr("127.0.0.1", 57110),
	ServerOptions.new
	.numOutputBusChannels_(2)
	.numInputBusChannels_(2)
	.sampleRate_(44100)
);

inbuf = Buffer(z, 65536, 1);
resultbuf = Buffer(z, f.duration.calcPVRecSize(fftSize, 0.5, z.options.sampleRate), 1);

x = Score([
	[0, inbuf.allocMsg],
	[0, resultbuf.allocMsg],
	[0, inbuf.readMsg(p, leaveOpen: true)],
	[0, [\d_recv, SynthDef(\pv_ana, {
		var sig = VDiskIn.ar(1, inbuf, f.sampleRate / SampleRate.ir);
		var fft = FFT(LocalBuf(fftSize, 1), sig);
		fft = PV_RecordBuf(fft, resultbuf, run: 1);
		Out.ar(0, sig);
	}).asBytes]],
	[0, Synth.basicNew(\pv_ana, z).newMsg],
	[f.duration + (fftSize / z.options.sampleRate),
		resultbuf.writeMsg(q, "wav", "float")
	]
]);

x.recordNRT(
	outputFilePath: if(thisProcess.platform.name == \windows) { "NUL" } { "/dev/null" },
	headerFormat: "wav", sampleRate: z.options.sampleRate,
	options: z.options,
	duration: f.duration + (fftSize / z.options.sampleRate),
	action: { "done".postln }
);

z.remove;
)

And play it back:

s.boot;

// this is the pv_rec result
b = Buffer.read(s, q);

(
SynthDef(\pvmouse, { |out = 0, recBuf = 1, fftSize = 512|
	var in, chain, bufnum;
	bufnum = LocalBuf.new(fftSize);
	chain = PV_BufRd(bufnum, recBuf, MouseX.kr(0, 1));
	Out.ar(out, IFFT(chain).dup);
}).add;
)

a = Synth(\pvmouse, [recBuf: b]);
a.free;

hjh

3 Likes

This got over my head quite quickly. Have to read up on some basics, I think. Good to now there are solutions, though. Thanks very much guys!

My NRT example is mostly based on an example in the help: Non-Realtime Synthesis (NRT) | SuperCollider 3.11.1 Help

NRT is very different from normal real-time usage. It does make sense but it takes some time to understand the differences. The linked help file tries to walk you through them – if you expect to want to do more of this type of processing, it’s worth it to work your way through it.

hjh

but … I ran James’ code and …so fun to scrub so thanks to you both!

1 Like

Hi @semiquaver do you have the exact code you used to hand?

Don’t worry, got it working. And it’s a LOT of fun!!

Thank you very much, guys!!!

Hi @jamshark70 I’m really enjoying developing my PV scrubbing script. Thanks so much for your example!

I wonder if it would be possible to analyse the pitch of the original audio, and save that to a separate file, in the same way as the FFT analysis, and to load that back into a buffer and scrub through it at the same time as the FFT data.

With a known fundamental frequency, I could do frequency-domain pitch-shifting/flattening, which would be really cool!

Do you think that’s possible?

It’s certainly possible.

The analysis example creates one resultBuf – there’s nothing stopping you from creating a second resultBuf for a control-rate pitch detector result. What is the size of this buffer? Control rate is one value per blockSize audio samples, so:

inbuf = Buffer(z, 65536, 1);
resultbuf = Buffer(z, f.duration.calcPVRecSize(fftSize, 0.5, z.options.sampleRate), 1);
pitchbuf = Buffer(z, (f.numFrames / z.options.blockSize).roundUp.asInteger, 2);

(2 channels because pitch detectors usually output a confidence value along with the frequency.)

Then add a second analysis and second buffer recorder into the analysis SynthDef.

SynthDef(\pv_ana, {
	var sig = VDiskIn.ar(1, inbuf, f.sampleRate / SampleRate.ir);
	var fft = FFT(LocalBuf(fftSize, 1), sig);
	var pitch = Tartini.kr(sig);
	RecordBuf.kr(pitch, pitchbuf, loop: 0);
	fft = PV_RecordBuf(fft, resultbuf, run: 1);
	Out.ar(0, sig);
})

And where it writes the FFT analysis buffer, add a second command to write the pitch buffer.

	[f.duration + (fftSize / z.options.sampleRate),
		resultbuf.writeMsg(q, "wav", "float")
	],
	[f.duration + (fftSize / z.options.sampleRate),
		pitchbuf.writeMsg(q.dirname +/+ "pitchbuf.wav", "wav", "float")
	]

One thing I would note here is that – there isn’t much new in the above advice. The example already shows how to create, fill and write an analysis buffer – and the answer to the question is just to do that with two buffers.

Or, as I’ve been preaching lately: solve one problem at a time. If you want to integrate pitch analysis into a NRT analysis code snippet, then… the integration can’t happen without having something to integrate (that is, without working out the pitch analysis by itself first – i.e., generating the pitch analysis signals, and then storing them in a buffer).

Quite often, I see people trying to handle all of the requirements in one go. That’s a good recipe to get confused.

hjh

2 Likes

That’s cool!

There are specific uGens for FFT data, but I wasn’t sure if it was possible to save arbitrary data to file, and read it back. It seems it is, which might help me with a future project, too, actually!

The pitch data file seems to be mono, rather than stereo, despite this line (I assume) creating a 2-channel buffer

pitchBuf = Buffer(z, (f.numFrames / nrtServer.options.blockSize).roundUp.asInteger, 2);

I’m not sure if the data recorded to it its actually the pitch, or confidence values. I’ll need to find a source with a more obvious pitch, I think, to find out.

Hm, did you also b_alloc the buffer? (I forgot to spell that out explicitly… occupational hazard of helping out – every error or omission comes back to bite you :laughing: )

If the buffer object is created with 2 channels, then its allocMsg should specify 2 channels to the server, and if the server-side buffer is allocated with 2 channels, then it should write to disk with 2 channels. If you get only one channel, I’d have to suspect an allocation problem first.

See the NRT guide helpfile… the initial code I posted is adapted from it. It’s using a buffer to store and write onset times for audio events in a soundfile – which is pretty arbitrary as data go.

hjh

Ah no, I didn’t. Noob here, I’m afraid.

Does this look good?

(

///////////////////////////////////////
// Non-Realtime FFT Analysis To File //
///////////////////////////////////////

~analyseWav = {
	arg wavIn, fftOut, pitchOut;

	// Define function vars
	var inBuf, fftBuf, pitchBuf, fftSize = ~fftSize, f, nrtServer, nrtScore, env;

	f = SoundFile.openRead(wavIn);
	f.close;
	f.duration;

	// Define Non-RealTime server
	nrtServer = Server(\nrt, NetAddr("127.0.0.1", 57110),
		ServerOptions.new
		.numOutputBusChannels_(2)
		.numInputBusChannels_(2)
		.sampleRate_(44100)
	);

	// Create buffer for input file
	inBuf = Buffer(nrtServer, 65536, 1);

	// Create output buffer for FFT analysis data
	fftBuf = Buffer(nrtServer, f.duration.calcPVRecSize(fftSize, 0.5, nrtServer.options.sampleRate), 1);
	pitchBuf = Buffer(nrtServer, (f.numFrames / nrtServer.options.blockSize).roundUp.asInteger, 2);

	// Create score for recording of FFT analysis to WAV file
	nrtScore = Score([
		[0, inBuf.allocMsg],
		[0, fftBuf.allocMsg],
		[0, pitchBuf.allocMsg],
		[0, inBuf.readMsg(wavIn, leaveOpen: true)],
		[0, [\d_recv, SynthDef(\pv_ana, {
			var sig = VDiskIn.ar(1, inBuf, f.sampleRate / SampleRate.ir);
			var fft = FFT(LocalBuf(fftSize, 1), sig);
			var pitch = Tartini.kr(sig);
			RecordBuf.kr(pitch, pitchBuf, loop: 0);
			fft = PV_RecordBuf(fft, fftBuf, run: 1);
			Out.ar(0, sig);
		}).asBytes]],
		[0, Synth.basicNew(\pv_ana, nrtServer).newMsg],
		[f.duration + (fftSize / nrtServer.options.sampleRate),
			fftBuf.writeMsg(fftOut, "wav", "float");
		],
		[f.duration + (fftSize / nrtServer.options.sampleRate),
			pitchBuf.writeMsg(pitchOut, "wav", "float");
		]

	]);

	// Run score
	nrtScore.recordNRT(
		outputFilePath: if(thisProcess.platform.name == \windows) { "NUL" } { "/dev/null" },
		headerFormat: "wav",
		sampleRate: nrtServer.options.sampleRate,
		options: nrtServer.options,
		duration: f.duration + (fftSize / nrtServer.options.sampleRate),
		action: { "done".postln }
	);

	// Free score
	nrtScore.free;

	// Free NRT server
	nrtServer.remove;

	inBuf.free;
	fftBuf.free;
	pitchBuf.free;
}

)

(

/////////////////
// Global Vars //
/////////////////

~fftSize = 2048;

// Default soundfile to be analysed
~wavPath = Platform.resourceDir +/+ "sounds/a11wlk01.wav";

// Default location for analysis file
~fftPath = "~/fft.wav".standardizePath;

~pitchPath = "~/pitch.wav".standardizePath;

~analyseWav.value(~wavPath, ~fftPath, ~pitchPath);

)

It certainly produces a stereo pitch output file, now. I can’t be sure it’s accurate, but then I imagine the analysis algorithm is going to struggle to extract meaningful pitch from a11wlk01.wav.

It doesn’t seem to work reliably at all though, unfortunately. More often than not, it produces an analysis file that’s corrupt or empty, apart from a loud click at the beginning.

I just tried it with a solo flute recording:

~analyseWav.value(~wavPath, ~fftPath, ~pitchPath);  // with my file

f = SoundFile.openRead(~pitchPath);
[f.numFrames, f.numChannels]  // OK

d = Signal.newClear(f.numFrames * f.numChannels);
f.readData(d);
f.close;

e = d.as(Array);
p = e.plot(numChannels: 2);

p.specs_([ControlSpec(0, 2500), ControlSpec(0, 1)]).refresh;

This much is as expected – it is tracking the melody’s contour.

Your scrubbing synth should be responsible for avoiding unreasonable values for shifting, then… (which can be tricky).

hjh

I should clarify: when I say it’s been unreliable, I mean if I repeatedly attempt to analyse the same file, most of the time it fails, as I mentioned, and occasionally it appears to work.

It seems to work more reliably when I comment-out all the lines related to the pitch analysis.

I’ve also found that often (but not all the time) when I trigger a note to instantiate a synth, the server immediately quits, with status 0 (which isn’t very helpful).

Here’s the synth (note I’m not doing anything with the pitch analysis data yet):

(

b = Buffer.read(s, ~fftPath);
p = Buffer.read(s, ~pitchPath);

//////////////////
// Define Synth //
//////////////////

SynthDef(\pvscrub, {
	arg out = 0, fBuf = 1,
	pOffset = 0, pBend = 0,
	amp = 0.1, gate = 1,
	ampEnvAttack = 0.01, ampEnvDecay = 0, ampEnvSustain = 1.0, ampEnvRelease = 2,
	grainSize = 0.1, grainPitchDispersion = 0.01, grainTimeDispersion = 0.004,
	//filterCutoff, filterResonance, filterShape,
	scrubPos = 0.1, scrubPosInertia = 0.01,
	pvParam1 = 0, pvParam2 = 0;

	var fftSize = ~fftSize, chain, bufNum, ampEnv, centerFreq, result;

	// Amplitude envelope Attack Sustain Release type
	// Free synth after envelope finishes
	ampEnv = Env.adsr(ampEnvAttack, ampEnvDecay, ampEnvSustain, ampEnvRelease, amp).kr(2, gate);

	// Read FFT buffer
 	bufNum = LocalBuf.new(fftSize);
	chain = PV_BufRd(bufNum, fBuf, min(scrubPos.lag(scrubPosInertia), 1));

	// FFT FX
	chain = PV_RectComb(chain, pvParam1.lag(1), 0);
	chain = PV_MagSmooth(chain, pvParam2.lag(1));

	// TODO

	// Resynthesise FFT
	result = IFFT(chain).dup;

	// Time-domain pitch-shift
	result = PitchShift.ar(
        result,
        grainSize,
		pOffset,   // Pitch-shift ratio
		grainPitchDispersion.lag(2),
        grainTimeDispersion.lag(2)
    );

	// Filter

	// TODO

	// Resonant low-pass

	// Non-resonant hi-Pass/shelf

	// Final output
	Out.ar(out, result * ampEnv);
}).add;

)

And here’s the MIDI setup

(

//////////////////
// Setup Busses //
//////////////////

~setupControls = {
	// Setup control buses
	~scrubControl = Bus.control(s, 1);
	~scrubInertiaControl = Bus.control(s, 1);
	~bendControl = Bus.control(s, 1);
	~grainPitchDispersionControl = Bus.control(s, 1);
	~pvParam1Control = Bus.control(s, 1);
	~pvParam2Control = Bus.control(s, 1);
	//~filterCutoffControl = Bus.control(s, 1);
	//~filterResonanceControl = Bus.control(s, 1);
	//~filterShapeControl = Bus.control(s, 1);
}.value;

////////////////
// Setup MIDI //
////////////////

~setupMIDI = {
	var notes, on, off, cc, centreNote = 60;

	MIDIClient.init;
	MIDIIn.connectAll;


	notes = Array.newClear(128);    // array has one slot per possible MIDI note

	// Handle note-on events
	on = MIDIFunc.noteOn({
		arg vel, note, chan, src;

		var noteScaled, pOffsetRatio;

		// Note clamped to 48..72 range
		// 2 octaves either side of C4, -24..+24 semitone range
		noteScaled = max(min(note, centreNote + 24), centreNote - 24) - centreNote;

		// Pitch-offset as ratio
		pOffsetRatio = noteScaled.midiratio;

		// Push synth to notes array
		notes[note] = Synth(\pvscrub, [\out: 0,\fBuf: b,\pOffset: pOffsetRatio,\amp: vel.linlin(0, 127, 0, 1)]);
		// Setup control busses
		notes[note].set(\scrubPos, ~scrubControl.asMap);
		notes[note].set(\scrubPosInertia, ~scrubInertiaControl.asMap);
		notes[note].set(\pBend, ~bendControl.asMap);
		notes[note].set(\grainPitchDispersion, ~grainPitchDispersionControl.asMap);
		notes[note].set(\pvParam1, ~pvParam1Control.asMap);
		notes[note].set(\pvParam2, ~pvParam2Control.asMap);
		//notes[note].set(\filterCutoff, ~filterCutoffControl.asMap);
		//notes[note].set(\filterResonance, ~filterResonanceControl.asMap);
		//notes[note].set(\filterShape, ~filterShapeControl.asMap);

	});

	// Handle note-off events
	off = MIDIFunc.noteOff({
		arg vel, note, chan, src;
		notes[note].release;
	});

	// Handle CC events
	cc = MIDIFunc.cc({
		arg val, num, chan, src;

		switch (num,
			1,  { ~scrubControl.set(linlin(val, 0, 127, 0.0, 1.0)); postln("scrub: " + val); },
			76, { ~scrubInertiaControl.set(linlin(val, 0, 127, 0.01, 5.0)); postln("scrub inertia: " + val ); },
			74, { ~grainPitchDispersionControl.set(linlin(val, 0, 127, 0.0, 0.5)); postln("grain pitch dispersion: " + val); },
			71, { ~pvParam1Control.set(linlin(val, 0, 127, 0.0, 32)); postln("pv param 1: " + val); },
			77, { ~pvParam2Control.set(linlin(val, 0, 127, 0.0, 0.99)); postln("pv param 2: " + val); }
		);
		//num.postln;
	});

	// Handle pitch-bend events

	// TODO

}.value;

)

If the code is correct, perhaps it’s a SuperCollider bug. I’m using v3.11.2 on macOS 10.15.6.

It feels a bit like there may be an issue with the RecordBuf and PV_RecordBug uGens. Maybe files are being opened for writing, the write fails, for some reason, and the files are not properly closed again.

It could be that the RT server crashes with the synth are related to attempting to read from corrupt analysis files (though loading the files into the “b” and “p” buffers doesn’t report errors).

“It fails” could mean a lot of things… “[you] mentioned” that it fails but I really have no idea what “it” means.

I’d like to see a really concrete example of failure vs success. If the pitch analysis file is corrupt, how is it corrupt? (I’m skeptical of this – Tartini should be deterministic – if it isn’t deterministic for you, it would be helpful to present evidence). Or, is the analysis file fine, and it’s a playback issue?

As for playback with the pitch shifting – I’d suggest to solve one problem at a time: first, a very simple SynthDef using the pitch data, and only after this is working, integrate it into the larger context. You’ll be tempted to go right to the full patch, but there are a lot of moving parts. If you’re not sure how the pitch part will work yet, it would be better to step back temporarily.

hjh

Sorry, I’m being wooly again. When I say “it fails” I meant in the way mentioned in a previous message - the analysis files don’t seem to be written properly. In these cases, the FFT analysis file previews in Finder as silent except for a loud click at the beginning, and won’t open in Audacity, though I think (from memory) Get Info in the finder does report the expected file-type, sample-rate and length for the file.

In fact, I’ve had instances of the data from the FFT analysis buffer being written into the pitch file, as well as the FFT file being apparently corrupt.

With the exception of the rare occasions (maybe only once) I’ve seen this happen, the pitch-analysis data actually seems to be generated and written properly.

The pitch-tracking uGens itself seems to work as intended.

If I repeatedly run the analysis function on the same audio, sometimes it will work as intended, and sometimes not.

I’ve found that commenting-out the lines in the NRT server definition related to pitch-analysis seems to reduce these errors in writing the FFT data, so thew problem appears to be related to the two operations happening at the same to (though as with all intermittent issues, it’s hard to say if removing the pitch analysis eliminated them altogether).

I’m at work at the moment, and on my work machine (a slightly higher-spec iMac), I can’t get the same glitch to occur. Analysis and file-writing seems to work fine, now, with the NRT server defined as above…

See above. But playback does seem to crash the SC server sometimes.

In testing, I usually run each code block in turn, from the top, but I haven’t always been checking that the analysis files have been correctly written, so it’s possible that the crashes occur when the files have not been correctly written.

This is very good advice! The time-domain pitch-shifting using the PitchShift uGen is OK for downward-shifting, but predictably not so great for pitching up, when it goes metal chipmunk after a few semitones.