iFFT Freeze/Scrub

It’s certainly possible.

The analysis example creates one resultBuf – there’s nothing stopping you from creating a second resultBuf for a control-rate pitch detector result. What is the size of this buffer? Control rate is one value per blockSize audio samples, so:

inbuf = Buffer(z, 65536, 1);
resultbuf = Buffer(z, f.duration.calcPVRecSize(fftSize, 0.5, z.options.sampleRate), 1);
pitchbuf = Buffer(z, (f.numFrames / z.options.blockSize).roundUp.asInteger, 2);

(2 channels because pitch detectors usually output a confidence value along with the frequency.)

Then add a second analysis and second buffer recorder into the analysis SynthDef.

SynthDef(\pv_ana, {
	var sig = VDiskIn.ar(1, inbuf, f.sampleRate / SampleRate.ir);
	var fft = FFT(LocalBuf(fftSize, 1), sig);
	var pitch = Tartini.kr(sig);
	RecordBuf.kr(pitch, pitchbuf, loop: 0);
	fft = PV_RecordBuf(fft, resultbuf, run: 1);
	Out.ar(0, sig);
})

And where it writes the FFT analysis buffer, add a second command to write the pitch buffer.

	[f.duration + (fftSize / z.options.sampleRate),
		resultbuf.writeMsg(q, "wav", "float")
	],
	[f.duration + (fftSize / z.options.sampleRate),
		pitchbuf.writeMsg(q.dirname +/+ "pitchbuf.wav", "wav", "float")
	]

One thing I would note here is that – there isn’t much new in the above advice. The example already shows how to create, fill and write an analysis buffer – and the answer to the question is just to do that with two buffers.

Or, as I’ve been preaching lately: solve one problem at a time. If you want to integrate pitch analysis into a NRT analysis code snippet, then… the integration can’t happen without having something to integrate (that is, without working out the pitch analysis by itself first – i.e., generating the pitch analysis signals, and then storing them in a buffer).

Quite often, I see people trying to handle all of the requirements in one go. That’s a good recipe to get confused.

hjh

2 Likes

That’s cool!

There are specific uGens for FFT data, but I wasn’t sure if it was possible to save arbitrary data to file, and read it back. It seems it is, which might help me with a future project, too, actually!

The pitch data file seems to be mono, rather than stereo, despite this line (I assume) creating a 2-channel buffer

pitchBuf = Buffer(z, (f.numFrames / nrtServer.options.blockSize).roundUp.asInteger, 2);

I’m not sure if the data recorded to it its actually the pitch, or confidence values. I’ll need to find a source with a more obvious pitch, I think, to find out.

Hm, did you also b_alloc the buffer? (I forgot to spell that out explicitly… occupational hazard of helping out – every error or omission comes back to bite you :laughing: )

If the buffer object is created with 2 channels, then its allocMsg should specify 2 channels to the server, and if the server-side buffer is allocated with 2 channels, then it should write to disk with 2 channels. If you get only one channel, I’d have to suspect an allocation problem first.

See the NRT guide helpfile… the initial code I posted is adapted from it. It’s using a buffer to store and write onset times for audio events in a soundfile – which is pretty arbitrary as data go.

hjh

Ah no, I didn’t. Noob here, I’m afraid.

Does this look good?

(

///////////////////////////////////////
// Non-Realtime FFT Analysis To File //
///////////////////////////////////////

~analyseWav = {
	arg wavIn, fftOut, pitchOut;

	// Define function vars
	var inBuf, fftBuf, pitchBuf, fftSize = ~fftSize, f, nrtServer, nrtScore, env;

	f = SoundFile.openRead(wavIn);
	f.close;
	f.duration;

	// Define Non-RealTime server
	nrtServer = Server(\nrt, NetAddr("127.0.0.1", 57110),
		ServerOptions.new
		.numOutputBusChannels_(2)
		.numInputBusChannels_(2)
		.sampleRate_(44100)
	);

	// Create buffer for input file
	inBuf = Buffer(nrtServer, 65536, 1);

	// Create output buffer for FFT analysis data
	fftBuf = Buffer(nrtServer, f.duration.calcPVRecSize(fftSize, 0.5, nrtServer.options.sampleRate), 1);
	pitchBuf = Buffer(nrtServer, (f.numFrames / nrtServer.options.blockSize).roundUp.asInteger, 2);

	// Create score for recording of FFT analysis to WAV file
	nrtScore = Score([
		[0, inBuf.allocMsg],
		[0, fftBuf.allocMsg],
		[0, pitchBuf.allocMsg],
		[0, inBuf.readMsg(wavIn, leaveOpen: true)],
		[0, [\d_recv, SynthDef(\pv_ana, {
			var sig = VDiskIn.ar(1, inBuf, f.sampleRate / SampleRate.ir);
			var fft = FFT(LocalBuf(fftSize, 1), sig);
			var pitch = Tartini.kr(sig);
			RecordBuf.kr(pitch, pitchBuf, loop: 0);
			fft = PV_RecordBuf(fft, fftBuf, run: 1);
			Out.ar(0, sig);
		}).asBytes]],
		[0, Synth.basicNew(\pv_ana, nrtServer).newMsg],
		[f.duration + (fftSize / nrtServer.options.sampleRate),
			fftBuf.writeMsg(fftOut, "wav", "float");
		],
		[f.duration + (fftSize / nrtServer.options.sampleRate),
			pitchBuf.writeMsg(pitchOut, "wav", "float");
		]

	]);

	// Run score
	nrtScore.recordNRT(
		outputFilePath: if(thisProcess.platform.name == \windows) { "NUL" } { "/dev/null" },
		headerFormat: "wav",
		sampleRate: nrtServer.options.sampleRate,
		options: nrtServer.options,
		duration: f.duration + (fftSize / nrtServer.options.sampleRate),
		action: { "done".postln }
	);

	// Free score
	nrtScore.free;

	// Free NRT server
	nrtServer.remove;

	inBuf.free;
	fftBuf.free;
	pitchBuf.free;
}

)

(

/////////////////
// Global Vars //
/////////////////

~fftSize = 2048;

// Default soundfile to be analysed
~wavPath = Platform.resourceDir +/+ "sounds/a11wlk01.wav";

// Default location for analysis file
~fftPath = "~/fft.wav".standardizePath;

~pitchPath = "~/pitch.wav".standardizePath;

~analyseWav.value(~wavPath, ~fftPath, ~pitchPath);

)

It certainly produces a stereo pitch output file, now. I can’t be sure it’s accurate, but then I imagine the analysis algorithm is going to struggle to extract meaningful pitch from a11wlk01.wav.

It doesn’t seem to work reliably at all though, unfortunately. More often than not, it produces an analysis file that’s corrupt or empty, apart from a loud click at the beginning.

I just tried it with a solo flute recording:

~analyseWav.value(~wavPath, ~fftPath, ~pitchPath);  // with my file

f = SoundFile.openRead(~pitchPath);
[f.numFrames, f.numChannels]  // OK

d = Signal.newClear(f.numFrames * f.numChannels);
f.readData(d);
f.close;

e = d.as(Array);
p = e.plot(numChannels: 2);

p.specs_([ControlSpec(0, 2500), ControlSpec(0, 1)]).refresh;

This much is as expected – it is tracking the melody’s contour.

Your scrubbing synth should be responsible for avoiding unreasonable values for shifting, then… (which can be tricky).

hjh

I should clarify: when I say it’s been unreliable, I mean if I repeatedly attempt to analyse the same file, most of the time it fails, as I mentioned, and occasionally it appears to work.

It seems to work more reliably when I comment-out all the lines related to the pitch analysis.

I’ve also found that often (but not all the time) when I trigger a note to instantiate a synth, the server immediately quits, with status 0 (which isn’t very helpful).

Here’s the synth (note I’m not doing anything with the pitch analysis data yet):

(

b = Buffer.read(s, ~fftPath);
p = Buffer.read(s, ~pitchPath);

//////////////////
// Define Synth //
//////////////////

SynthDef(\pvscrub, {
	arg out = 0, fBuf = 1,
	pOffset = 0, pBend = 0,
	amp = 0.1, gate = 1,
	ampEnvAttack = 0.01, ampEnvDecay = 0, ampEnvSustain = 1.0, ampEnvRelease = 2,
	grainSize = 0.1, grainPitchDispersion = 0.01, grainTimeDispersion = 0.004,
	//filterCutoff, filterResonance, filterShape,
	scrubPos = 0.1, scrubPosInertia = 0.01,
	pvParam1 = 0, pvParam2 = 0;

	var fftSize = ~fftSize, chain, bufNum, ampEnv, centerFreq, result;

	// Amplitude envelope Attack Sustain Release type
	// Free synth after envelope finishes
	ampEnv = Env.adsr(ampEnvAttack, ampEnvDecay, ampEnvSustain, ampEnvRelease, amp).kr(2, gate);

	// Read FFT buffer
 	bufNum = LocalBuf.new(fftSize);
	chain = PV_BufRd(bufNum, fBuf, min(scrubPos.lag(scrubPosInertia), 1));

	// FFT FX
	chain = PV_RectComb(chain, pvParam1.lag(1), 0);
	chain = PV_MagSmooth(chain, pvParam2.lag(1));

	// TODO

	// Resynthesise FFT
	result = IFFT(chain).dup;

	// Time-domain pitch-shift
	result = PitchShift.ar(
        result,
        grainSize,
		pOffset,   // Pitch-shift ratio
		grainPitchDispersion.lag(2),
        grainTimeDispersion.lag(2)
    );

	// Filter

	// TODO

	// Resonant low-pass

	// Non-resonant hi-Pass/shelf

	// Final output
	Out.ar(out, result * ampEnv);
}).add;

)

And here’s the MIDI setup

(

//////////////////
// Setup Busses //
//////////////////

~setupControls = {
	// Setup control buses
	~scrubControl = Bus.control(s, 1);
	~scrubInertiaControl = Bus.control(s, 1);
	~bendControl = Bus.control(s, 1);
	~grainPitchDispersionControl = Bus.control(s, 1);
	~pvParam1Control = Bus.control(s, 1);
	~pvParam2Control = Bus.control(s, 1);
	//~filterCutoffControl = Bus.control(s, 1);
	//~filterResonanceControl = Bus.control(s, 1);
	//~filterShapeControl = Bus.control(s, 1);
}.value;

////////////////
// Setup MIDI //
////////////////

~setupMIDI = {
	var notes, on, off, cc, centreNote = 60;

	MIDIClient.init;
	MIDIIn.connectAll;


	notes = Array.newClear(128);    // array has one slot per possible MIDI note

	// Handle note-on events
	on = MIDIFunc.noteOn({
		arg vel, note, chan, src;

		var noteScaled, pOffsetRatio;

		// Note clamped to 48..72 range
		// 2 octaves either side of C4, -24..+24 semitone range
		noteScaled = max(min(note, centreNote + 24), centreNote - 24) - centreNote;

		// Pitch-offset as ratio
		pOffsetRatio = noteScaled.midiratio;

		// Push synth to notes array
		notes[note] = Synth(\pvscrub, [\out: 0,\fBuf: b,\pOffset: pOffsetRatio,\amp: vel.linlin(0, 127, 0, 1)]);
		// Setup control busses
		notes[note].set(\scrubPos, ~scrubControl.asMap);
		notes[note].set(\scrubPosInertia, ~scrubInertiaControl.asMap);
		notes[note].set(\pBend, ~bendControl.asMap);
		notes[note].set(\grainPitchDispersion, ~grainPitchDispersionControl.asMap);
		notes[note].set(\pvParam1, ~pvParam1Control.asMap);
		notes[note].set(\pvParam2, ~pvParam2Control.asMap);
		//notes[note].set(\filterCutoff, ~filterCutoffControl.asMap);
		//notes[note].set(\filterResonance, ~filterResonanceControl.asMap);
		//notes[note].set(\filterShape, ~filterShapeControl.asMap);

	});

	// Handle note-off events
	off = MIDIFunc.noteOff({
		arg vel, note, chan, src;
		notes[note].release;
	});

	// Handle CC events
	cc = MIDIFunc.cc({
		arg val, num, chan, src;

		switch (num,
			1,  { ~scrubControl.set(linlin(val, 0, 127, 0.0, 1.0)); postln("scrub: " + val); },
			76, { ~scrubInertiaControl.set(linlin(val, 0, 127, 0.01, 5.0)); postln("scrub inertia: " + val ); },
			74, { ~grainPitchDispersionControl.set(linlin(val, 0, 127, 0.0, 0.5)); postln("grain pitch dispersion: " + val); },
			71, { ~pvParam1Control.set(linlin(val, 0, 127, 0.0, 32)); postln("pv param 1: " + val); },
			77, { ~pvParam2Control.set(linlin(val, 0, 127, 0.0, 0.99)); postln("pv param 2: " + val); }
		);
		//num.postln;
	});

	// Handle pitch-bend events

	// TODO

}.value;

)

If the code is correct, perhaps it’s a SuperCollider bug. I’m using v3.11.2 on macOS 10.15.6.

It feels a bit like there may be an issue with the RecordBuf and PV_RecordBug uGens. Maybe files are being opened for writing, the write fails, for some reason, and the files are not properly closed again.

It could be that the RT server crashes with the synth are related to attempting to read from corrupt analysis files (though loading the files into the “b” and “p” buffers doesn’t report errors).

“It fails” could mean a lot of things… “[you] mentioned” that it fails but I really have no idea what “it” means.

I’d like to see a really concrete example of failure vs success. If the pitch analysis file is corrupt, how is it corrupt? (I’m skeptical of this – Tartini should be deterministic – if it isn’t deterministic for you, it would be helpful to present evidence). Or, is the analysis file fine, and it’s a playback issue?

As for playback with the pitch shifting – I’d suggest to solve one problem at a time: first, a very simple SynthDef using the pitch data, and only after this is working, integrate it into the larger context. You’ll be tempted to go right to the full patch, but there are a lot of moving parts. If you’re not sure how the pitch part will work yet, it would be better to step back temporarily.

hjh

Sorry, I’m being wooly again. When I say “it fails” I meant in the way mentioned in a previous message - the analysis files don’t seem to be written properly. In these cases, the FFT analysis file previews in Finder as silent except for a loud click at the beginning, and won’t open in Audacity, though I think (from memory) Get Info in the finder does report the expected file-type, sample-rate and length for the file.

In fact, I’ve had instances of the data from the FFT analysis buffer being written into the pitch file, as well as the FFT file being apparently corrupt.

With the exception of the rare occasions (maybe only once) I’ve seen this happen, the pitch-analysis data actually seems to be generated and written properly.

The pitch-tracking uGens itself seems to work as intended.

If I repeatedly run the analysis function on the same audio, sometimes it will work as intended, and sometimes not.

I’ve found that commenting-out the lines in the NRT server definition related to pitch-analysis seems to reduce these errors in writing the FFT data, so thew problem appears to be related to the two operations happening at the same to (though as with all intermittent issues, it’s hard to say if removing the pitch analysis eliminated them altogether).

I’m at work at the moment, and on my work machine (a slightly higher-spec iMac), I can’t get the same glitch to occur. Analysis and file-writing seems to work fine, now, with the NRT server defined as above…

See above. But playback does seem to crash the SC server sometimes.

In testing, I usually run each code block in turn, from the top, but I haven’t always been checking that the analysis files have been correctly written, so it’s possible that the crashes occur when the files have not been correctly written.

This is very good advice! The time-domain pitch-shifting using the PitchShift uGen is OK for downward-shifting, but predictably not so great for pitching up, when it goes metal chipmunk after a few semitones.

Ok… Since you have control over the code generating these wav files, then you know the meaning of the contents.

So first question to think about is: Do these files contain audio waveforms or not?

They don’t. One contains a series of FFT frames. The other contains one channel of frequencies in Hz and one channel of confidence values.

Now. How does the Finder preview WAV files?

It plays the contents as audio.

But we know that the files don’t contain audio waveforms.

Therefore, it is an invalid test to play these files in the Finder.

That’s bizarre. I can’t see anything in the analysis function that would cause this… have to think about it some more.

Maybe print out all the messages in the score and double check for the right buffer numbers everywhere. nrtScore.score.do(_.postln)

hjh

I realise that. But equally, the Finder treats them as audio files, and SC writes them as such, so the fact Finder properly reports audio file properties associated with audio files (presumably encoded in the file header) is significant.

It means SC has at least managed to open the file and begin writing to it.

The fact I’m able to hear some audio when previewing the FFT analysis file as audio in Finder following the analysis sometimes is also significant, when re-running the analysis on the same source file on other occasions produces an output file that previews as silent (save for the click at the beginning).

The file produced after the second analysis should be identical to the first.

Thought the FFT data is clearly audible, the pitch file is mostly too low-frequency to hear. I’m not playing these files through studio speakers, incidentally. I do value my hearing…

I do appreciate your patience. I work in (sort of) tech-support, so my lack of precision is exactly the kind of thing I complain about in others :wink:

OK, I understand… FWIW I posted a code snippet above to read the pitch file within SC and plot it. I would trust this sooner than I would trust the Finder or Audacity, for non-audio data.

So, if I could summarize the problem so far: You’re running deterministic code, and getting a nondeterministic result.

… and, I’m being persistent about this issue because, of course, deterministic code should produce deterministic results, and if it doesn’t, that shakes one’s confidence.

The catch is… I’m unable to reproduce the faulty results.

I made a slight change to your ~analyseWav function, to run an action upon completion. Then I could run it repeatedly – greater chance of observing the wrong behavior. You can find this in a gist, FFT + pitch analysis testing code · GitHub .

You can run it two ways: 1. As is (each loop iteration writes into different files, e.g. fft00.wav, fft01.wav…). Or 2. Modify the third block of code to take out the index – writing to the same two paths each time, that is, overwriting old results.

		var fpath = ~fftPath; // .splitext[0] ++ i.asPaddedString(2) ++ ".wav";
		var ppath = ~pitchPath; // .splitext[0] ++ i.asPaddedString(2) ++ ".wav";

After each test, copy/paste the post window contents into an editor and search for the word “corrupt.”

I’ve run it multiple times, both ways (separate result files, or overwriting the same 2 result files), and I never see any deviation in the results at all. (Curiously, the Unix cmp utility finds slight differences in the file headers – which is why I went with reading and comparing the contents in SC.)

I posted the code to gist so that you could try it yourself – i.e., please try it yourself, since I can’t reproduce the problem.

One good thing about the multi-file version is – any iterations that fail, the files remain on disk for further investigation. (But, if the issue doesn’t reproduce in the multi-file version and it does happen with the overwriting version, that would suggest maybe the old file isn’t being properly deleted.)

(There is one other thing I could try… I’m running from the development branch. I’ll go ahead and build 3.11.2. Maybe there’s an obscure bug in 3.11.x? This seems unlikely, but… if /b_write is not trustworthy, that’s pretty serious. It’s worth the effort to be sure. EDIT: Tried the Version-3.11.2 tag – no difference, everything is fine on my machine. Maybe a Mac-specific problem, then.)

I used to work in tech support. From the above, can you tell? :wink:

hjh

I have a feeling it may be a problem with this specific iMac at home, given the same script was working much more reliably on my work machine yesterday.

I saw another file-related issue in Finder last night that suggests there may be a file-system problem on this machine.

I’ll try that snippet as soon as I get a chance, though.

I ran your gist snippet. Now I can reproduce the problem!

The plots for both analysis file type look like this:

Pitch

FFT:

I ran your loop to analyse the input audio 30x.

I scoped all 30 fit/pitch files using this little loop

(

~wavPath = "~/Desktop/Norns_Coding/PVScrub/Audio/miao-edit.wav".standardizePath;
~fftPath = "~/fft.wav".standardizePath;
~pitchPath = "~/pitch.wav".standardizePath;


30.do { |i|
	
	var fpath, f, d, e, p;
	
	fpath = ~fftPath.splitext[0] ++ i.asPaddedString(2) ++ ".wav";
    //fpath = ~pitchPath.splitext[0] ++ i.asPaddedString(2) ++ ".wav";
	
	f = SoundFile.openRead(fpath);
	
	d = Signal.newClear(f.numFrames * f.numChannels);
	f.readData(d);
	f.close;
	
	e = d.as(Array);
	p = e.plot(numChannels: 2);
	
	p.specs_([ControlSpec(0, 2500), ControlSpec(0, 1)]).refresh;
	
};

)

No reports of corrupt files when running the analysis loop.

No errors reported, one warning related to the FFT analysis, I assume:

Channel mismatch. File ‘/Users/mcadmin/Desktop/Norns_Coding/PVScrub/Audio/miao-edit.wav’ has 2 channels. Buffer has 1 channels.

It’s consistently failing to write either file, now, even after quitting and restarting SC.

I’m going to restart the machine, and try again.

Same, after a restart. It seems not to be able to write the files at all now.

These errors seem to show up in the System log every time I run the analysis function:

|error|11:41:33.324873+0100|kernel|Sandbox: QuickLookUIServi(486) deny(1) file-issue-extension target:/Users/mcadmin/Desktop/Norns_Coding/PVScrub/Analysis class:com.apple.app-sandbox.read|
|---|---|---|---|
|error|11:41:33.325141+0100|QuickLookUIService|Could not create FPSandboxingURLWrapper to parent directory of url: <private>, error: Error Domain=NSPOSIXErrorDomain Code=1 UserInfo={NSDescription=<private>}|
|error|11:41:33.368253+0100|QuickLookUIService|CGImageCreate: invalid floating-point bits/component: 8.|
|error|11:41:33.368300+0100|QuickLookUIService|CGImageCreate: invalid floating-point bits/component: 8.|
|error|11:41:33.368330+0100|QuickLookUIService|CGImageCreate: invalid floating-point bits/component: 8.|
|error|11:41:33.368354+0100|QuickLookUIService|CGImageCreate: invalid floating-point bits/component: 8.|

I have a feeling the “CGImageCreate” errors are QuickLookUIService failing while attempting to build a waveform preview of the file.

I’m pretty sure, based on the above, and an existing suspicion, that this is related to sandboxing and file-access control, but that it’s an error/bug, rather than intended behaviour (macOS blocks applications from having write access to certain file locations by default, but I’ve granted SuperCollider “Full Disk Access”, which should allow it to write anywhere).

I’m not running the latest version of Catalina on this machine. I’m going to try updating to the current version, and see if that helps at all.

Hmm… SuperCollider consists of 3 executables (scide, sclang and scsynth). Typically “launching SuperCollider” means launching scide. But here, it’s scsynth that’s writing the files. So I wonder if maybe scsynth is still running under the reduced permissions?

hjh

There was clearly something very wrong with my Mac, so I ended up doing a clean-install of the current macOS, Big Sur.

It’s going to take me a while to get everything up and running again.

@jamshark70 thanks for these examples!

Is there any reason for using this 65536 value for the number of frames?

Using the fft size value would not be enough ?

The key is, what is inBuf used for?

var sig = VDiskIn.ar(1, inBuf, f.sampleRate / SampleRate.ir);

VDiskIn help: “NOTE: The Buffer’s numFrames must be a power of two and is recommended to be at least 65536 – preferably 131072 or 262144. Smaller buffer sizes mean more frequent disk access, which can cause glitches.”

In NRT, glitches shouldn’t be an issue, but I don’t see a compelling reason to have a very small buffer here.

hjh

1 Like

@jamshark70 for a situation where you need to analyze lots of features (pitch, onsets, amplitude, centroid, spread…) in NRT and also pass different arguments (Amplitude.ar(attackTime=0.01), Amplitude.ar(attackTime=0.05), …) the advise structure is to have one analysis synth per NRT score (and also one ~analyseWav function per audio feature) ?

The .asBytes method does not allow one to use multiple NRT SynthDefs stored on disk ?

I would run them concurrently, in the same NRT server instance. An NRT server can naturally run as many nodes as you need, based on as many synthdefs as you need, writing into as many buffers as you need.

Also – I’m not sure exactly how to discuss the etiquette here, but I have a feeling sometimes that “at-jamshark70” in a post conveys a couple of ideas: 1/ I feel like there’s an expectation that I, individually, should take the main responsibility to answer, and 2/ others may feel like, since jamshark70 is supposed to answer, then they would just wait for me rather than contributing their own ideas.

I wrote up the FFT analysis prototype as an example, but I don’t want to be considered the “owner” of this code. It’s not part of my normal working method – questions about it are low-priority for me. I think, if others are interested in extending this example to do more kinds of work, or more analyses at the same time – please do! But I’m afraid I have other things on my plate.

No, asBytes doesn’t interfere with synthdefs on disk (unless there’s a name collision).

hjh

1 Like