Quick exploratory (very much noob) question: is it possible to “freeze” and arbitrarily “scrub” through a buffer of FFT data?
here’s a SynthDef copied from somewhere in the docs IIRC
SynthDef("pvrec", { |recBuf=1 soundBufnum=2 fftSize=2048 hop= 0.5 window=1|
var in, chain, bufnum;
bufnum = LocalBuf.new(fftSize);
Line.kr(1, 1, BufDur.kr(soundBufnum), doneAction: 2);
in = PlayBuf.ar(1, soundBufnum, 1, loop: 0);
chain = FFT(bufnum, in, hop, window);
chain = PV_RecordBuf(chain, recBuf, 0, 1, 0, hop, window);
// no ouput ... simply save the analysis to recBuf
}).add;
…and heres one that plays it back - you can fiddle with the rate (negative ok) -
SynthDef("pvplay", { | out=0, recBuf=1, rate=1 fftSize=2048 hop= 0.5 window = 1|
var in, chain, bufnum;
bufnum = LocalBuf.new(fftSize);
chain = PV_PlayBuf(bufnum, recBuf, rate, 0, 1, 1, 0.25, 1);
Out.ar(out, IFFT(chain, window).dup);
}).add;
1 Like
Great, thank you very much @semiquaver! I will give that a go.
Another quick question: is it possible to transpose the resynthesised FFT data to a pitch scale (ie up/down in semitones)?
None of the FFT uGens seem designed for this particular task to my untrained eye, based on their descriptions, but I may be missing something.
Ah, this one, I guess:
https://doc.sccode.org/Classes/PV_BinShift.html
But is it necessary to stretch/compress the bin locations, as well as offsetting them up/down?
I will experiment. Need to do some reading 
Is it possible to analyse an audio file to an FFT data buffer in faster than realtime (your example plays back audio and analyses it in realtime to the FFT buffer, as I understand it)?
One possibility: check out @scztt Offline Process Quark
(though now that I look I see a complication that the quark expects to produce audio rate file - not sure if side effects are produced! )
The output file is not necessarily audio rate - it writes to an audio file, but that’s only because the only output format NRT supports is audio files. If you e.g. specify a control rate process, you’ll get an audio file w/ control rate data - likewise with outputting triggers via putTr. The quark will read these audio files back into sclang as arrays, so you don’t need to interact with the on-disk audio files at all unless you explicitly need to.
1 Like
it was the language: “Offlice Process expects audio rate output Ugen” in the help that made me wonder - the OP is hoping to store a PV chain - possible?
(and something like this should really be included in SC imo)
Here’s an example of capturing an FFT chain:
(
~o = OfflineProcess();
~o.putTrig(\fft, {
|in|
var fft, magsPhases;
fft = FFT(LocalBuf(256), in);
magsPhases = UnpackFFT(fft, 256);
[fft, magsPhases];
});
fork {
~run = ~o.process(Platform.resourceDir +/+ "sounds/a11wlk01.wav");
~run.wait();
"DONE".postln;
~run.resultData(\fft).do(_.postln);
};
)
fft in your SynthDef is actually a trigger that is fired every time there’s a new FFT buffer available, so you can use it as the trigger for putTrig. One caveat here is that the above won’t naively work for FFT sizes larger than 256 - I think this might be a limitation of UnpackFFT? You may need to use multiple UnpackFFT's, or roll your own with an array of Unpack1FFT.
1 Like
I get
“Execution warning: Class ‘Deferred’ not found
ERROR: Message ‘new’ not understood.”
After compiling your Quark and trying that code
Deferred quark also on szctt github IIRC
The OfflineProcess looks interesting, but it wouldn’t result in a buffer that could be used with PV_PlayBuf. So, playing back may be difficult.
You can run a PV_RecordBuf synth in a NRT server.
p = Platform.resourceDir +/+ "sounds/a11wlk01.wav";
q = "~/pvtest.wav".standardizePath;
// get duration
f = SoundFile.openRead(p);
f.close;
f.duration // 4.2832879818594
(
var resultbuf, inbuf;
var fftSize = 512;
z = Server(\nrt, NetAddr("127.0.0.1", 57110),
ServerOptions.new
.numOutputBusChannels_(2)
.numInputBusChannels_(2)
.sampleRate_(44100)
);
inbuf = Buffer(z, 65536, 1);
resultbuf = Buffer(z, f.duration.calcPVRecSize(fftSize, 0.5, z.options.sampleRate), 1);
x = Score([
[0, inbuf.allocMsg],
[0, resultbuf.allocMsg],
[0, inbuf.readMsg(p, leaveOpen: true)],
[0, [\d_recv, SynthDef(\pv_ana, {
var sig = VDiskIn.ar(1, inbuf, f.sampleRate / SampleRate.ir);
var fft = FFT(LocalBuf(fftSize, 1), sig);
fft = PV_RecordBuf(fft, resultbuf, run: 1);
Out.ar(0, sig);
}).asBytes]],
[0, Synth.basicNew(\pv_ana, z).newMsg],
[f.duration + (fftSize / z.options.sampleRate),
resultbuf.writeMsg(q, "wav", "float")
]
]);
x.recordNRT(
outputFilePath: if(thisProcess.platform.name == \windows) { "NUL" } { "/dev/null" },
headerFormat: "wav", sampleRate: z.options.sampleRate,
options: z.options,
duration: f.duration + (fftSize / z.options.sampleRate),
action: { "done".postln }
);
z.remove;
)
And play it back:
s.boot;
// this is the pv_rec result
b = Buffer.read(s, q);
(
SynthDef(\pvmouse, { |out = 0, recBuf = 1, fftSize = 512|
var in, chain, bufnum;
bufnum = LocalBuf.new(fftSize);
chain = PV_BufRd(bufnum, recBuf, MouseX.kr(0, 1));
Out.ar(out, IFFT(chain).dup);
}).add;
)
a = Synth(\pvmouse, [recBuf: b]);
a.free;
hjh
3 Likes
This got over my head quite quickly. Have to read up on some basics, I think. Good to now there are solutions, though. Thanks very much guys!
My NRT example is mostly based on an example in the help: Non-Realtime Synthesis (NRT) | SuperCollider 3.11.1 Help
NRT is very different from normal real-time usage. It does make sense but it takes some time to understand the differences. The linked help file tries to walk you through them – if you expect to want to do more of this type of processing, it’s worth it to work your way through it.
hjh
but … I ran James’ code and …so fun to scrub so thanks to you both!
1 Like
Hi @semiquaver do you have the exact code you used to hand?
Don’t worry, got it working. And it’s a LOT of fun!!
Thank you very much, guys!!!
Hi @jamshark70 I’m really enjoying developing my PV scrubbing script. Thanks so much for your example!
I wonder if it would be possible to analyse the pitch of the original audio, and save that to a separate file, in the same way as the FFT analysis, and to load that back into a buffer and scrub through it at the same time as the FFT data.
With a known fundamental frequency, I could do frequency-domain pitch-shifting/flattening, which would be really cool!
Do you think that’s possible?
It’s certainly possible.
The analysis example creates one resultBuf – there’s nothing stopping you from creating a second resultBuf for a control-rate pitch detector result. What is the size of this buffer? Control rate is one value per blockSize audio samples, so:
inbuf = Buffer(z, 65536, 1);
resultbuf = Buffer(z, f.duration.calcPVRecSize(fftSize, 0.5, z.options.sampleRate), 1);
pitchbuf = Buffer(z, (f.numFrames / z.options.blockSize).roundUp.asInteger, 2);
(2 channels because pitch detectors usually output a confidence value along with the frequency.)
Then add a second analysis and second buffer recorder into the analysis SynthDef.
SynthDef(\pv_ana, {
var sig = VDiskIn.ar(1, inbuf, f.sampleRate / SampleRate.ir);
var fft = FFT(LocalBuf(fftSize, 1), sig);
var pitch = Tartini.kr(sig);
RecordBuf.kr(pitch, pitchbuf, loop: 0);
fft = PV_RecordBuf(fft, resultbuf, run: 1);
Out.ar(0, sig);
})
And where it writes the FFT analysis buffer, add a second command to write the pitch buffer.
[f.duration + (fftSize / z.options.sampleRate),
resultbuf.writeMsg(q, "wav", "float")
],
[f.duration + (fftSize / z.options.sampleRate),
pitchbuf.writeMsg(q.dirname +/+ "pitchbuf.wav", "wav", "float")
]
One thing I would note here is that – there isn’t much new in the above advice. The example already shows how to create, fill and write an analysis buffer – and the answer to the question is just to do that with two buffers.
Or, as I’ve been preaching lately: solve one problem at a time. If you want to integrate pitch analysis into a NRT analysis code snippet, then… the integration can’t happen without having something to integrate (that is, without working out the pitch analysis by itself first – i.e., generating the pitch analysis signals, and then storing them in a buffer).
Quite often, I see people trying to handle all of the requirements in one go. That’s a good recipe to get confused.
hjh
3 Likes