Frequency warping advice

Questions
1

Hello guys,
I am having a hard time trying to code frequency warping, similarly to what happens with the GRM tools ‘Warp’ plugin. I am looking at this paper here → link which does the DSP with Csound and a couple of tables. I cannot seem to be able to access the FFT data and apply a transform (via an Env.asArray, for instance) to the [mags, phases] spectral data that I get via UnpackFFT or pvcollect/pvcalc. The process seems quite straightforward, simply moving mags and phases to different bin(s) depending on the transform.
Is there any PV Ugen I am not aware of that can apply such a transform in an efficient way? Or am I stuck trying to get my head around pvcollect/pvcalc?

Thank you!
Stefano

3

Maybe you could use PV_MagMap and use a MultiSliderView to write into the buffer instead of passing an envelope.

4

Hi,
I considered that, but unfortunately PV_MagMap does not shift the phase data as well, only the magnitudes. A “PV_PhsMap” would be awesome.
I’m currently fighting against pvcollect, and I’ll post something here in case I manage to win.

Stefano

5

So this is what I got so far, surely an unelegant solution for now:

(

var fftsize = 1024;
s.newBufferAllocators;
e = Array.fill(fftsize/2, //just making an array of values representing my transform
	{
		|i|
		var n = 0.5; //at 1 we have original version
		//then warping happens going above or below that
		//with n > 1 everything is shifted in high frequencies very rapidly
		i ** n

});
e = e.normalize(0, (fftsize/2)).asInteger;
e.postln;
e.plot;

z = Buffer.read(s, Platform.resourceDir +/+ "sounds/a11wlk01.wav");
{

	var in, chain, magsphases;
	
	in = PlayBuf.ar(1, z, BufRateScale.kr(z), loop: 1);
	chain = FFT(LocalBuf(fftsize), in);
	
	magsphases = UnpackFFT(chain, fftsize, 0, fftsize);
	//here I collect all the magnitudes and phases of the original FFT
	 
	chain = chain.pvcollect(fftsize, { //looping through the bins
		|mags, phs, bin, index|
		var warpIndexTable = e[index]; //the corresponding bin of my transform
		
		mags = magsphases[(warpIndexTable*2)]; //replace magnitude output with the transformed one
		phs = magsphases[(warpIndexTable*2)+1];
		//replace phase output with the transformed one
		[mags, phs]
	}, 0, (fftsize/2) - 1, 1
	);

	IFFT(chain).dup
}.play
)

It does work and it sounds quite good at certain n values, e.g. 0.5 < n < 1, but it has a few issues. It needs amplitude compensation for duplicated bins, since it loses quite a bit of spectral power if the transform changes the sound radically, and I’m unsure how to make it a real-time process. From what I gather from the helpfile, pvcollect only runs once at startup (making the Ugen graph?) and it does not calculate new values unless I start it back again.

Any ideas on how to make this better?

Best,
Stefano

6

I slightly tweaked your code by: using Env to create the transform curve, using a larger FFT size to create more bins and only transforming the lower half of the spectrum.
Some interesting sounds to be found this way!
Hope that helps.
Best,
Paul

(
// var fftsize = 1024;
var fftsize = 2048;  // more FFT bins might sound better?
// var numFXBins = fftsize/2;
var numFXBins = fftsize/4;  // transform just lower half of spectrum

// e = Env([0, 1], [1], -1).discretize(numFXBins).as(Array);
// e = Env([0, 1], [1], -2).discretize(numFXBins).as(Array);
e = Env([0, 0.5, 1], [0.5, 0.5], [-2, 2]).discretize(numFXBins).as(Array);
// e = Env([0, 0.5, 1], [0.4, 0.6], [-3, 3]).discretize(numFXBins).as(Array);
// e = Env([0, 0.5, 1], [0.4, 0.6], [-1, 1]).discretize(numFXBins).as(Array);
// e = Env([0, 0.7, 0.2, 0.8, 0.6, 1], [0.2, 0.2, 0.2, 0.2, 0.2], 'sin').discretize(numFXBins).as(Array);

e = (e * numFXBins).asInteger; // turn into indices
//e.postln;
e.plot;

z = Buffer.read(s, Platform.resourceDir +/+ "sounds/a11wlk01.wav");
x = {

	var in, chain, magsphases;
	
	in = PlayBuf.ar(1, z, BufRateScale.kr(z), loop: 1);
	chain = FFT(LocalBuf(fftsize), in);
	
	magsphases = UnpackFFT(chain, fftsize, 0, fftsize);
	//here I collect all the magnitudes and phases of the original FFT
	 
	chain = chain.pvcollect(fftsize, { //looping through the bins
		|mags, phs, bin, index|
		var warpIndexTable = e[index]; //the corresponding bin of my transform
		
		mags = magsphases[(warpIndexTable*2)]; //replace magnitude output with the transformed one
		phs = magsphases[(warpIndexTable*2)+1];
		//replace phase output with the transformed one
		[mags, phs]
	}, 0, numFXBins - 1, 0
	);

	IFFT(chain).dup
}.play
)

x.free; z.free;
1 Like
7

Thanks Paul,
that surely made a noticeable difference in both sound and elegance of the code.
I still cannot figure out how to make any update to the array ‘e’ to immediately effect the process, without having to restart the whole thing: i.e. I want to make a real time GUI and visualisation out of this. I’m not sure how/if pvcollect can receive any argument or it is stuck in place once the synth starts.

Stefano

8

I’m no expert, but I suspect it might be that you have to rebuild the synth each time.
Maybe someone with deeper FFT knowledge knows? (e.g. @josh ?)

9

You’ll need to put the array into a Buffer, and then instead of e[index], use Index.kr(bufnum, index).

e[index] happens only when building the SynthDef and cannot be reevaluated in the server. If you want the server to update the values, it has to be a UGen. For the size of data you’re sending over, I think a buffer is better.

hjh

10

Hi James,
that’s where my issue arises: the array magsphases in the Synth necessitates an integer index, but Index.kr returns an Index object (so server side) rather than a Int number (language side).
This is where I hit a brick wall.

Stefano

11

Did you try Select for server-side indexing?

If you want the indexing to adapt to new data on the server side, then you’d have to use server side indexing, which is Select for an array, or Index for a buffer. (This will make a big SynthDef bigger, but if you use client-side indexing, there’s no way except to rebuild the synth.)

hjh

12

I did try Select to arrange the values, but the Synth takes literally more than a minute to build and then the interpreter stops working. With Index this happens as well. It’s probably best to do this on the UGen level and make one up from scratch?

Stefano

13

It may work for a smaller number of FFT bins (but that would compromise the sound). Yeah, I was a bit worried about it not scaling up – but didn’t expect it to be that slow.

If it’s too complex to do with pvcollect, then a custom UGen is probably the best way.

hjh

14

In case it’s useful, here’s another approach: instead of FFT, it uses band-pass filters for analysis and sines for re-synthesis. The frequency bins are spaced logarithmically (unlike FFT which is linear) so there are the same number of bins is each octave which I prefer for FX.
This way it uses less bins and the morph curve can be updated live. Still it uses quite a bit of CPU. This example includes a MultiSliderView for updating the curve:

(  // spectral morph without FFT
// init
d = ();  // data store
d.numFreqs = 60;
d.minFreq = 60;
d.maxFreq = 6000;
d.sampleBuf = Buffer.read(s, Platform.resourceDir +/+ "sounds/a11wlk01.wav");
d.outFreqBuf = Buffer.loadCollection(s,
	Env([0.0, 1.0], [1]).discretize(d.numFreqs).linexp(0, 1, d.minFreq, d.maxFreq),
	numChannels: d.numFreqs);
)

(
d.outSynth = SynthDef(\help_Buffer, { arg out = 0, sampleBuf, outFreqBuf;
	var in = PlayBuf.ar(1, sampleBuf, BufRateScale.kr(sampleBuf), loop: 1);
	var arrFreqs, arrBWRs, arrAmps, arrOutFreqs;
	var outSig;
	arrFreqs = Array.interpolation(d.numFreqs, 0, 1)
	.linexp(0, 1, d.minFreq, d.maxFreq);
	arrBWRs = d.numFreqs.collect({arg i;
		var val;
		if (i == 0, {
			val = abs(arrFreqs[1] - arrFreqs[0])/ arrFreqs[i];
		}, {
			if (i == (d.numFreqs - 1), {
				val = abs(arrFreqs[d.numFreqs - 1] - arrFreqs[d.numFreqs - 2]) / arrFreqs[i];
			}, {
				val = abs(arrFreqs[i + 1] - arrFreqs[i - 1]) * 0.5 / arrFreqs[i];
			});
		});
		val;
	});
	// analysis
	arrAmps =  Amplitude.kr(Resonz.ar(in, arrFreqs, bwr: arrBWRs),
		attackTime: 0.01, releaseTime: 0.01);

	// use Buffer or NamedControl
	arrOutFreqs = BufRd.kr(d.numFreqs, outFreqBuf, phase: 0);
	// arrOutFreqs = \outFreqs.kr(Array.interpolation(n, 0.0, 1.0)
	// 	.linexp(0.0, 1.0, d.minFreq, d.maxFreq));

	outSig = 0.1 * SinOsc.ar(arrOutFreqs, mul: arrAmps).sum;

	Out.ar( out, outSig ! 2)
}).play(s,[\sampleBuf, d.sampleBuf, \outFreqBuf, d.outFreqBuf]);

//gui to change outFreq buffer
(
d.sliderView = MultiSliderView(bounds: Rect(100, 100, 400, 300))
.size_(16).indexThumbSize_(28).elasticMode_(1).isFilled_(true)
.background_(Color(1, 1, 1, 0)).fillColor_(Color(0.8, 0.92, 1, 0.5))
.value_(Array.interpolation(d.numFreqs, 0, 1))
// .action_({arg view;
// 	d.outFreqBuf.loadCollection(view.value);
// })
.mouseUpAction_({arg view;
	d.outFreqBuf.loadCollection(view.value.linexp(0, 1, d.minFreq, d.maxFreq));
})
.front;
)

)
// change outFreq buffer
d.outFreqBuf.loadCollection(Env([0.0, 0.5, 1.0], [0.5, 0.5], [-1, 1])
	.discretize(d.numFreqs).linexp(0, 1, d.minFreq, d.maxFreq));

d.outFreqBuf.loadCollection(Env([0.0, 0.5, 1.0], [0.5, 0.5], [2, -2])
	.discretize(d.numFreqs).linexp(0, 1, d.minFreq, d.maxFreq));

d.outFreqBuf.loadCollection(Env([0.0, 0.5, 1.0], [0.5, 0.5], [3, -3])
	.discretize(d.numFreqs).linexp(0, 1, d.minFreq, d.maxFreq));

d.outFreqBuf.loadCollection(Env([0.0, 0.5, 1.0].reverse, [0.5, 0.5], [3, -3])
	.discretize(d.numFreqs).linexp(0, 1, d.minFreq, d.maxFreq));

// reset outFreq buffer
d.outFreqBuf.loadCollection(Env([0.0, 1.0],  [1])
	.discretize(d.numFreqs).linexp(0, 1, d.minFreq, d.maxFreq));

// cleanup
(
d.outSynth.free;
d.sampleBuf.free;   d.outFreqBuf.free;
)

Best,
Paul

1 Like
15

I happen to have a bit more time on my hands these days and kept looking at this. I’ve managed to use Index.kr without crashing the interpreter every time. However, even though I’m using a Buffer to store and update my values, I still cannot have the process dynamically update them.
This time I used a bit more efficient and flexible process with Unpack1FFT and PackFFT.

(
var fftsize = 1024;  
var numFXBins = fftsize/2;  // transform just lower half of spectrum
// e = Env([0, 1], [1], \lin).discretize(numFXBins).as(Array);
// e = Env([0, 1], [1], -1).discretize(numFXBins).as(Array);
 e = Env([0, 1], [1], -2).discretize(numFXBins).as(Array);
// e = Env([0, 0.5, 1], [0.5, 0.5], [-2, 2]).discretize(numFXBins).as(Array);
// e = Env([0, 0.5, 1], [0.4, 0.6], [-3, 3]).discretize(numFXBins).as(Array);
// e = Env([0, 0.5, 1], [0.4, 0.6], [-1, 1]).discretize(numFXBins).as(Array);
// e = Env([0, 0.7, 0.3, 0.45, 0.2, 1], [0.2, 0.2, 0.2, 0.2, 0.2], 'sqr').discretize(numFXBins).as(Array);
// e = Env([rrand(0.0, 1.0), rrand(0.0, 1.0), rrand(0.0, 1.0), rrand(0.0, 1.0), rrand(0.0, 1.0), rrand(0.0, 1.0)], [rrand(0.0, 1.0), rrand(0.0, 1.0), rrand(0.0, 1.0), rrand(0.0, 1.0), rrand(0.0, 1.0)], [rrand(-10, 10), rrand(-10, 10), rrand(-10, 10), rrand(0.0, 1.0), rrand(0.0, 1.0)]).discretize(numFXBins).as(Array);
// e = Env([0, 0.6, 0.3], [0.6, 0.4], \wel).discretize(numFXBins).as(Array);
e = (e * numFXBins).asInteger; // turn into indices
e.plot;
~warpBuffer = Buffer.loadCollection(s, e);
z = Buffer.read(s, Platform.resourceDir +/+ "sounds/a11wlk01.wav");

x = {
	|rate = 0.5|
	var in, chain, idx;
	var outArray = Array.fill(fftsize, {0});
	in = PlayBuf.ar(1, z, rate, loop: 1);
	chain = FFT(LocalBuf(fftsize), in);
	numFXBins.do({
		|item, i|
		outArray[i * 2] = Unpack1FFT(chain, fftsize, Index.kr(~warpBuffer, i), 0);
		outArray[i * 2 + 1] = Unpack1FFT(chain, fftsize, Index.kr(~warpBuffer, i), 1);
	});
	chain = PackFFT(chain, fftsize, outArray.flop.flatten, 0, 250);
	IFFT(chain).dup
}.play;
)

If i use the index i within the Index.kr Ugen then the process works as intended: static, but great sound. If I try and use a Ugen (I tried with Dbufrd, but no avail) the sound is distorted as if one very specific function is applied (harsh high pitched), and still no dynamic control over it.
Moreover, if I try to use a buffer as argument (e.g. play(s, [\bufWarp, ~warpBuffer]) and the corresponding argument within the Synth, no sound at all is present for unknown reasons.
Probably still best to go for a Ugen?

Paul, lovely code! It sounds great even with not that many frequency bands. With 256 its quite expensive but sounds amazing.

Thanks!
Stefano