Maximum value for numwireBufs

Hello,

For a project with very big synthDefs, I have to set very high values for the option numWireBufs of the server. Otherwise, I get an error mentioning I have to increase this value.

And I would like to know which is the maximum value I can set for numWireBufs (on both OSX and Linux) ?
Because I have remarked that setting very high numWireBufs values is OK on OSX, but generates GREY count errors on Linux.

I do not have a linux computer right now, but which is the max value for numWireBufs on Linux I can have without generating GREY count errors ?

s.options.numWireBufs = (8192*12000); s.boot;

Many thanks,

Christophe

And I would like to know which is the maximum value I can set for numWireBufs

I don’t think there’s a maximum value.

but generates GREY count errors on Linux.

What do you mean by that? Can you post an example?

What is the action you’re taking, right before the grey count errors? Because there are a lot of moving parts in your scenario, so it’s hard to guess based on “I did something (unspecified) and got this message.”

hjh

Grey count errors are a part of sclang (related to garbage collection), not the audio server, so the wirebuf count should be completely unrelated.

If your SynthDef is complex/large enough that you’re having to set an extremely high wirebuf count, it’s possible that you’re uncovering a memory-related bug on Linux - this could result in a grey count error. If you have one mega-SynthDef you might consider trying to break it up into multiple smaller SynthDefs you can chain together - this could side-step any potential problems resulting from the SynthDef being too large.

If you have code that reliably reproduces the problem, consider filling out a bug report on the issue (and post the code of course).

Also, fwiw: I often use very complex SynthDefs, to the point that I can only run one or two copies at a time before maxing out my CPU. My numWireBufs is still only set to 4092… orders of magnitude smaller than what you’ve suggested. If you’re needing a count as high as 8192*12000, it might be that you’ve got a bug in your SynthDef that’s generating a lot more complexity than you intend to. You might try using assertChannels in your SynthDef to ensure that you actually have the channel counts that you expect. (extAssertShape.sc · GitHub)

SynthDef(\channelTest, {
	var sig;
	sig = SinOsc.ar([100, 200, 300]);
	sig.assertChannels(3);  	// expect an array[3]
	sig = [sig, sig];
	sig.assertChannels(2, 3); 	// expect an array[2] of array[3]'s
	
	sig = SinOsc.ar([100, 200, 300, 400]);
	sig.assertChannels(3); 		// this errors because sig is 4 channels now
});

Did you try a power of 2 also ?

SynthDef’s topological sort is designed to minimize wirebuf usage (that is, it prefers narrow over wide graphs). This may increase usage of CPU cache over RAM, improving processing speed. It performs best on the very common case of parallel chains of UGens, mixed down at the end.

(
SynthDef(\test, {
	var sig = Saw.ar((100 .. 199));
	sig = LPF.ar(sig, (100 .. 199) * 5);
	Out.ar(0, sig.sum)
}).dumpUGens;
)

[ 0_Saw, audio, [ 100 ] ]
[ 1_LPF, audio, [ 0_Saw, 500 ] ]
[ 2_Saw, audio, [ 101 ] ]
[ 3_LPF, audio, [ 2_Saw, 505 ] ]
[ 4_Saw, audio, [ 102 ] ]
[ 5_LPF, audio, [ 4_Saw, 510 ] ]
[ 6_Saw, audio, [ 103 ] ]
[ 7_LPF, audio, [ 6_Saw, 515 ] ]

Each Saw → LPF can use 1 wirebuf, and they’re separate chains, so you need 4 wirebufs so far.

Note that the SynthDef code creates 100 Saws first, then 100 LPFs, so you’d think this is a wide graph, but the topo-sort links them up into vertical chains.

[ 8_Sum4, audio, [ 7_LPF, 1_LPF, 3_LPF, 5_LPF ] ]

Now these 4 are mixed down: back to 1 wirebuf. And you don’t need these LPFs for anything else, so this can reuse one of them.

[ 9_Saw, audio, [ 104 ] ]
[ 10_LPF, audio, [ 9_Saw, 520 ] ]
[ 11_Saw, audio, [ 105 ] ]
[ 12_LPF, audio, [ 11_Saw, 525 ] ]
[ 13_Saw, audio, [ 106 ] ]
[ 14_LPF, audio, [ 13_Saw, 530 ] ]
[ 15_Sum4, audio, [ 14_LPF, 8_Sum4, 10_LPF, 12_LPF ] ]

Mix in 3 more chains – but these can reuse wirebufs, so in theory we’re still using only 4.

… and so on, all the way down.

233 units, but could be handled with only 4 wirebufs.

If you need ten million wirebufs, you have a very interesting, odd case that the devs should check out (so the absence of code sharing in this thread limits the scope of the investigation).

hjh

2 Likes

Buuuut… I just found a case where multichannel expansion → mixdown produces a wide graph: (a * b).sum can result in a chain of MulAdd units, and it puts all of the a units upfront (requiring one wire buffer per channel).

SynthDef('wide!', {
	var n = 100;
	var a = SinOsc.ar(440 * (1..n));
	var b = LFSaw.kr({ ExpRand(1, 4) } ! n).range(0, 1);
	Out.ar(0, (a * b).sum)
}).add;

exception in GraphDef_Load: exceeded number of interconnect buffers.

// check the structure
SynthDef('wide!', {
	var n = 10;
	var a = SinOsc.ar(440 * (1..n));
	var b = LFSaw.kr({ ExpRand(1, 4) } ! n).range(0, 1);
	Out.ar(0, (a * b).sum)
}).dumpUGens;

[ 0_SinOsc, audio, [ 440, 0.0 ] ]
[ 1_SinOsc, audio, [ 880, 0.0 ] ]
[ 2_SinOsc, audio, [ 1320, 0.0 ] ]
[ 3_SinOsc, audio, [ 1760, 0.0 ] ]
[ 4_SinOsc, audio, [ 2200, 0.0 ] ]
[ 5_SinOsc, audio, [ 2640, 0.0 ] ]
[ 6_SinOsc, audio, [ 3080, 0.0 ] ]
[ 7_SinOsc, audio, [ 3520, 0.0 ] ]
[ 8_SinOsc, audio, [ 3960, 0.0 ] ]
[ 9_SinOsc, audio, [ 4400, 0.0 ] ]  -- here we're up to 10 wirefbufs
[ 10_ExpRand, scalar, [ 1, 4 ] ]
[ 11_LFSaw, control, [ 10_ExpRand, 0.0 ] ]
[ 12_MulAdd, control, [ 11_LFSaw, 0.5, 0.5 ] ]
[ 13_ExpRand, scalar, [ 1, 4 ] ]
[ 14_LFSaw, control, [ 13_ExpRand, 0.0 ] ]
[ 15_MulAdd, control, [ 14_LFSaw, 0.5, 0.5 ] ]
[ 16_*, audio, [ 1_SinOsc, 15_MulAdd ] ]   -- second channel is complete
[ 17_MulAdd, audio, [ 0_SinOsc, 12_MulAdd, 16_* ] ]  -- 1+2 are complete
[ 18_ExpRand, scalar, [ 1, 4 ] ]
[ 19_LFSaw, control, [ 18_ExpRand, 0.0 ] ]
[ 20_MulAdd, control, [ 19_LFSaw, 0.5, 0.5 ] ]
[ 21_MulAdd, audio, [ 2_SinOsc, 20_MulAdd, 17_MulAdd ] ]  -- +3
...

So * with + is a special case (but a very common one!). I’d go so far as to call this a bug (or, at least, suboptimal behavior).

hjh

1 Like

Hello,

Thanks to all for the answers and still sorry for my long delay for answering (I was lost with this and other things).

Unfortunately, I cannot post a simple example, since it is linked to a big experimental project for spatial performance, I have been working on for several years, that I will announce very soon on this forum.

@scztt The reason why I wrongly linked high values of numWireBufs with grey count errors is because I noticed setting high values on Linux (above 1024) would generate automatically grey count errors with my program, whereas on Mac I can rarely have this issue.
That is why I set a limit for Linux on this line of the project.

Many thanks for your help,

Christophe

BTW I might have a fix for this case: Classlib: Invert the SynthDef topological sort (to fix a wide-graph case) by jamshark70 · Pull Request #5811 · supercollider/supercollider · GitHub

hjh

In Sc graphs are only partially ordered by their edges? I.e.

{
Out.ar(0, PinkNoise.ar() * 0.1);
ReplaceOut.ar(0, LPF.ar(In.ar(0, 1), 440))
}.play

Won’t your patch reverse this?

Probably not, actually. It’s a double-reverse. The current algorithm adds ancestorless units to an “available” array, in reverse order, and then pulls from the tail of that array to find the earliest UGens to evaluate. My PR adds descendantless units in forward order (later UGens at the end – first reversal), and then pulls from the tail of the array to find the latest UGens to evaluate (second reversal). So initially, with my change, the available units would be Out and ReplaceOut, in that order, and ReplaceOut would be the tail of the SynthDef. Its inputs would precede that, and Out would then come before those.

Though I’d say that’s dodgy SynthDef code style, to do something side-effect-y when it could be done functional-style.

hjh

A nice aspect of the “framed” (mutating) graph builder in Sc is that you don’t need to annotate disconnected sub-graphs. It’s very idiomatic?

{
	var nf = 2 * 48000;
	var buf = LocalBuf.new(numFrames: nf, numChannels: 1);
	var in = SinOsc.ar(LFNoise1.kr(2) * 300 + 400, 0) * 0.1;
	var ph = Phasor.ar(0, 1, 0, nf, 0);
	BufWr.ar(in, buf, ph, 1);
	BufRd.ar(1, buf, ph, 1, 2);
}.play

I’m probably misunderstanding, but if they are truly disconnected, then order wouldn’t matter…? What you’re proposing instead is to use an extra two UGens to make a connection by some means other than direct input, adding CPU cost for no functional gain. (Perhaps it’s unfashionable now to try to write things efficiently but I’ll happily remain a dinosaur in that regard then :laughing: )

Besides… a BufWr / BufRd pair was one of those cases where you could be sure of the order only by forcing the order using <!. I’ve seen it dozens of times where BufRd got pulled up earlier in the graph, before the writer, introducing a block delay. It was never safe to rely on the order in which the units were written.

hjh

Back at the computer… some concrete examples.

I did find that my PR had one mistake. I had intended to “[add] descendantless units in forward order” but the first version of it doesn’t change reverseDo to do. I’ve fixed that now in the PR.

(
SynthDef(\outSideEffect, {
	Out.ar(0, PinkNoise.ar() * 0.1);
	ReplaceOut.ar(0, LPF.ar(In.ar(0, 1), 440));
}).dumpUGens;
)

[ 0_PinkNoise, audio, [  ] ]
[ 1_*, audio, [ 0_PinkNoise, 0.1 ] ]
[ 2_Out, audio, [ 0, 1_* ] ]
[ 3_In, audio, [ 0 ] ]
[ 4_LPF, audio, [ 3_In[0], 440 ] ]
[ 5_ReplaceOut, audio, [ 0, 4_LPF ] ]

^^ This example produces the expected, useful order in both the develop and my PR branches (with the forward order fix) – “Won’t your patch reverse this?” No, it doesn’t (though maybe it did, at the time you asked).

I’ve seen it dozens of times where BufRd got pulled up earlier in the graph, before the writer…

(
SynthDef(\notOK, { |out, bufnum|
	var ph = Phasor.ar(0, 1, 0, BufFrames.kr(bufnum));
	var sig = SinOsc.ar(440);
	var writer = BufWr.ar(sig, bufnum, ph);
	var reader = BufRd.ar(1, bufnum, ph);
	Out.ar(out, reader);
}).dumpUGens;
)

notOK  // this is the current dev-branch ordering
[ 0_Control, control, nil ]
[ 1_BufFrames, control, [ 0_Control[1] ] ]
[ 2_Phasor, audio, [ 0, 1, 0, 1_BufFrames, 0.0 ] ]
[ 3_BufRd, audio, [ 0_Control[1], 2_Phasor, 1.0, 2 ] ]
[ 4_Out, audio, [ 0_Control[0], 3_BufRd[0] ] ]
[ 5_SinOsc, audio, [ 440, 0.0 ] ]
[ 6_BufWr, audio, [ 0_Control[1], 2_Phasor, 1.0, 5_SinOsc ] ]  <<-- NO! This is not what you want

Before testing these defs, I had thought that maybe moving the Phasor to immediately precede BufWr might fix the order – but this didn’t pan out.

Curiously, your LocalBuf example does produce the right order, but with a bit of testing, it turns out that this is only because your Phasor has no antecedents. If nf = BufFrames.kr(...), then the order is broken! But basing a Phasor on BufFrames is an entirely normal thing to do – so I’d say that your example is an unusual case, and that in reality, considering all the cases we want to support, it’s brittle and quite unreliable to assume that BufWr will come first just because it was written first.

In current SC, it is always safer to force the order using firstArg.

(
SynthDef(\ok, { |out, bufnum|
	var ph = Phasor.ar(0, 1, 0, BufFrames.kr(bufnum));
	var sig = SinOsc.ar(440);
	var writer = BufWr.ar(sig, bufnum, ph);
	var reader = BufRd.ar(1, bufnum <! writer, ph);
	Out.ar(out, reader);
}).dumpUGens;
)

ok
[ 0_SinOsc, audio, [ 440, 0.0 ] ]
[ 1_Control, control, nil ]
[ 2_BufFrames, control, [ 1_Control[1] ] ]
[ 3_Phasor, audio, [ 0, 1, 0, 2_BufFrames, 0.0 ] ]
[ 4_BufWr, audio, [ 1_Control[1], 3_Phasor, 1.0, 0_SinOsc ] ]
[ 5_firstArg, audio, [ 1_Control[1], 4_BufWr ] ]
[ 6_BufRd, audio, [ 5_firstArg, 3_Phasor, 1.0, 2 ] ]
[ 7_Out, audio, [ 1_Control[0], 6_BufRd[0] ] ]

BUT… with my patch… let’s go back to the notOK version… suddenly it’s become OK.

[ 0_SinOsc, audio, [ 440, 0.0 ] ]
[ 1_Control, control, nil ]
[ 2_BufFrames, control, [ 1_Control[1] ] ]
[ 3_Phasor, audio, [ 0, 1, 0, 2_BufFrames, 0.0 ] ]
[ 4_BufWr, audio, [ 1_Control[1], 3_Phasor, 1.0, 0_SinOsc ] ]  <<-- YES! This is the right way
[ 5_BufRd, audio, [ 1_Control[1], 3_Phasor, 1.0, 2 ] ]
[ 6_Out, audio, [ 1_Control[0], 5_BufRd[0] ] ]

… because Out pulls its immediate ancestor (BufRd) into the tail before getting to the other BufWr branch.

So I still think the PR is onto something (though more testing is needed).

hjh