How to deal with big SynthDefs? (UGen Graph size)

Questions
1

Hello everybody!

From time to time i run into some problems with my SynthDefs sizes, since i make a lot of use of multichannel expansion.

I did not find resources yet where the limitations are described and how to work with these.
Can someone guide me with some information with these problems?

The current SynthDef i have problems with is this one:

(
SynthDef(\playBuf, {|buf = 0, num = 32|
  var maxNum = 32;
  var start = 0;
  var sig = {|i|
    PlayBuf.ar(
      numChannels: 2,
      bufnum: buf,
      rate: BufRateScale.ir(buf) * { Rand(-32.0, 0.0) }.midiratio,
      trigger: 1,
      startPos: 0) * (i < num)} ! maxNum;

  Line.kr(0, 1, BufDur.ir(buf) * 8, doneAction: 2);
  sig = Splay.ar(sig);
  Out.ar(0, sig);
}).add;
)

and the error message is this:

ERROR: Primitive '_NetAddr_SendMsg' failed.
caught exception 'send_to: Message too long [system:40 at /Users/runner/work/supercollider/supercollider/external_libraries/boost/boost/asio/detail/reactive_socket_service.hpp:261:33 in function 'send_to']' in primitive in method NetAddr:sendMsg
RECEIVER:
Instance of NetAddr {    (0x7f9c3014a448, gc=D0, fmt=00, flg=00, set=02)
  instance variables [4]
    addr : Integer 2130706433
    port : Integer 57110
    hostname : "127.0.0.1"
    socket : nil
}
PATH: /Users/philippneumann/Documents/Kompositionen/SommerAmU/SommerAmU.scd
CALL STACK:
	MethodError:reportError
		arg this = <instance of PrimitiveFailedError>
	Nil:handleError
		arg this = nil
		arg error = <instance of PrimitiveFailedError>
	Thread:handleError
		arg this = <instance of Thread>
		arg error = <instance of PrimitiveFailedError>
	Object:throw
		arg this = <instance of PrimitiveFailedError>
	Object:primitiveFailed
		arg this = <instance of NetAddr>
	Server:sendMsg
		arg this = <instance of Server>
		arg msg = [*3]
	SynthDef:doSend
		arg this = <instance of SynthDef>
		arg server = <instance of Server>
		arg completionMsg = nil
		var bytes = <instance of Int8Array>
		var path = nil
		var resp = nil
		var syncID = nil
	< FunctionDef in Method Set:do >
		arg item = <instance of Server>
	ArrayedCollection:do
		arg this = [*4]
		arg function = <instance of Function>
		var i = 3
	Set:do
		arg this = <instance of Set>
		arg function = <instance of Function>
		var i = 0
	SynthDef:add
		arg this = <instance of SynthDef>
		arg libname = nil
		arg completionMsg = nil
		arg keepDef = true
		var servers = <instance of Set>
		var desc = <instance of SynthDesc>
	Interpreter:interpretPrintCmdLine
		arg this = <instance of Interpreter>
		var res = nil
		var func = <instance of Function>
		var code = "(
SynthDef(\playBuf, {|buf =..."
		var doc = nil
		var ideClass = <instance of Meta_ScIDE>
	Process:interpretPrintCmdLine
		arg this = <instance of Main>
^^ The preceding error dump is for ERROR: Primitive '_NetAddr_SendMsg' failed.
caught exception 'send_to: Message too long [system:40 at /Users/runner/work/supercollider/supercollider/external_libraries/boost/boost/asio/detail/reactive_socket_service.hpp:261:33 in function 'send_to']' in primitive in method NetAddr:sendMsg
RECEIVER: a NetAddr(127.0.0.1, 57110)
2

First, editing a bit for readability (I misinterpreted the code at first):

(
d = SynthDef(\playBuf, { |buf = 0, num = 32|
	var maxNum = 32;
	var start = 0;
	var sig = { |i|
		PlayBuf.ar(
			numChannels: 2,
			bufnum: buf,
			rate: BufRateScale.ir(buf) * { Rand(-32.0, 0.0) }.midiratio,
			trigger: 1,
			startPos: 0
		) * (i < num)
	} ! maxNum;
	
	Line.kr(0, 1, BufDur.ir(buf) * 8, doneAction: 2);
	sig = Splay.ar(sig);
	Out.ar(0, sig);
});
)

d.asBytes.size  // 16104

~= 16 KB, which is not much, really. Per a c++ stack overflow post, e.g., UDP packets use two bytes for the size, so a single packet could not exceed 65536 bytes. (The practical limit for transmission over a network may be less than this – but we are typically sending to the local machine, and, if we experienced significant packet loss on localhost, there wouldn’t have been a million bug reports about that over the last 20+ years, which we haven’t seen.)

So I suspect there may be some OS-level setting that is preventing larger UDP packets from being sent on the same machine. That is, the error dump indicates that SC built the SynthDef, and believes that it isn’t too large to transmit in UDP (see note, though), but the socket layer rejects the packet.

Note: SC language uses 16383 as the maximum synthdef size for network transmission before falling back to disk transmission beyond that size. You’re coming in just under that limit on the SC side, but just above some limit in the network socket layer.

BTW which SC version are you using? SC 3.14 (currently released as a test build) increases the socket buffer size, which might(?) make a difference in this case.

hjh

3

ok, thank you!

I use SC 3.14 already. i also have this warning when the sclang interpreter is starting:

(sclang) SC_UdpInPort: WARNING: failed to set send buffer size
(sclang) SC_UdpInPort: WARNING: failed to set receive buffer size

This looks related to the problem?

so my take away so fare:

  • sclang is sending an OSC message via UDP to the server, which describes the SynthDef. part of the message is the size of the SynthDef
  • network transmission is limited to 2^16 message size
  • disk transmission is automatically used, when SynthDefs are bigger?

PvN

4

Is this _NetAddr_SendMsg "Message too long" errors in 3.14.0-rc1 on SuperDirt startup · Issue #6969 · supercollider/supercollider · GitHub ?

5

This was the bug! I loaded the new release and the warning disappeard and i can add my SynthDefs again!

Thank you for mentioning it!

1 Like
6

i want to ask some more questions related to this:

  • what are theses OSC messages sending? i guess these messages are in some kind of form instructions to the sever. are theses instructions compile instructions, already transpiled to c++? or how do they look like?
  • how does the change from network transmission to dis transmission look like? do we notice them in any form? do wo have controll over these?
  • what server settings are relevant to this whole process? what should i be aware off?

thanks!
PvN

7

The complete list of server messages is at Server Command Reference | SuperCollider 3.14.0-dev Help .

Transmitting SynthDefs:

  • /d_recv – the SynthDef is encoded into an array of bytes and this array is part of the message.
  • /d_load – the command accepts a path to a file on disk, containing the bytes generated from the SynthDef.

None of the above – see Synth Definition File Format | SuperCollider 3.14.0-dev Help

Normally it’s transparent – .add decides which approach to use based on the data size.

You can manually .writeDefFile and issue /d_load with the path by yourself… actually I have a vague recollection of a .load method for SynthDef, which should do those two steps in one go (but this is not quite the same as .add).

Not much. If a SynthDef is very “wide” then it might be necessary to increase numWireBufs. There’s also a setting to load SynthDefs from disk automatically upon boot (default is true, there’s really no reason to change that).

hjh

1 Like
8

Thank you! I will take a lot into that to get a deeper understanding how SC actually works.