Selector table too big

Questions
1

Hi,

Just happened on a new-to-me error:

ERROR: Selector table too big: too many classes, method selectors or function definitions in this function. Simplify the function.
Next literal was:
   instance of FunctionDef - closed
  in interpreted text
  line 1 char 113993:

  [ [ ESTrack([ ESSynthClip(0.0, 1.0, 'sin', {[freq: 6.25, verbbus: ~verbbus, verbamt: 1, pan: -0.75824594497681]}), ESSynthClip(3.0, 1.0, 'sin', {[freq: 7.5, verbbus: ~verbbus, verbamt: 1, pan: -0.44909715652466]}), ESSynthClip(6.0, 1.0, 'sin', {[freq: 8.7
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 Interpreter has crashed or stopped forcefully. [Exit code: 15]

while trying to restore state from previous state saved as compile string. Wondering, is there a way to know this hard limit is coming?

2

Could you post the contents of the string? There is a limit on how big functions can be (actually, there’s limits on most things).

3

The string is a compile string for an object that takes some arrays of other objects… in this case it’s a lot of objects (~10,000)

here is the full thing: https://gist.githubusercontent.com/esluyter/a21ab631f5023ec482ffcb32097174a9/raw/4bac45c33f9a95302fd25cae26e87cdf21c6170b/gistfile1.txt

the original lives happily in memory and generates the compile string fine, it’s just that calling .interpret on the string hangs the interpreter with a series of these “selector table too big” errors. So I guess maybe the better question is, is there a better way to store and recover previous state? I’m in the habit of using compile strings because it’s easy to check if one is the same as another, and they are also human readable.

4

The problem is if you have a lot of functions living at the same function level.

What’s a “selector”? It’s a non-special class name, or a non-special method, or a functiondef.

a = { Array.new(5) };
a.def.selectors
-> nil

a.def.dumpByteCodes
BYTECODES: (8)
// aha -- Array is a SpecialClass
  0   06 19    PushSpecialClass 'Array'
  2   2C 05    PushInt 5
  4   B0       TailCallReturnFromFunction
// and 'new' is a SpecialMsg
  5   C2 00    SendSpecialMsg 'new'
  7   F2       BlockReturn

// more exotic
a = { MixerChannel.newFromDef(440) };

a.def.selectors
-> [newFromDef, MixerChannel]

// now let's put a function in there too:
(
a = {
	var x = { |argument| argument.doSomething };
	MixerChannel.newFromDef(440);
};

a.def.selectors
)
-> [a FunctionDef, newFromDef, MixerChannel]

There’s a hard limit, and a relatively low limit, on the size of the ‘selectors’ array – I think it’s 256 (?). This is because functions are really not supposed to get so large – when you’re writing code, it’s always possible to break down a large function into multiple smaller functions.

interpret makes a function out of the code string, so the entire string is subject to that limit. That means, if you have a big flat data structure with a LOT of functions, they will all be ‘selectors’ and you could run into trouble.

The general problem that you’re facing is how to serialize, and de-serialize, a large data structure (same as LibreOffice Writer serializing your document into an .odt file, and de-serializing it when you open it). asCompileString might handle some serialization cases, but it isn’t designed as a general-purpose object-save-and-restore mechanism. If you’re making a full timeline, with tracks and clips, it doesn’t surprise me that asCompileString eventually reaches a limit of scale.

Basically, you’ll need to design a file format.

A possible “cheat”: in your gist, you have a lot of argument lists that are double-enclosed: {[args...]} – and the only reason for the double-enclosure is ~verbbus. If you can find a way to handle bus routing without using functions to lazy-access variables, then a lot of the present problem might disappear (or, at least, push the size limit further into the future).

E.g.

var verbbus = { [verbbus: ~verbbus] };

ESTimeline([
	ESTrack([
		// actually on second thought I'm not sure about this
		ESSynthClip(0.0, 1.0, 'sin', verbbus.value ++ [freq: 6.25, verbamt: 1, pan: -0.75824594497681]),
		...
		...
	]),
	...
])

(Though you might need to add an arg to ESSynthClip that accepts a function only for the routing parameters… it occurred to me after posting that verbbus.value might prematurely evaluate the bus number.)

The idea would be to replace hundreds of distinct functions with one for bus maps. Will it take more work to render like this? Yes. Can you avoid that work? If you’re hitting a scalability limit of asCompileString, then no, you can’t avoid it anymore.

hjh

5

OK if I understand… this means at a certain scale these data structures won’t be expressible anymore with SCLang, at least in a way that is reversible.

So the only sure way to store and recall the data structure is to make a file format, although if I were to optimize somehow the limits of SCLang could be pushed farther.

(The reason they’re double enclosed btw is I want to accept any valid function that produces an argument list… not just for bus routing. And each clip might have a unique function, so they can’t necessarily be condensed. I guess this is at odds with also wanting to express the enitre data structure in SCLang…)

Thanks as always for your insights

6

In that case, I’d at least try to devise a way to “lift” the functions out of the compile string syntax. If the functions are listed on separate lines with some kind of tag, then you could extract them from the file one by one and .interpret each function separately. Each .interpret call produces its own function context, so you wouldn’t overload a single FunctionDef.

Not easy exactly, but what you’re doing isn’t easy (and it would be cool to have a really well functioning timeline in SC!).

hjh

7

not sure if this is helpful but when I have a file or code block that runs into the selector table limit (which happens all the time! I wrap a bunch of it in {..}.() and that reduces the count since the new function when evaluated only counts as one selector (the return value).

(
a = {var a; SinOsc};
b = {var a; Saw};
) //two selectors

(
{ 
	a = {var a; SinOsc};
	b = {var a; Saw};
}.() //one selector
)

This has allowed me to pass arbitrarily large strings to the interpreter in one go (well up to 256 chunks of 256 selector sized functions anyhow :wink:

8

I guess this is a question for the developers: Would it be possible to increase this limit in SC in the future or is it fixed for a reason?
Maybe having a larger limit has extra costs or other implications? (I’m not a developer so it’s just a guess)
One of the issues I’ve found with this restriction is that in a SynthDef function, the total number of arguments plus variables cannot be above this limit.
This has forced me in the past to split code up into multiple SynthDefs which isn’t always easy - sometimes you want a single deep, complex SynthDef with lots of controls.
It would be very useful if the limit could be made larger.
I wonder if this made sense for SC 20 years ago (when I remember a single vocoder synth took up 75% CPU!) - but is it still a necessary limit now (with faster computers/ more memory etc)?
Thanks for any insights,
Paul

2 Likes
9

I’m giving it my all :slight_smile: it’s a real puzzle, to do well in a way that doesn’t restrict. And of course not sure the solutions that work best for me will work generally, but we’ll see. Trying to keep it as vanilla as possible but thinking I’ll try to incorporate ddwplug because otherwise I can see bus management spinning out of control fast and you have already solved a lot of these problems.

10

I have solved this for my own case by making a compile stack that only compiles one object at a time… posting here in case this is helpful for others

+ Object {
  asESArray {
    ^this.asCompileString;
  }

  *fromESArray { |arr|
    var stack = [];

    if (arr.isString) {
      ^arr.interpret;
    };

    arr.do { |item|
      if (item == $() {
        stack = stack.add((type: \class, args: []));
      };
      if (item.class.class == Class) {
        stack.last[\class] = item;
      };
      if (item == $[) {
        stack = stack.add((type: \array, args: []));
      };
      if (item.class == String) {
        stack.last[\args] = stack.last[\args].add(item.interpret);
      };
      if ((item == $]) or: (item == $))) {
        var ev = stack.pop;
        if (stack.size == 0) {
          stack = stack.add((type: \result, args: []));
        };
        switch (ev[\type])
        { \class } {
          stack.last.args = stack.last.args.add(ev[\class].new(*ev[\args]));
        }
        { \array } {
          stack.last.args = stack.last.args.add(ev[\args]);
        };
      };
    };
    ^stack.last.args[0];
  }
}

+ Array {
  asESArray {
    ^[$[, this.collect(_.asESArray), $]].flatIf(_.isString.not); ///ugg .flat makes strings chars
  }
}

+ ESTimeline {
  asESArray {
    ^[$(, this.class, this.storeArgs.collect(_.asESArray), $)].flatIf(_.isString.not);
  }
}

+ ESTrack {
  asESArray {
    ^[$(, this.class, this.storeArgs.collect(_.asESArray), $)].flatIf(_.isString.not);
  }
}

+ ESClip {
  asESArray {
    ^[$(, this.class, this.storeArgs.collect(_.asESArray), $)].flatIf(_.isString.not);
  }
}
2 Likes