To Proto or not to Proto?

Splitting off from:

For fun, I just counted up how many lines of code in my quarks are in .sc class files, and how many are in .scd.

  • .sc = 26232
  • .scd = 23419

Now… not everything in .scd will be a Proto, but it would still have to be, oh, 30-35% (?) of my public code being “soft”-defined.

I noticed in the last couple of years the opinion being expressed more frequently that “object prototyping doesn’t work.” Weeelll… I dunno… it’s been working quite well for me, for a couple of decades.

IMO whether to use prototyping or not depends on individual inclination, and the situation.

I started prototyping because, in 2004, I was losing a lot of time while preparing a concert piece because my object design required me to recompile the class library and reload the entire environment, just to change one small thing. That was an awful experience. I decided at that time to make all of my creative code “soft” – to change something compositionally, it would save a lot of time to be able to delete one process and re-create only that process, rather than having to blow away and rebuild the entire environment.

Are there trade-offs with prototypes? Yes. Are they worth it when, for instance, I need to debug one of the generator objects in my live coding environment?

  • If those generators were hard classes: Change something, recompile the class library, reload the LC environment, step through a few init lines in a testing script, test.
  • Actual process: Change something, reevaluate the generators.scd file, reload the one line LC pattern, play the process.

In that context, I find Protos are slightly less readable but significantly more flexible, so I choose to continue to use them.

By contrast, the parser in my LC environment needs to be stable and reliable: malleability is a negative. So those are hard classes.

Criticism of object prototyping usage is usually:

  • You have to avoid methods defined in Object and Environment.
    • ddwProto offers a partial solution by “has-a” wrapping Environment, and not implementing many of the Environment methods.
    • Beyond that: In practice, I have only very, very seldom gotten tripped up by this. It’s really not that big of a deal.
  • self. I’ve vocally disagreed with this design decision before. An object’s internal code is normally accessing local member variables. self inverts that: in “vanilla” object prototyping, it’s easier to access globals than it is to access locals. Of course that’s unpleasant! One reason why I’ve gotten good results out of prototyping is that I just don’t do it that way. My Proto class prioritizes access to local variables, and a whole lot of readability problems just disappear.
DietCVUnitShaperLoader {
	*initClass {
		StartUp.add {
			(this.filenameSymbol.asString.dirname +/+ "unit-shapers.scd").load;
		}
	}
}

Aaanyway… just a reminder that mileage may vary, and one person’s opinion may not apply to your needs.

hjh

PS To make it more concrete what I mean about local vars:

// "vanilla“ prototyping
(
~collatz = (
	n: rrand(10000, 50000),
	// cannot use `next`
	nextValue: { |self|
		if(self.n.odd) {
			self.n = self.n * 3 + 1
		} {
			self.n = self.n div: 2
		};
	}
);
)

~collatz.nextValue;

// Proto
(
~collatz = Proto {
	~n = rrand(10000, 50000);
	// Proto implements 'next' to redirect to ~next
	~next = {
		if(~n.odd) {
			~n = ~n * 3 + 1
		} {
			~n = ~n div: 2
		};
	};
};
)

~collatz.next;

Is recompiling such a pain today when it takes about a second?

Another down side to prototypes is that you can have an old version still active, by recompiling you know all the objects of that type behave the same - this would also be a concern if we allowed class changes at runtime.

1 Like

Reloading the live coding environment:

  • 2-3 seconds to boot the server (haven’t timed it exactly);
  • 1-5 seconds to open MIDI;
  • Dialog box for options (which clock, which external controller);
  • 1-2 seconds to load SynthDefs;
  • If I’m using my tablet controller, there’s a manual ping.

In a dev cycle, I might not use the full environment and just \loadAllCl.eval and use a simplified testing script, but if I hit a bug while jamming at home and I want to look into it, I rather like not being forced to restart the system.

In any case, my intent here was just to balance the picture. I’m perfectly fine with it if the vast majority of users don’t use prototypes – it’s not up to me to tell others which approaches should suit them. I just think the assessment of prototypes should not be one-sided.

hjh

1 Like

That is quite a while…

Whilst adding method to classes isn’t going to be the easiest without switching to a hashmap, redefining existing methods without recompiling could be done today pretty easily, the hardest design bit would be telling the interpreter that’s the intention.

1 Like

That’s what pretty much every other dynamic scripting language does (JS, Python, Ruby, Lua, etc.), so I’m surprised that people would consider this a big deal.

One important drawback of prototyping you haven’t mentioned is the lack of documentation support, i.e. no auto completion or Ctrl+D. Some scripting languages solve this with docstrings or special comments, but unfortunately sclang does not have such a feature (yet).

1 Like

Thanks for opening the thread, this was really insightful :slight_smile:
Im just naively using self with the proto objects and havent encountered a problem yet.

For now i saved the ProtoDef inside a unit shaping.scd file with s.waitForBoot (could probably also be a Routine) and s.sync before the evaluation of Prot.

s.waitForBoot({

	ProtoDef(\unitShaping) {

	};

	s.sync;

	~unitShapers = Prot(\unitShaping);

	s.sync;

	"Unit Shapers loaded".postln;
});

And then im running "unit shaping.scd".loadRelative; from the main file inside s.waitForBoot, which im doing for all the relatives im loading.

Either "utils.scd".loadRelative; if they are in the root of the main file
or
if i have to load several files from a subfolder
"%/Fx/*.scd".format(thisProcess.nowExecutingPath.dirname).loadPaths;

For all of my projects i have the same folder structure with a private github repo.

That at least has worked for me for some time now :slight_smile:
But especially in the last year i have felt the urge to share some of my discoveries with the community and thought about whats most accessible.

1 Like

I remember when I first started using SC and had this bug…

a = (numChannels: 4);
a.numChannels; // 1

While this is easy to avoid if you know about it, if you don’t it, can be really hard to debug. This made me quite annoyed at the time, so I now only use snake case in SC, doesn’t work for some names like size so I also try to use naming conventions from other languages just in case. It doesn’t catch me anymore, but because of this I wouldn’t recommend it to a new user.

It’s also worth mentioning that future changes to the class library or what quarks you have installed may break existing proto code by adding new methods to object or dictionary (and friends).

Obviously collisions can be checked at instantiation time, but this isn’t a part of the class library. Perhaps this check should be performed when known is true?

That’s indeed a big flaw. The fundamental problem is that Event always prefers methods over dictionary entries. Other dynamic scripting languages do not have this problem because every method is just a dictionary entry and objects are either empty by default or they use a specific naming convention for “built-in” methods/properties (e.g. dunder methods in Pythons). In comparison, sclang’s object prototyping feels quite brittle to me indeed.

Obviously collisions can be checked at instantiation time, but this isn’t a part of the class library. Perhaps this check should be performed when known is true?

That’s a great idea! IdentityDictionary.put is implemented as a primitive so the additional check would not cost too much I think.

// "identDictPut" (PyrListPrim.cpp):

if (knows && IsSym(key)) {
    if (slotRawSymbol(key) == s_parent) {
        slotCopy(&dict->slots[ivxIdentDict_parent], value);
        g->gc->GCWrite(dict, value);
        return errNone;
    }
    if (slotRawSymbol(key) == s_proto) {
        slotCopy(&dict->slots[ivxIdentDict_proto], value);
        g->gc->GCWrite(dict, value);
        return errNone;
    }
    // NEW: check if symbol matches a method name
    auto sel = slotRawSymbol(key);
    auto idx = slotRawInt(&dict->classptr->classIndex) + sel->u.index;
    auto meth = gRowTable[idx];
    if (slotRawSymbol(&meth->name) == sel) {
        // should this be an error or should we just print a warning?
        return errFailed;
    }
}
1 Like

I’m surprised that these languages wouldn’t prefer locality.

Well, I Am Not A Computer Scientist, so I’m probably “wrong,” but if a prototype object is an environment, and you have a token to access the current environment, why would you not use that token as this?

I do recall JMc said it was to allow access to the outside environment variables from within the prototype – but, what I was trying to say is: inside an object, the common case is to access locals. Accessing external environment variables should be relatively rare, because restricting the flow of information in and out of local scope is designed to prevent mistakes related to global variable collisions. I cut my teeth on Apple II BASIC – everything is a global! – so I know why local scope was invented in the first place :wink: so I find it quite odd that scripting languages would adopt a less convenient syntax self.varname for the common case, and a more convenient syntax (~global in SC) for what should be a rare case.

If my test case depends on server resources, I’d like library recompilation to be only under my direct control.

True, currently there is no way to insulate my prototypes from changes to Object.

But Proto is not an Environment. It “has-a” environment, but not “is-a.”

So if the dictionary tree picks up a new method, Proto does not inherit it.

(I feel a little bit like this probably reflects the common bias in the SC community towards subclassing over Decorator usage. For this case, the Decorator approach allows much more control over the Proto interface. There are very good reasons not to use Environments or Events directly for prototyping, but somehow we are still only talking about using them directly :laughing: )

This is not a bad idea! I think it would be fine to return an error here, because the class library can implement a fallback (i.e., a user option to either throw an Error, or do something else).

hjh

1 Like

Well, I Am Not A Computer Scientist, so I’m probably “wrong,” but if a prototype object is an environment, and you have a token to access the current environment, why would you not use that token as this?

In these languages, this or self refers to the object itself and the prototype/class is just another object. The function environment (= the place where variables are looked up) is completely separate from that.

You typically have two different look-up chains:

  1. self.fooself → prototype/class of self → super prototype/class → etc.
  2. foo → local variable → (closure upvalues) → global variable

Note that 1. always happens at runtime, but 2. can be done at compile time (the compiler knows if a variable is a local variable or upvalue). The global variable access can itself be implemented as a runtime lookup in a chain of environments.

In prototypical languages like JS or Lua, methods are not special; they are just functions stored in a dictionary, and when they are called, they receive the object as the first argument. The function itself does not know that it is a “method”. Implicit this/self would require modifying the function environment on every method call.

Class-based languages can indeed make this/self implicit. Ruby allows this for method calls:

class Foo
  def initialize()
    @x = 100
  end

  def method_2
    method_1 # implicit this!
  end

  def method_1
    puts @x
  end
end

Instance variables are prefixed with @, though.

So if the dictionary tree picks up a new method, Proto does not inherit it.

Ah, I totally missed that Proto solves this issue! That’s pretty cool!

1 Like

Thanks for this! The class library could also print a error explaining what known is as people might be using square bracket access and not have this issue, so they need a way to disable it.

I’m a little too busy to put forward a pr right now but could maybe do so in March sometime, unless you wanted to?

1 Like

It also enabled me to let Protos work as patterns. This can never work with Event prototypes because .embedInStream gets swallowed.

(
~collatz = Proto {
	~canEmbed = true;  // if false, the proto is embedded as an object, not a stream
	~prep = { |n|
		~n = n;
		currentEnvironment
	};
	~embedInStream = { |inval|
		while {
			~n > 1
		} {
			inval = ~n.yield;
			if(~n.odd) {
				~n = ~n * 3 + 1;
			} {
				~n = ~n div: 2;
			};
		};
		~n.yield  // return last inval to caller
	};
};
)

p = Pseq([~collatz.copy.prep(10), Pseries(0, 1, 3)], 1).asStream;
p.nextN(12)

-> [ 10, 5, 16, 8, 4, 2, 1, 0, 1, 2, nil, nil ]

hjh

1 Like

Turns out that sclang already prints a warning, just not in the constructor! I’ve just opened an issue: Warn/error in Event constructor when trying to override existing method · Issue #6594 · supercollider/supercollider · GitHub

3 Likes

One reason why I like protos is that if you share your code with beginners, it just works. There’s no installation process needed, Quarks explanation, code can be modified without having to explain what ‘recompiling the library’ is, etc.

When we were discussing the current class library trimming, I asked myself if we could somehow try to only keep Object (and Event) as a real class, and turn every other class into a proto, which would allow to test what breaks if your remove a particular class without preventing compiling.

Since then, I had in mind the idea that, with a strict syntax, one could write a Proto > Class.sc or a Class.sc > Proto converter, which might have some uses in some particular cases. But I didn’t take the time to try to implement it.

2 Likes

Oh, that’s extreme! :blush:

This is a short coming of quarks. I’d like to see SC move towards a project/directory based structure where if you have a directory called Classes at the top level it automatically includes it. I think some steps where taken towards this, but haven’t kept up to date.

Class method dispatch is much faster than prototype dispatch.

1 Like
(
f = Point(0, 0);
g = (myValue: 0);
h = Proto { ~myValue = 0 };

"number: %, event proto: %, hjh Proto: %\n"
.postf(
	{ 1000000.do { f.x } }.bench(false),
	{ 1000000.do { g.myValue } }.bench(false),
	{ 1000000.do { h.myValue } }.bench(false),
);
)

number: 0.03019864800001, event proto: 0.16074079499998, hjh Proto: 0.16034691600001

For variable access, about 5x worse.

(
f = { 0 };
g = (myValue: { 0 });
h = Proto { ~myValue = { 0 } };

"number: %, event proto: %, hjh Proto: %\n"
.postf(
	{ 1000000.do { f.value } }.bench(false),
	{ 1000000.do { g.myValue } }.bench(false),
	{ 1000000.do { h.myValue } }.bench(false),
);
)

number: 0.051263259, event proto: 0.165710842, hjh Proto: 0.866826356

For function dispatch, about 3x? (ddwProto is even slower because it uses the environment when calling a pseudomethod, adding error-handling weight. I’ve been willing to make that trade-off because I wanted local scope for the environment variables, and I’ve tended to use Protos for code that is less sensitive to performance. Speed isn’t always the main factor; if it were, we’d all still be writing assembler.)

hjh

I agree that class method dispatch is faster. I would also point out that I think that since it uses less steps to access methods, it reduces the heat the computer produces and thus is more ecological.

However, I don’t know about you, but I personally can’t notice the speed gain in most cases. I rarely have functions that perform 1 million actions. Of course, if I do, I would write a class for efficiency reasons. In this particular case, I think it would be even better to have a primitive to be even faster.

I’m working on a monstrosity, a SuperCollider DAW. It uses so many protos it has a folder for them. Not kidding, I think there’s around 50 ‘proto classes’, with inheritance and stuff. I never experienced any lag problem with it. I click, it produces sound, I’m happy.

Mostly you’re benching the speed of do there.

1 Like