Object prototyping is NOT broken for higher oder functions – but WHY?

1

for context of discovery, see #6227.

I found that functionPerformList is defined everywhere as

functionPerformList { arg selector, arglist;

But in IdentityDictionary, it is used as:

^func.functionPerformList(\value, this, args);
// and for forwarding
^func.functionPerformList(\value, this, selector, args);

So because the primitive _ObjectPerformList is called, this does something that is not really documented and clear:

{ |...args| args }.functionPerformList(\value, 1, 2, 3, [7, 8, 9]) // ->  [1, 2, 3, 7, 8, 9]

A shame, because we could have extended the functionality (e.g. adding a dictionary of keywords).

I remember that @rjk complained about the excessive use of ellipsis arguments many years ago.

Now I thought that, object prototyping is broken for higher order functions (Maybe, UnaryOpFunction, BinaryOpFunction and NaryOpFunction). This is (I thought) my mistake long ago, when I reproduced the signature of Function::functionPerformList for the others.

But it isn’t,



(
q = ();
q.func = { |self, x, y, z| [x, y, z] };
q.unary = { |self, x, y, z| [x, y, z] }.neg;
q.binary = { |self, x, y, z| [x, y, z] } + 2;
q.maybe = Maybe({ |self, x, y, z| [x, y, z] });

)


q.func(1, 2, 3);
q.unary(1, 2, 3);
q.binary(1, 2, 3);
q.maybe(1, 2, 3);


q[\func].functionPerformList(\value, q, 1, 2, 3);
q[\unary].functionPerformList(\value, q, 1, 2, 3);
q[\binary].functionPerformList(\value, q, 1, 2, 3);
q[\maybe].functionPerformList(\value, q, 1, 2, 3);

This works, because:

// this is how BinaryOpFunction defines it:
functionPerformList { arg selector, arglist;
		^this.performList(selector, arglist)
	}

b = { |x, y, z|  [x, y, z] } * 2;
b.functionPerformList(\value, 1, 2, 3); // ->  [2, 4, 6]
// I would have expected this to be: [1, nil, nil]

f = { |selector, arglist| arglist };
f.(1, 2, 3); // -> 1, and not [1, 2, 3]

And :drum: perhaps, for some very weird reason?
It might be due to an optimization in the compiler, that aliases

BinaryOpFunction::functionPerformList
with
Object::performList

which automatically expands the arguments.

This is how to shoot yourself in the foot by building on implicit features, which make it impossible in this case to extend the interface …

Thoughts?

2 Likes
2

What you’re talking about with how SuperCollider’s functionPerformList and _ObjectPerformList work together, and the whole thing about keeping structures intact while trying to add new features, really hits on some big ideas.

there’s a lot of focus on keeping certain structures the same even when you’re moving stuff from one place to another. This is done with some pretty abstract ideas called categories, functors, and arrows. These ideas help mathematicians understand how to keep the important bits the same when they’re changing other parts.

Now, bringing this back to SuperCollider and using functionPerformList to do some dynamic stuff, like adding a new way to handle keywords, is sort of like trying to keep the spirit of those mathematical ideas alive. But there’s a bit of a snag because the way _ObjectPerformList works isn’t super clear, which can make it hard to add on to what you’re doing without hitting some deadends (and then it’s not easy to change the language because of it’s huge community etc).

It’s kind of cool and a bit weird because there are parts of SuperCollider that almost act like they’re following these principles by having a bind method (your Maybe class for example), which is a big deal in math land. But then, they don’t fully commit to the whole package (are not functors, and don’t map the value inside them etc), like making sure everything matches up with the strict rules folks expect.

When you talked about using functionPerformList to mix and match arguments in all sorts of ways, It’s a bit like how other folks need everything to follow certain rules to keep the structure in check (and sometimes we don’t do it… because of our artistic side trying to explore a bit more). It’s out contradictory nature maybe… ))))

SuperCollider is a really interesting place to explore these ideas because it lets you play around in ways that you might not expect to see

it’s a fun place to play, actually. But it’s hard to fix things.

3

Probably in every systematic approach there are implicit assumptions that come out when the system is used or extended. Sometimes they are “good”, in the sense that you can suddenly do something you didn’t intend, sometimes they are “bad”, in the sense that you can suddenly not do something you want, because of the consequences that would have. One can try to make everything explicit at the start, but also the way of making it explicit carries implicit assumptions.

I find it an interesting question in how far pure static type systems (which have some similarities with mathematical systems like category theory or homotopy type theory) work differently than pure dynamic type systems like smalltalk.

1 Like
4

It’s pretty interesting.

SuperCollider (in the early 2000s) was at the top of its game when ad hoc blended “pure” and “impure” coding styles (please correct me if that comes from Smalltalk, but I think ST is strict).

I wonder what is the " top of our game" today))

It moved beyond the traditional Scheme approach, which combines dynamic and strictness. In this particular aspect, SuperCollider draws inspiration from pure lazy programming languages like Haskell. However, it differentiates itself by not emphasizing equational reasoning, which is as crucial as laziness for them. (They thought laziness and equational reasoning were expressions of the same core idea; ER was the basis for laziness in this world.)

Interestingly, category theory, which might seem like a “foundation” for the “pure” languages, did not initially inspire their development. Instead, it emerged later as a solution to some of their problems, effectively rescuing them. This shift occurred in the mid-1990s, with some contributions like Philip Wadler’s introduction of the monad concept into pure languages.

There is a nirvana in the middle of all this that tries to balance all those aspects. Maybe…

We can’t base everything on math and formal logic because they are also in motion. The path of pure languages in the last 30 years is an exciting story of how this world is also developing.

(On the other hand, they may seem scary but can simplify much of the messiness of the “impure” world. And hacking C++ is a testament to how crazy this can be. Holy shit.)

I don’t want a world that tries to come up with a definition of art, for example ))

5

I apologize for the detour in the discussion; feel free to move to the “whatever”/off-topic category of the forum. Yet, I believe it’s essential to underline a connection between philosophy and computing that often goes unnoticed.

Philip Wadler embodies, with his profound intellectual depth, which I’m grateful for, at the same time, what Imre Lakatos famously described as the ‘formalist’ school of mathematics (or “metamathematics”). At least from a philosophical point of view.

This distinction touches the core of our approach to computing and programming languages. What has always captivated me about your contributions, as well as those of Alberto de Campo and the broader SuperCollider community, is our subtle challenge to the metamathematical stance. It’s curious how SuperCollider united those people that way.

See his introduction to ‘Proofs and Refutations’ https://dl1.cuni.cz/pluginfile.php/730446/mod_resource/content/2/Imre%20Lakatos%3B%20Proofs%20and%20Refutations.pdf

Another take in a recent blog post: Philosophical questions about programming - Tomas Petricek

Screenshot_20240319_033305
Screenshot_20240319_0333051227×644 59.5 KB

1 Like
6

Could this be the line, because it is declared as both taking two arguments (this and the selector), and a variable arg for the passed args — as opposed to three arguments?

Meaning the real number of args is on the stack and will be looked up, and that Object’s definition for performList is perhaps incorrect and should have a var arg?

1 Like
7

Well, I think you are right about the diagnosis, but as far as I can tell, it is intended behaviour.
In the documentation, this is encoded in the following Haiku:

If the last argument is a List or an Array, 
then its elements are unpacked 
and passed as arguments.

The sclang representation is misleading:

performList { arg selector, arglist;
		_ObjectPerformList;
		^this.primitiveFailed
	}

Because it suggests that there can only be two arguments.
But this would also be a little misleading:

performList { arg selector ... arglist;
		_ObjectPerformList;
		^this.primitiveFailed
	}

Because it would discourage the use of an array as arglist.

The really complicated thing that happened and motivated the post is that this behaviour is also true for cases when the arglist is explicitly forwarded by a different methor to the object method, like:

performWhateverYouWouldLike { |selector, array|
    ^this.performList(selector, array)
}

Here the method performWhateverYouWouldLike magically becomes an n-ary method:

a.performWhateverYouWouldLike(\value, 1, 4, 3, 8, 1) // will pass on *all* arguments!

And this is bad, if you had originally kept performWhateverYouWouldLike slim, so you can add an extra argument later, like this (contrieved example):

performWhateverYouWouldLike { |selector, array, function|
    var res = this.performList(selector, array);
    ^if(function.notNil) { function.value(res) { res }
}

That would break for anyone who had already used the method in the above way.
Now, one could say this is undocumented behaviour! It is your fault! Don’t rely on hidden features!
But this is not a good excuse, especially when the class library itself uses it in this way (see functionPerformList in IdentityDictionary).

1 Like
8

I agree that it should be discussed here. It is connected. My question was not a “technical” problem, it was a philosophical (or “architectural”) problem.

For conversations inlined between solving problems, we just have to see how to organise stuff so that it doesn’t become tedious. But here it is unproblematic.

It seems to be like this, right? There is some love for a bit crazy math that many of us share.
(Tomas Petricek is great, I am very happy to be working with him and others in a project on the philosophy of programming)

Proofs and Refutations is an excellent book, I have given some seminars where we read the parts in roles and discussed it bit by bit. And indeed it is very related to what we are discussing here.

1 Like
9

I had no idea of this connection. Share the work here, would be great.

10

As a practical matter (for this, and the other “flexibility” thread), it would be good to list some safe usages, and then be clear in the documentation that usage outside of these is at one’s own risk = SC will not guarantee expected results, if something breaks, it’s on you, etc. etc…

The example at the top of this thread calls a variant of performList with multiple arguments, instead of the expected usage (arguments preformatted into an array).

// OK: perform with multiple arguments
{ |...args| args }.perform(\value, 1, 2, 3, [7, 8, 9])
-> [1, 2, 3, [7, 8, 9]]

// OK: performList with an array containing all arguments
// (for cases where you already have all the arguments in an array)
{ |...args| args }.performList(\value, [1, 2, 3, [7, 8, 9]])
-> [1, 2, 3, [7, 8, 9]]

// ??: performLst with multiple arguments
{ |...args| args }.performList(\value, 1, 2, 3, [7, 8, 9])
-> [1, 2, 3, 7, 8, 9]

So it’s tried to make an array out of the multiple arguments, and ended up flattening it. I can see where this might not be desired… but, there is a method for the multiple-argument case. If you have multiple arguments for a perform, then the right way is perform. performList is defined for a different format of arguments. (In IdentityDictionary:doesNotUnderstand, args is guaranteed to be an array, so functionPerformList is appropriate.)

So in a way, this thread is about “I did something wrong, and SC’s way of handling this is not the right way to be wrong.” And… I’m not sure it’s really worth figuring out what is the right way to be wrong, when it’s more straightforward to clarify correct usage.

hjh

11

The primitive function is the original JCM code. It reflects the general idea that the “Blame” for everything is never on the sender, but the receiver must use all kinds of tricks to deal with it.

It’s a blend of poor documentation, no comments in the language code, and the philosophy of keeping things going until the last moment; the blame is always with the interpreter. But it does not deal well with error handlers (including messages); it’s very sketchy, and that’s part of the problem.

In this case, the “blame” for not following this design is on the user. The problem is that the design does not have much logic.

12

I was thinking along similar lines after sending that –

If the objective is to pass keyword args through to downstream methods / functions, the problem is that there is no first-class representation of keyword args. Currently the interpreter dumps them all onto the stack and it’s up to the primitives to sort it out.

But what if func.value(x: 1, y: 2) compiled to:

  1. Push func into the stack.
  2. Push class KeywordArgs, \x, 1, \y, 2 onto the stack.
  3. Call new with 4 args and leave this on the stack.
  4. Then call value, where ... args collects the args into an array. Then the argument array is [ KeywordArgs([\x, 1, \y, 2) ].

It could get tricky figuring out when KeywordArgs should be passed through and when it should be unpacked, but at least it tries to disambiguate the keyword args case, and make that info available to language-side methods. Edit: Certain methods could be designated, by some new syntax, as “pass-through” methods and these would not unpack KeywordArgs, while most methods would unpack.

Edit: Another benefit of a first-class representation is that you could manufacture, in sclang code, keyword args for “perform.” Currently that’s only possible indirectly through valueEnvir. What if we could do x.perform(\something, 1, 2, KeywordArgs(*[a: 1]))? Instead of hacking something else onto what is already a hack.

We often in SC-land try to press existing structures into service for more complex cases, instead of designing a proper representation first. The idea of adding argument keywords as an additional argument to functionPerformList falls into that category, I think.

hjh

1 Like
15

Colouring functions based on whether they unpack arguments sounds like it would lead to many complexities.

That interpreter code hasn’t been touched in 20 years and would require a great deal of effort to understand - I think it’s unreasonable to expect it to change (love to be proved wrong).

1 Like
17

I’ll just say again: IMO it’s a mistake to write any code about this without designing the data structure.

As I see it, the central problem is that we are passing around argument lists in these prototyping methods, and the currently available data structures cannot disambiguate between Symbol argument values and symbolic keyword argument names.

These prototyping methods and perform variants need to be able to receive a list of arguments as data, not as argument values already bound to arguments of the prototype-support or perform method.

Hacking functions or primitives further will always leave some code smell behind, won’t it? There may be an entirely different design that would work, but I think we can all agree the current object designs are not up to this challenge.

hjh

1 Like
19

Waters have become a bit muddled, sorry if I didn’t express my intention correctly in the opening. I may add what I write here up on top, but then the conversation you just had becomes harder to understand. Anyway.

My proint was (as written in the part with the Haiku):

  • there is a clear and intended behaviour of performList (expressed in the Haiku)
  • the behaviour is not generally known (at least I didn’t know it)
  • the behaviour is inherited by means of a mechanism that is not generally known (primitive method forwarding)
  • this inherited behaviour is a rather exception to an undocumented, but (I think) assumed rule (below)
  • there are some interesting philosophical issues with this that concern implicit features and the evolution of mathematical systems as well as programming systems

implicit rule:

someMethod { |a, b ... args|
    ^this.someOtherMethod(args)
}

Calling someMethod, we do not expect a and b to be passed to someOtherMethod. In the case of method forwarding they are forwarded, as the first two elements of args.

We can work around it, I think, by doing something on someMethod that obstructs direct forwarding.

discovery:

This whole thing was discovered while writing up a better way to pass keyword arguments if necessary, but this is a separate issue (at least I’d find it too much for this thread!).

1 Like
20

Even on the issue of the Curry-Howard correspondence (isomorphism between proofs and programs, specifically the lambda calculus), I think people forget to include refutations, which have the same weight. In a way, this isomorphism also happens when they fail. The fine irony is that the beginning of computation (Church and Turing) is a reaction to the Entscheidungsproblem, to the refutation of logic as “complete”.

That’s why the community didn’t reply to my proposals on tests as proofs ))))

21

It goes far back, and around 2004, it seems there was a change in syntax. Maybe that’s why the functions accept inputs we do not know about. And ‘perform’ is very close to an ‘eval’ function in a simple interpreter; it must be part of the oldest core.

image
image1798×612 111 KB

1 Like
22

Yes, initially there was no x.method(*arrayOfArgs). The only way to do that was x.performList(\method, arrayOfArgs). And yes, perform goes all the way back.

Object prototyping was not part of the original design. It is, I think, exhibit A of implementing something by exploiting a loophole rather than designing a solution: when we complain about “object prototyping doesn’t really work,” it’s because Event / IdentityDictionary should never have been used for object prototyping!

But going back to the point – arguments as data. If an array is the way to represent argument list as a first-class object to pass around, then this brings with it the limitation that you can’t tell the difference between Symbol values and symbolic keywords. I suspect that the messiness in the interfaces around this (“just use an array, not everything on the stack, and let the primitives try to make sense of it”) is because JMc… well… maybe just ran out of time, or didn’t want to deal with it, or had to make a snap decision about how to proceed, and this handles the basic cases but didn’t handle everything.

I think it’s worth brainstorming solutions. My idea goes like this:

  • Normal case: User writes Synth(\x, args: [a: 1]) – method selector is understood, and the keyword arg exists.
    • Behind the scenes, the stack would get
      • Synth
      • \x (as an arg)
      • Instance of KeywordArgs([\a, 1]) – this is directly on the stack!
    • The interpreter, when dispatching to *new, would assign \x to defName positionally, and then iterate over the KeywordArgs array to assign args by name.
      • I think, here, it would also be possible to write Synth(\x, KeywordArgs([a: 1])) – KeywordArgs would again go directly onto the stack. So you could build method calls programmatically too. This could be quite useful and is currently not at all well supported.
  • Error case: User writes nil.xyz(100, a: 1) – method selector is not understood.
    • Behind the scenes, the stack would get
      • nil
      • 100
      • Instance of KeywordArgs([\a, 1])
    • The interpreter will manufacture a call to object doesNotUnderstand(\xyz, [100, KeywordArgs([\a, 1])]):
      • Here, I had thought maybe doesNotUnderstand would have to be marked somehow as a “pass-through method” for KeywordArgs – but in this case, I think it’s not necessary because the interpreter, when preparing the call, will wrap the KeywordArgs in the args array – so KeywordArgs would not be encountered directly on the stack, and thus not unpacked. Then doesNotUnderstand would have full access to the args contents, including keywords.
      • Then IdentityDictionary:doesNotUnderstand could call functionPerformList(\value, this, args), and the primitive would unpack KeywordArgs found in the args array.

“Brainstorming” also includes: Please post a better idea than this!

hjh

1 Like
23

Yes, that C++ function would do just the same thing. And the hacked version I posted here (that would copy Arrays if they’re inside another array already) would probably break other things.

24

Ok, so since we all move this way, let the thread cover this as well (btw. @jamshark70 thanks for chiming in! I just tried to keep things easier to overview).

If we support KeywordArgs as a way to pass keyword arguments to objects in messages, then this would have to be true all over the system. Would be great to have, let’s see how clearly it can be done.

Here is also a Pull Request that solves the problem “from the other side”. It is ready for discussion, too, was a lot of work, so be kind and careful :):

1 Like