Keeping SuperCollider evolving with minimal impact on users work

Development
1

At yesterday’s dev meeting, we had a discussion about the philosophy of making changes to SuperCollider.

We asked ourselves what kind of ‘guarantees’ we should keep in mind when introducing new versions of SuperCollider, so that those changes doesn’t negatively impact the work of people using SuperCollider.

Please note that due to the collaborative nature of the project, we’re not talking about strict guarantees a legal contract might offer.

We discussed three specific examples.

Non-breaking changes : how do we handle deprecation ?

Sometimes, developers introduce a better way to implement a functionality. For example, the Pstutter class has been replaced by Pdup, encouraging the use of the new class.

The Pstutter class is still documented, for backwards compatibility reasons (and historical purposes).

Those kind of changes accumulates artifacts inside the base code, which can make the base code difficult to apprehend, and might also confuse people who don’t know which functionality is deprecated or not when this hasn’t been documented yet. But keeping those deprecations in the base code allows backward compatibility.

For now, it seems that the implicit rule is to keep deprecated functionalities to favor backward compatibility.

Breaking changes : technical difficulties.

The example we discussed here concerns the implementation of QT 6. This would allow SuperCollider to keep evolving, and most notably to implement some new functionalities to the IDE.

But implementing QT 6 over QT 5 requires using a new version of C++, which isn’t supported on ‘old systems’. If I’m not mistaken, this would cause new SuperCollider versions to be unsupported around MacOs X, which currently supports SuperCollider.

In practical term, this means that if you need to present a piece you wrote 20 years ago, new SuperCollider versions might not be able to run your piece anymore (for hardware or OS compatibility reasons, code changes, etc).

Old SuperCollider versions are referenced and available to download, but that might not be sufficient for you to easily get you piece working.

Breaking changes : impact on artistic decisions

At some point, we might have to rewrite UGens, either to fix them, adapt them to a new functionality, etc. This is a complex subject, because rewriting an UGen will certainly have an impact on the sound of your piece if you use it, thus interfering with the artistic decisions you made when you wrote the piece.

Your whole piece might even be based on the ‘bad’ functionality we’re trying to fix.

As much as we’d like to separate artistic context from technical issues, they remain dependant on each other. How do we deal with this?

Getting users feedback on this topic would help us anticipating those problems as we keep SuperCollider evolving. Feel free to comment, especially if you can think of other related issues those examples do not cover, if you ever had a problem related to an SC update, or if you have a magic trick that can solve all these problems at once while turning lead into gold.

Regards,
Simon

4 Likes
2

(this doesn’t apply to the third point about OS depreciation, only about internal sclang issues)

Hot take

Not possible.

SC3 is dead.

It died because it didn’t have a way to mange growth and deprecation built into its design — it assumed the code it wrote would be correct forever.

This is the reason why the conversation around threading behaviour just fizzled out into nothing, not because there wasn’t agreement, but because it was impossible to implement a better design within these constraints.

@scztt’s suggestion is the only one (that I have found) that actually provides a way forward, but it involves breaking backwards compatibility, and with it, creating sc4.

I also think namespaces are a requirement, or at least, imposing that distributed quarks make use of a fake namespace by prefixing initials.

3 Likes
3

Just to clarify…

By this, I meant that supercollider’s development is dead, in that, it isn’t possible to continue developing in a sustainable manner due to the inability to do the things listed above. SC3 might continue to exist from a user’s perspective for many years — albeit in a ‘zombified’ state.

There have been similarities drawn between the switch between python2 and python3 and the mess that caused. Unfortunately (again, hot take), I think its the only way forward and currently, instead of making the tough decision to leap, sclang development has stagnated, being unable to make progress on issues that many users would like to see improved (not to diminish the contribution of bug fixers and maintainers, but those are different from the types of changes implied here). I think this might have also dissuaded people from contributing, in that, maintaining and otherwise managing a legacy codebase isn’t what contributors to a project like supercollider want to do, they want to improve user experience and create new tools for music — sclang should be structured to make this process as easy as possible. Also, I don’t think it is fair to compare the user-base of python2 to that of supercolloder, both in size and use case, as when an art piece is finished, it doesn’t tend to need continuous development, only enteral deployment: the cost for a company to move their enormous code base from python2 to python3 is substantially different from the time and cost an artist would have to put in to update their own code. This means the python2 to python3 switch is not a trivial comparison for such an update in supercollider.

2 Likes
4

NameSpaces would indeed make progress much easier as old and new Classes could be simultaneously available making the decision to fix things less fraught.

a little OT but this i how Smalltalk handles namespaces: GNU Smalltalk User's Guide: Namespaces. - IIUC there is a hierarchy of NameSpaces (basically Environments) which allows classname polymorphism

Imagine if Environments held classname mappings among other variables. Then we could write something analogous to:

a = MyCoolTuningQuark.use{ Scale(...) |> some stuff } //my nice Scale class
...
Scale() //SClangs usual annoying class

or vice versa -

cool modern sclang ;
fizz ;
buzz ;
legacyMode.push ;
a = Scale()

In any case sorry for the excursus - NameSpaces probably deserve their own thread…

3 Likes
5

In the past, we’ve encountered similar situations. For instance, Fredrik (if I’m not mistaken) once recreated UGens using the previous code but under a different name, because of this exact problem. His pieces used glitches of the old code. The issue was promptly resolved, after it was announced in the mailing list.

Effective communication is crucial. Announcing changes well in advance and keeping users informed with warning messages during an additional release cycle can make all the difference.

Additionally, the transition of the GUI from Cocoa to Qt resulted in significant code disruptions, and some of that code remains non-functional to this day. For example, ixi quark.

2 Likes
6

SC exists at all only because James McCartney had a point of view when he built it. There’s been ample disagreement lately with some aspects of that point of view, but if SC didn’t reflect a point of view, then it would be boring and nobody would use it.

An SC4 needs a point of view. That is, someone, or a small group, will have to make decisions and stick to them, even if some current users get upset about it.

I tend to agree that SC3 is a done deal… not much wiggle room. SC4 needs to break stuff (as SC3 did wrt SC2).

Of the threading discussion fizzling out, I thought there was something approaching a consensus that a Promise with an explicit .await was the way to go…? Promises seem relatively uncontroversial; I don’t see a reason to have dropped that idea.

That’s perhaps tangential though, because SC4 might not benefit from too much consensus-building (because seeking too broad of a consensus is likely to err on the side of “no, don’t break that” → nothing gets done).

hjh

7 Likes
7

Thank you for your answers!

I think we shouldn’t worry too much about ‘off topic’ on Development subjects. I have the feeling that this whole ‘SC3 vs SC4 - Class Library - Package Manager’ thing is encapsulating every specific topic we might discuss, so we can’t really act like it wasn’t there. And every time it pops out on a specific subject, that’s one more piece to the big puzzle we’re trying to solve.

Regarding the death of SC3, I’d like to disagree. I totally agree with the stagnating situation you point out, technical difficulties, and I can see you know way more than I do about such topics. But there’s people meeting to talk about this, either on this forum or on Jitsi, and they have concrete ideas to keep improving SC’s current state (QT6, IDE, Pitch Class, etc).

Those ain’t grandiose big changes that make the language revolve around itself, but they’re definitely pushing it forward. I was ‘born with Python3’, so I don’t know how things went during the transition, but I suppose that most of the improvement made to Python2 after it’s ‘death state’ got implicitly declared was still useful to implement Python3, whether it was only a cool function that could be ported to the new version, or some development that didn’t work but still got the developers to understand what they should avoid in Python3.

I’d like to propose a conjecture :

SuperCollider will always lack at least one major feature
it needs to be properly called SuperCollider

So I only started looking to the SC’s underlying architecture recently, and I’m not used to manipulate such complex concepts, I still have a naïve representation of the subject. So here’s a noob question.

It would seem, looking at jordan’s graphs, that it’s mostly a spaghetti code dependency issue?

Now let’s imagine that I take every class file, reference every methods the library is using, and copy all those methods inside every class file, then replace every occurrence of Class.method by this.method. Now, every class is a Quark and I brutally solved the issue?

What I see from the current answers is that backwards compatibility looks good, but is in fact slowing the development too much.

I’ve asked the ‘guarantee’ question, I don’t have an answer.

“We do not guarantee backwards compatibility for technical reasons. You should stick to the current version of SuperCollider you are using if you do not want to be impacted by version changes. The list of SuperCollider versions are available to download…”.

8

That graph is pretty poor, I made it wrong so don’t take it too literally. Most of the connections represent a small link to another class, one that would be easy to remove, but breaks backwards compatibility. A related issue is the design of functions like ‘as’, which does too much and places conversions in the wrong class.

9

The way Go is promising backward compatibility is interesting, this has some interesting thoughts:

2 Likes
10

Throwing in my 2¢s here (acknowledging this may sound contentious), I’m increasingly of the view that one route to a sustainable SC4 would be implementing it as a package, or more likely a collection of packages, distributed as part of a modern language ecosystem. (Small packages → small core.)

As much as I enjoy the the beauty of SC’s Smalltalk heritage, I’m up for abandoning that, if it means much of the backend infrastructure support moves into a much wider development community. (Heresy, I know!!!)

One (presumed) benefit is that SC4’s dev community would be able to focus on all the fun audio synthesis / musical signal processing stuff rather than backend logistics.

(Ok… I’m ducking now…)

1 Like
12

SuperCollider is dependent on QT? I assume the development of the IDE doesn’t influence the development of Supercollider? If it does, it sounds like a design flaw.

Concerning MacOs X, that’s a general problem of closed source products, that’s not a problem you can fix (or should try to fix probably).

FLTK is written in C++, still in development and backwards compatible. Zig is a new language which is able to (re)use C code. I’ve mentioned Go. Interesting projects to look at when the topic is compatibility.

13

Before SC with Qt, there were two GUI kits for SC: 1/ macOS had Cocoa, bundled into the old SuperCollider.app IDE (not cross platform). 2/ Windows and Linux had to run a Java GUI server (SwingOSC).

Two problems: 1/ SwingOSC was slow. It was a brilliant effort and worked well in general but it never felt responsive. 2/ There were little slippery differences between the Cocoa and Swing widgets – they were 99% compatible but I found myself at times on the wrong side of that dividing line. Not fun.

So the reason to go Qt was to have fully cross-platform GUI widgets in sclang that would be equally responsive on every platform (i.e. no more second-class user experience in Linux and Windows). This had nothing to do with the IDE.

Two devs in Europe wrote up a grant to write the Qt GUI system for SC, and included the now-current IDE in that proposal, which is how SC got where it is now.

hjh

2 Likes
14

I meant it isn’t possible to implement that into the current buffer class and maintain backwards compatibility.
Well you could implement a series of new methods with the word promise in, e.g., Buffer.readPromise(), but this just causes too much clutter to go in the Buffer class, never mind else where, and so it is better as a quark. Point was, lots of good ideas and solutions going around, but there is no way to clear out the old and replace it - that is something a more modular system with a proper dependency manager and project local quarks would give us.

15

My suggestion would have simply added an tail argument to Buffer, and the default for which would be backwards compatible. It wasn’t an end-all solution and likely incomplete, more in line with Scott proposal to address the handful of cases where it could have most impact. It seemed that your quark took on larger ambitions than were originally stated (making async behavior easier on new users), and accomplished a good deal as well as stimulated a productive discussion—both good outcomes. That’s the development process at work (hardly dead). From another perspective, your successful quark is a case for the upside of an easily extensible language? :upside_down_face:

1 Like
16

There was some pseudo-consensus at some point on the specific case of breaking changes in UGens:

3 Likes
17

And here you hit the well-known design value difference between the communities of eager devs of CCE and eager creative coders using CCEs for art. Many people straddle both communities, but we have to acknowledge their agenda is radically different. Miller Puckette has mused on this for Pd forever. Think of it what you want, it has a strong community going and pieces/installations working within that constraint. Contributors are sad it is slow-moving as you can see on their repos but that stability is needed to have a user community.

This brings me to my point: if SC forgets who it is for (artists who don’t want to code in Cpp via other more CPU efficient and generic environments) then it will properly die. or die slowly. there are a lot of computer music languages that have gone that way.

But I know also that without a lively community of devs, it will also die of deprecation.

so that interdisciplinary tense dance has to find its way slowly… which leads me to

indeed, but too much of a view and it dies when the holder stops, and/or when it goes out of fashion. Do you remember Chuck? I’m told it has a revival now but there was this dip of 10 years…

I think this is the best example. SC3 and SC4 as parallel endeavours for as long as python2 and python3 have cohabitated. Exciting enough for people to want to switch, slow enough to give them the time, and overlapping enough for gains made in the old code (both user and maintenance gains) to profit the new generation.

Anyway, SC4 definitely needs to take priority and cut on the 4983208942 ways of doing the same things, especially as most of them are not equivalent for real. a vision, a curated set. but it can be managed with some SC3 maintenance and backward compatibility ideas.

An example: asInt spits a warning (and doesn’t autocomplete iirc) and suggests using the asInteger. it irritated me enough that I eventually changed the code. Eventually (maybe now) it doesn’t work anymore, but I never noticed as I moved on - but didn’t have to do that urgently between 2 classes and 2 gigs.

sorry to state the obvious again, but we need to start to agree to disagree and move forward while not losing the most precious thing SC has: a cool critical sparkling user base who invested a lot of years making tunes with it.

6 Likes
18

Sorry, maybe I just read this wrong, but what are suggesting everyone agree upon?
Was it…

and

?
Because having a lighter core and relying more on quarks seems counter to the latter point.

Does anyone know if having two versions of sclang going at once is a realistic goal given the current number of sclang devs? How many new people would be willing to contribute to a completely new SC4 vs how many people would want to stay and maintain SC3, would existing devs want to do both?

19

There have been ~100 commits to the project from 14 developers over the last year. If you consider actual changes to code and not just documentation, this number falls to about 10 developers. I would say this is not enough momentum to maintain one version of a project this size, let alone two.

4 Likes
20

now, if we all rely on 10 core volunteer devs, SC4 seems an immense endeavour…

Maybe we should ask them. They might have no interest in SC3 maintenance, or no interest in SC4 dev, but in all cases, maybe I should be writing a grant (with @muellmusik) to fund dev time somehow… it is on my radar to try to find a way to maintain codebases that are a the root of practices…

3 Likes
21

What about a SuperCollider without GUI and stick with the core functionality. A audio programming language, with a extension to work on it in Visual Code, Atom or Scnvim. Probably magical thinking, but what if it was very very easy to hook up a OSC GUI application to synthdefs or hardware like LaunchPad or a FaderFox, who needs a integrated GUI then? Ok, I’ll leave the discussion to others.