I’m glad to see @scztt taking the lead in organizing the dependency and backward compatibility situation. I read a lot of good ideas in there.
Couple of points:
a) Yes I’m working on a new version of Hadron, written in Rust. It’s early days but I have some ideas about how I could get some help for those interested. I’m very busy at the moment but will try to write up some documentation (if it’s not too verbose! That’s the first time I’ve ever read about someone complaining something of mine is overdocumented! ;))
b) So one thing about my (now >1 year old!) modular class library proposal - reading the reactions to it now, I think using the word core was probably a poor choice. I think most people are reading it as a value judgement around what is “essential” for the “average” SuperCollider developer. But that’s not all what I meant. SuperCollider allows for many many different workflows and usage styles. Folks may agree or not on what idiomatic sclang looks like, but as an interpreter/compiler writer, I don’t have that luxury. Any valid sclang interpreter needs to accommodate all syntactically and semantically valid programs.
Instead, my definition of “core” is explicit and technical. it turns out there are a very few classes, and a very few methods on some of those classes, that are required for the compiler/interpreter to work correctly. I think kernel is a better choice for those things that must be forever distributed with the language. I’m talking about literal data types, things like String, Symbol, Array, Function, and Integer. Also things that the interpreter uses to manage its own internals, like everything in Kernel.sc which includes Class and the Interpreter itself. That’s what I meant by core/kernel. Basically if you pared the SC class library down to the absolute minimum that you could without breaking the compiler.
c) The big objective of my proposal was to break the class library down into smaller, more manageable components, so that we could all have an easier time understanding it and evolving it. I think it’s too much to ask a single person (or even a small committee) of people to try and re-design the entire class library. Rather, I’d like to see it broken down by areas of responsibility and interest. Again, not a value judgement on importance. I also wanted to try and give the maintainers a lever to remove untested and unmaintained code from their maintenance burden. There’s so much cruft in there! So, why not find someone who is dependent on it being there, and get them to maintain it. If you can’t find anyone dependent on it, then you can delete it.
d) Super stoked to hear about other folks developing new music languages! I think that’s 3 or 4 I’ve heard about now by my own counting. Yay for a world with more, different takes on music programming!
e) I don’t think SC3 is dead, far from it! SC3 will only die when nobody cares about it anymore. I think SC3 is suffering from severe gridlock brought on by technical debt, maintainer burnout, and a monolithic architecture, but folks are motivated for finding paths forward. I’m going to publish more of my “extensive verbosity” on hadron-sclang.org, but my new Hadron roadmap I’ve been envisioning is somewhat compatible with @scztt’s proposal, and could run in parallel. Essentially I want to version 3 different pieces of the SuperCollider interpreter:
- The compiler, also sometimes called 'the frontend," which produces bytecode for execution on a virtual machine, either Ahead-of-Time (AoT) when compiling the class library or Just-in-Time (JIT) when interpreting code,
- The virtual machine (VM), which executes that compiler bytecode, and
- The “runtime environment” which supports the VM, think things like the garbage collector and all 700+ the primitives in the class library (more on that in a minute).
Using versioning for each, testing each individually, and documenting each piece, we could end up with some very interesting options for moving stuff forward.
For instance, I’m working on (1) right now, the frontend, in Rust. My goal is to produce VM bytecode that could be suitable for consumption by the current SuperCollider VM in sclang. That means we could start testing and deploying the Hadron frontend much earlier. It also means that we can test Hadron’s frontend against sclang’s.
This modular design exposes some interesting possibilities, like being able to pretty freely mix SC3 and SC4 code, on the file or even smaller level, or automatic translation of SC3 into SC4 code, or other fun things like new language frontends for the virtual machine.
For (2) I have some ideas for a byte-for-byte re-implementation of the SC3 VM, except for maybe that thing I complained about before with exception handling. I’d like to fix that in the existing SC implementation, too. More on that later, as well.
(3) is the sticky wicket. The monolithic primitives inside of SC3 are a real challenge to think about. There are over 700 primitives now in SCLang. They do all sorts of stuff, from core language functions like allocating memory for objects, to calling exotic math functions, to MIDI, file and network access, and the Qt UI system.
I think the primitives are holding the language back more than any other single problem. They’re not so easy to abandon, they represent a lot of value to sclang, I think @jamshark70 was saying on another thread that primitives are where the work really happens in sclang, and, I like 95% agree with him. The problem with the primitives is that they are completely not portable. If Hadron made really different choices with its compiler and virtual machine, I would have to re-write all of those primitives.
(3) is also tough to talk about because it’s back in the class library, and so gets mired in to some of those discussions about class library reorganization. As if the problem weren’t complex enough, there’s this whole other substrate of complexity on the C++ side of the primitives that will also require attention, and impacts some of the design choices folks have on the SC side.
Sure, we could break all of that with SC4, but I think then we’re really losing out on a ton of work that’s gone on over the years in adding important functionality to the primitives. I’d like to see more of a gentle onramp to the next thing. What if we could slowly do that? Port some things to SC4, deprecate some others, keep some things legacy but maintain a compatibility mode. I’m thinking about the break from python2 → python3. There always will be breakage, of course, but I’d rather we take our users along with us, and carve a path forward for everyone.
I’ve been going through the primitives and breaking them into buckets representing the different needs they address. I’m going to propose that we slowly evolve the primitives into a much smaller required set inside the kernel, but again need either help or more time to finish the writeup.
So as you can see, lots of plans and thoughts, and I was caught a bit off-guard by this attention to my old proposal and new activity. Happy to talk more 1-1 with folks who have questions or want to contribute, but I don’t monitor the forums actively so if you really want to get my attention either DM me or @ me in the thread. Otherwise, I’ll be back in a while with more articulated thoughts and plans for Hadron.
Toodles!
Lucile
)
, either as a Discussion or RFC over on GH.