Downsizing the Class Library

Development
8

Some of this is due to outdated (and occasionally bad) design. It is not possible to fix these problems as it would break backwards compatibility. By splitting things into removable chunks, the bad one can be removed, and a new easier to use piece can be put in its place. In my opinion, being able to remove and improve the old is the main benefit of a quark based approach.

1 Like
9

I agree with many of @Dindoleon’s points.
I also have some teaching (and of course learning) experience with students of various majors using the following software programs:

  • SuperCollider, Max (pd, csound, editing etc)
  • Audacity, Cockos Reaper, Audition, Ableton Live, Magix Samplitude Silver, (Logic Pro, Nuendo, ProTools)
  • Paul’s Extreme Stretch
  • MakeMusic Finale, Avid Sibelius
  • etc.

Students of computer science related subjects or those with programming experience learn SuperCollider with less of a learning curve, although some of them, who learn SuperCollider very quickly and write well written code, do not really understand the acoustic and compositional, musical basics behind the code. I think this shows that SuperCollider is somewhat biased towards the technical aspect, especially programming skills.

Musicians, including composers, generally find SuperCollider more difficult to learn than Max or PD. Musicians are usually used to using GUI based programs. I have not taught Max or PD to non-musicians, but students of other subjects usually find it difficult to use SuperCollider as well.

Where do these difficulties come from?

This difficulty starts with the installation of the plug-in and Quark, not to mention the server-client structure, which has recently been blocked by the OS due to security issues.

I would have to spend too much time checking that all the students had them installed correctly.
In contrast, GUI-based software such as Audacity and Reaper are easy to teach.

In short, using SuperCollider requires computer science skills, programming knowledge and a programmer’s mind, and reducing the core libraries and increasing the number of Quarks will increase this difficulty for the following reasons:

  1. Backwards compatibility. No need to explain again!

  2. There are too many Quarks, many of them are not well maintained and there is a lack of help documents. It is perfectly fine for the Quarks that are not listed in the official Quark directory file (https://github.com/supercollider-quarks/quarks/blob/master/directory.txt) not to provide a help document following the given help document guide. However, many Quarks in the directory files do not seem to be well maintained and do not provide the help documents, although they should. I think these Quarks are considered published, not private use for the publishers, and maintenance and help documents are important.

  3. Installing a Quark may cause compilation errors or overwrite the class method. There are too many such cases. Why does this happen? The reason is that Quarks are not maintained by central developers, but by individual needs. Reducing the size of the core library increases the number of such cases, and it becomes a vicious circle.

To overcome these three things, the skills and knowledge required are not musical or music theory things, but computer science and programming skills.

If SuperCollider is not only for the prominent users, who are not only musicians, artists and researchers, but also developers, but also for normal users (including musicians), the following is required

  • An easy to use installation method:
    • sc3 plugins should be in the core library
    • Useful quark should also be in the core library
  • Well-described help documents

Although there are no competitors to SuperCollider for algorithmic composition, it is very strange that

  • SuperCollider has no way of creating scores like nslider or the Bach library in Max.
  • It does not have its own way of creating musicXML to export the score.
  • It has no way of using pitch names, pitch class sets and pitch intervals. So getting the MIDI pitch number and frequency from the pitch name is not supported by the core class library.

So we should not only discuss how to reduce the sclang core classes, but also what features SuperCollider should have to make it more useful and practical for musicians, artists and researchers.

Many years ago, I asked about the nslider and kslider equivalents of pd in the pd user groups in Telegram. Someone wrote that I should (or could) implement them myself if I needed to. Oh… no… I hope that this kind of perspective is not in the SuperCollider developers.

I hope that SuperCollider will become a more beloved tool not only for computer science majors, but also for other majors, including musicians. (I feel that SC users are getting smaller than before. I hope this is just my misperception).

2 Likes
10

With a proper dependency system, students could just download prko’s-supercollider-flavour and all the quarks and their dependencies will be imported at once.

The difficulties might start here, but is this the main hurdle? Generally I have found supercollider’s design to cause the most issues, things like:

  • poor or no error messages;
  • old examples and inconsistent documentation, caused by a large and unmaintainable code base;
  • unexpected behaviour, e.g., (\numChannels: 2).numChannels
  • and synchronisation between server and client.

I take your conclusion that supercollider should be accessible to a wide range of people and the user base is shrinking, but I see it as the reason for changing to a quark/module based system, so that the older stuff can be depreciated and replaced by newer easier systems. While quarks are not necessary for this, the alternative is to outright break backwards compatibility, which I don’t think we’d ever reach a consensus on.

Perhaps, a quark based system could be implemented with minimal breaking changes?

Importantly, a modular system also solves some of the backwards compatibly issues supercollider might have in the future, whereas right not, there just isn’t a real mechanism to change things. I believe last breaking change was 20 years ago and done by James McCarthy? - that isn’t a sustainable system.

2 Likes
11

Thanks for the summary mike!

Could you give an example for this? I think the problem is that there are many different ways of SC - some which consider the other style “bad”, some consider the other style “bad”.

I’ll also ask the same.

Testing gets so much more complex (there are still core packages which now need “integration” tests across different versions - due to the duck-typing nature of sclang and a lack of any kind of type system this makes it really hard as there is no other way than writing many tests), existing code and extensions breaks, tutorials get deprecated, and on top SC has currently no package manager and to do this properly would either require writing a lexer in another language or add networking capabilities into sclang - both non-trivial tasks, and then one would still need to write a package manager :smiley:

Logging and documentation can already now be improved, and while unexpected behavior is definitely a thing within sclang I don’t see how this could not be fixed through quarks by starting a new dialect.

May I suggest an alternative path way to “downsize” the library by introducing namespaces, so one can tailor the environment according to ones needs. Maybe also introduce something like monkey patching to “overrule” any existing code within a scope?

IMO this would allow to have the best of both worlds: providing a backwards-compatible layer as a default environment while allowing for more custom, newer and explicit setups for more advanced users.
For .scd/.sc files the namespace from sc.legacy import * could be automatically applied to maintain compability, and .scdi/.sci files could explicitly state their imports and use for example from JitLib_v2 import Ndef or whatever.

The Legacy code scope would be put more into a maintenance mode which aims to maintain support for newer systems and do some bug-fixes, while anything outside of it could go as crazy as one would want. This would introduce of course an overhead on code-maintenance and could lead to a potential “fork” (as: why should I respect the bulky new code / legacy code?) but I think its better to build bridges between both styles than to burn existing paths.

2 Likes
12

No, it is not the main obstacle, but it does discourage students from wanting to learn Supercollider.

One more addition:
I think the new version of sclang should load classes dynamically, and the quark used should be specified in the SCD document (in each code? oh no…).

13

There’s some conflation going on in this thread between modularity of SuperCollider and easy user interface for installation. This conversation started as the first concern, and is getting sidetracked by the latter. These are separate concerns, and I want to assure people: it’s not a tradeoff at all! Both can be addressed. Changing the scope of what we call “core” doesn’t have much to do with making a complete SC setup harder to install. Pure Data gives us useful lessons to learn from, but it doesn’t refute the benefits of software modularity in general.

Modularity is probably the #1 weakness of sclang, which you’ll definitely notice if you try to install SC on any system that doesn’t have X11 or Qt (a common demand for those working on embedded systems or CI). The following desired features stick out to me:

  1. Isolated runtime: It’s fairly difficult and annoying to try to run SC in a manner isolated from any user installation. This is a basic feature of modern executables for interpreted languages (virtualenvs, node_modules), and SC doesn’t provide it. If SC had it, it would be so much easier to distribute SC code with class dependencies, make standalones, make standard installations for students as @prko asks for, switch between “profiles” of SC installations, etc. I have brought all this up before, and while unfortunately I can only talk about it without offering material help, I send my appreciation to @semiquaver for continuing to push for this one. I think it is definitely fixable without a huge amount of effort, and the gains would be huge! Distributing code - relative path for sclang.conf.yaml
  2. FFI: SC has no Foreign Function Interface that allows users (as opposed to devs) to write their own language primitives. As a result, any functionality that requires primitives has to go into core, which bloats the core SC installation in both dependencies and maintenance surface. I have worked little with this part of sclang and I’m not totally certain of the effort level here. EDIT: I should mention that the API for primitive authors is a total mess and was almost certainly never meant to be exposed to the public. Wrapping it up somehow would be the “proper” solution but would add significantly to the effort.
  3. Dynamic class library compilation: I don’t need to explain this one, and unfortunately I think this will end up being a huge project. I would guess that huge amount of sclang’s codebase (which is a massive tangle of legacy code) makes the assumption that the class library is fixed. Honestly, from my perspective it would make more sense to completely reimplement sclang than to apply this bandage :confused:

All of these things will have a big long-term impact in being able to divide up SC and making SC easier to install. The first, I stress again, requires the least amount of work of the three and has a huge payoff for users, and I believe it should be the center of near-term conversations about improving SC’s modularity.

I understand why people would recoil at the suggestion of classes being moved out of core. The good news is that that whole conversation is much lower stakes if it’s as easy as possible to distribute, install, and switch between class libraries.

Sorry to be annoying but please don’t use the term “OCD” this way, it has very little to do with the actual condition and trivializes a serious disorder: Attention Hipsters: OCD Is Not a Joke | HuffPost Life

6 Likes
14

Perhaps worth speculating about what it would take to have dynamic class loading.

Here’s a troublesome case:

MyClass {
    var <>a, <>c;
    ...
}

Now, in the middle of a session, I change it to var <>a, <>b, <>c; and reload.

Let’s assume there were some instances of the old object definition – two-slot objects. Those instances would be invalidated: 1/ they never allocated a third slot, so attempts to access c are likely to return garbage or crash; 2/ x.b would get data that should be c. IMO, then, adding slots to objects at runtime is dangerous and shouldn’t be allowed. Loading new extensions should probably not recompile class files that had been previously loaded. (To support that would require awareness in the compiler of code deltas, and a scan of all reachable slots to apply those deltas, and updating bytecodes to change slot indices … this all frightens me.)

Fortunately, we’re not talking about that. Adding extension files can only add new classes and new methods, and override methods. (+ MyObject { var newThing; } I believe isn’t allowed.) But it’s worth establishing why a dynamic full recompile is full of nasty problems that are better just avoided.

Adding an extension I think would mainly require rebuilding the “big table.” It should be possible to allocate new memory, copy existing entries, and then update with new stuff from the extension. (The big table is a 2d lookup with one axis being every class and Meta_ class, and the other axis being a union of all method names.) There may be interpreter details I’m overlooking.

Removing an extension may be riskier, e.g. a = MyObject.new and then dynamically remove the extension. Then a has no definition anymore = probable crash. So this should probably be disallowed too.

Take as speculative (especially “should be possible” comments) but the caveats may define some boundaries of what should and shouldn’t be attempted.

hjh

2 Likes
15

This is my perspective on this specific matter.

Regarding benefits, the primary deficiency (compared to modern langs) doesn’t appear to lie in the capacity to dynamically reconfigure classes, especially considering the short recompilation time now; it’s not that bad. What users genuinely desire, I imagine, is a linter to support them in the process of crafting their classes. Just this tool would avoid the need to recompile the class library too often and would prove considerably more advantageous. Also, it would prevent the necessity for language modifications (how complex wouldn’t that be, right? Can we even do this right now?) and instead utilize an external tool. Such a thing could substantially improve the situation without requiring direct language rewriting.

I see the discussion about decreasing the size of the standard library in the same way. That doesn’t seem to be the central problem, and can even create new ones. Sclang is used in a specific way; one doesn’t just write sclang scripts from the command line as we write Python scripts, so we would need a minimal sclang flavor, etc. We should not exaggerate the ‘problem’, I think. I just don’t get all this comparison with pd-vanilla as a model.

Introducing modularization would be a positive step forward. Implementing namespaces is another favorable option. The concept of “workspaces” holds considerable promise and could offer significant benefits. Additionally, pursuing an isolated runtime appears to be a readily achievable and impactful change.

1 Like
16

Smalltalk handles redefinition of Classes using a ClassBuilder tool - a new Class is built and existing instances are mutated - my understanding s that given your example: a two slot Object whose Class is redefined to have three slots, your Object would be recompiled as a three slot object using the new Class definition (a remains a, c remains c, b will be the new Class’s default value for b) . @rdd may have some experience with this procedure… No idea whether this would be practical in sclang but apparently a solved problem anyhow.

2 Likes
17

Another option could be something like a --file-watch parameter to sclang. So, every time a class file is changed and saved, automatically the the class library would be recompiled. Maybe that’s not so hard to achieve.

1 Like
18

For me, since recompiling takes basically no time, the benefit of dynamic classes would be more that sometimes it’s hard to rebuild an interpreter state that e.g. is the result of some random process…

1 Like
19

Many would argue that in the Python world (using as an example because it’s an interpreted oop lang, similar scenario), it’s preferable to restart the interpreter rather than reloading a module. As far as I understand, reloading modules may lead to unforeseen outcomes (example: old instances created before the reload retain the state and behavior of the old class definition, leads to inconsistencies in behavior between old and new instances, but I mean, it all depends of the context, of course). The autoreload tool aims to mitigate these issues by refreshing functions and parts of older classes.Yet, there are drawbacks: 1. Code replacement can fail. 2. Deleted functions prior to reloading aren’t refreshed.

Maybe tools like a --file-watch option, and a auto-evaluation of a test environment to test the class you’re hacking could be better ideas?

IDK, correct me if I’m wrong or missing something.

20

We already had something like that and nobody really used it (maybe, if it were more convenient or built in):

1 Like
21

I didn’t know that. It can be useful. Auto-reload could be more accessible, even integrated in the ide. I’ve never heard of this ruby tool before. Also, it doesn’t need to be specific to Unit Testing. It could be any custom code to create an environment.

22

Maybe something without the need to install ruby.

23

I just came across @lnihlen’s proposal from last year:

its worth reading in its entirety but I thought I would pull some quotes… She suggests three levels - Core, Middle (modules broken out from core but still shipped and supported) and finally Exterior: Quarks moved out of the project. She prefers a truly minimal Core:

The Core consists only of classes that define fundamental types, like Integer or Symbol , and services essential to the interpreter like Interpreter , Class , or Method . […] Developers can hold the test and documentation coverage to a higher standard with a smaller Core. […] The Core should be minimally small, including only enough code to support UnitTest and script validation.

she envisions automated “validation processes” for each of the modules broken out from Core:

The process can include UnitTest and integration test scripts maintained in the individual modules. Any developer making changes to the language or the Core library will need to validate the proposed change against every middle layer […] Developers can make better decisions because the validation signals are a credible source of confidence for developers (and reviewers) to gauge the impact […] of any change

She’s not worried about dependencies between the minimal core and the middle level since both will continue to be shipped. Re: the Exterior Quarks:

With improvements to library validation developers can gain the confidence needed to remove unowned modules from distribution.

She also points directly at Object being bloated (something @jordan has also posted about) suggesting that we should rely on class extensions:

Object is a clear candidate for the Core but has around 270 methods. […] Anything in Object not needed for validation should be moved to modules and added as class extensions. For example, things like isUGen and numChannels could likely go to an audio or synth module, whereas awake , beats , and clock could all move to a timing module.

1 Like
24

I find this Object situation a bit strange.

I’ve written a piece of code that ‘scans’ Class dependencies by reading the associated .sc file, and references every mention of a class name (like Jordan did before I think). This is biased because it doesn’t remove comments before doing so, and also reads the same .sc file several times if several classes are declared inside the same .sc file.

This can either only scan the class file, or the class file plus it’s parents.

This can also reference every class mentioned inside the starting class file, and keep scanning every class it encounters while doing so.

When I did this with Object, I had those results.

It would seem that Object.sc needs access to 882 classes (including itself) to be functioning properly. Due to inheritance, this means that every SC Class needs at least those classes to function properly.

I suppose those classes could be considered as the current ‘core’, where all other classes seems to be easier to be removed?

Anyway, I think that 881 dependencies is a lot for a class that is supposed to be a ‘root’ node, right? The trimming will be difficult if the trunk is too big, because the branches inherit from a lot of dependencies. But if we can somehow push those dependencies ‘away’ from the bottom of the tree, we might remove a lot of dependencies on areas where a dependency is not really needed (sorry didn’t find a better way to express this) ?

26

So to continue, the other way around :

A simple code to find a class dependants inside the Library. This only searches for the presence of the class name as a string inside .sc class files.

Due to inheritance, this isn’t enough. Finding dependencies also means looking directly for method calls inside a class. In the same manner as the one above, this code allows to search for method names (selectors) inside the class library files. It returns a list of every .sc file mentioning the method name as a string. There’s also a function to do this for every method a class provides.

This needs more work to be really usable.

  • I didn’t remove comments before searching for selectors. A comment containing the word “as” is sufficient for the algorithm to ‘think’ that the “as” function is used in the current class file.
  • Due to polymorphism, some results are collisions. It could be improved to detect method overriding, then it would need the ability to figure out if the method call is actually related to the analysed class or not.

I don’t know why you deleted your post. You proposed a particular example inside Object.sc where we could remove an (maybe) unnecessary dependency. I really think this is the right approach to figure out the difficulties we’re facing.

So I’ll be quoting you from the mail I received, tell me if you really want those words to be removed from the forum, but I think they’re useful.

We would only need to be sure that no other calls to these methods exist in Core.

Hence the code above. This still is a real thing to do, but I don’t think it’s scary. I doubt we can automate everything, but even if the numbers look ‘big’, for this particular problem, I think we’re facing a semantic issue, not a design issue. But maybe I’m wrong.

Where things could get tricky eventually is if there are calls to this method in modules other than the “synth” module…

I still don’t understand how Object can have so many ‘hard coded’ dependencies. Considering this is the root node every other class inherits from, it should have 0 dependency. Then, as you go up (or down if you live in Australia) the inheritance tree, there should only be dependencies related to superclasses.

Now I agree that derogating to this rule is a powerful way to implement things quickly and to have a modular syntax, but I think that’s how we end up in this ‘stagnation state’.

So to get back to your question, that is why I wanted to implement a tool that scans for dependencies.

  • When someone submits a new module, the module is scanned for it’s dependencies. It gets a position in the tree. This position is calculated so every dependency is placed before the module. This ensure the module can be removed without affecting the rest of the tree.

  • When the module is updated (code change), the new dependencies are scanned again. If one of the dependency is situated after the module inside the tree, the change is rejected. Ideally, the change isn’t just rejected : if there is no circular dependency between affected classes, we can rebuild the tree recursively. If there’s a circular dependency, the change is rejected. This is a breaking change.

  • This implies : removing a Class also remove all of it’s dependants. Cutting a branch from the tree means cutting the whole branch at once.

27

Before that, I’d like to point out that some .sc file declare only one class, where others declare several classes. It would seem there is no consensus about this, but I’d say that declaring one class per file and using folders to manage their position in the tree could reduce dependencies issues and increase readability. Regrouping several classes inside an .sc file could be useful too, but seems less manageable. Mixing both doesn’t seem a good idea.

This code allows to extract a class source code (as a string). This could be used to do the split mentioned above.

But since it reads the content of the .sc file, it doesn’t read additional manipulations that might have been altering the class after it has been declared, like debug.sc seems to do. I don’t really know how this file works, but this could be a solution to adapt Classes to versioning, or to dynamically add functionalities without introducing dependency.

So anyway I figured out that debug.sc is a ‘Class Library File’, but isn’t reachable using the Class class, because it is not directly related to a particular class. I don’t know if this situation can be considered a problem. I iterated through my class library folder, then removed every file that could be reached through Class, and here are the remaining files. I don’t know what the status of those files is.

28

yes sclang (like Smalltalk) allows methods to be added to classes in separate files (“class extensions”). So debug.sc adds a debug method to Object, RawArray and Collection. If (strictly hypothetical example!) a refactoring wanted to move Collection out of Core, the debug method extension for Collection in debug.sc would have to move with it.