Here’s some suggestions for a way forward, a bit of an informal roadmap. ALL of these things would not need to be done to properly “rev” quarks, but I would consider this to be a rough order of operations - I would expect this all to be “beta” until everything from this list is covered.
I think doing these things would be a HUGE step forward for SuperCollider - among other things, it would open the door to moving core library functionality like JITLib to properly versioned quarks, and manage separately from the “core” classes where there’s a MUCH higher bar for backwards compatibility. Note that moving core functionality to a quark does NOT mean removing it from the default install (e.g. the “minimal PD” edition that is equally loved and loathed) - it would be straightforward to have a default list of quarks already-installed as part of the base SuperCollider app package. It would simply mean that parts of what is now the core class library could be versioned separately from the core library, and in a predictable way - this would mean e.g. @julian could make major upgrades / breaking changes to JITLib without requiring that users with “old” jitlib code be stuck on an old version of SuperCollider forever - obviously a big benefit.
- Create a new spec for Quarks, e.g. Quarks 2.0. Most existing quarks are unversioned, or don’t follow a formal versioning convention, and trying to keep backwards compatibility here would force a huge level of complexity, and probably ruin the predictability you want from a package system. This is a chance to clean house. The existing quarks system will remain available for those who want to use it.
- Include specific requirements for versioning in Q2 - this should probably be git tags with a specific semver syntax. I’d opt for git tags, because anything filesystem-based would require iteratively checking out every commit for a quarks file to discover all the versions - this is untenable for an entire archive of hundreds of quarks. Additionally, git tags are discoverable via HTTP requests to most git servers (github, bitbucket, etc), so it’s possible to do package downloads without having a local copy of git at all.
- Specify YAML as the format for the Q2 specification. Right now, quark files are executable sclang code, which means they can’t be consumed by external tools, and can’t be used if you e.g. bork your SuperCollider install by installing a bad package. sclang can already ingest yaml.
- A new quark file spec should include a few things that are missing now:
- sclang version requirements specification, in semver format ofc
- A pointer to a unit test or set of tests that validate that a quark is installed and runnable in a basic way. This can be minimal (e.g. just create a class) but IMO is a hard requirement, since a LARGE number of quarks in the archive now are not installable, cannot be instantiated in even a basic way, or will immediately render sclang unbootable. These can be used to check compatibility with new sclang versions, and discover unmaintained quarks in the archive.
- A mandatory description and author contact info field (these both exist, but they are often unused)
- A dependencies section that includes fuzzy semver specifications (e.g.
>1.2) - A “special” version designation for “1.0” quarks. This would install via the old quarks system, and could probably be accompanied by a warning “e.g. you’re installing an Q1 quark, this can break your sclang install and may not play well with other quarks”.
- A specification for a help file that has an overview description of the quark, or e.g. to it’s core class - something that will enable a user to figure out how to use it without guessing where to look in the help system.
- Create a Q2 “front-end” class that is able to execute a new sclang instance WITHOUT any quarks installed (this should be possible by ignoring the default sclang_config.yaml). This would need to be able to (a) install a list of quarks (this could be passed via a temp yaml file), (b) boot a temporary sclang instance with the “new” quarks list to ensure that the configuration is valid, e.g. quarks are not breaking compilation, and (c) run unit tests for those quarks to ensure there are no obvious failures, e.g. errors creating a core class. If this fails, the quarks install should fail and the users config yaml shouldn’t be affected - this should prevent quarks from trashing your SuperCollider install in most cases. The best process would be to install and test individual quarks in a sort of depth-first traversal of dependencies, recompiling in between - that way, you can give a specific error message e.g. “installation of quark X failed”.
- In order to facilitate different quarks configurations installed that are usable on the same system, installed quarks should be copied to a version-based path RATHER than sharing a single git repository. In other words, if a quark was cloned to
downloaded_quarks/my_quark, then installingv1.2.3should check out that tag, copy that toversioned_quarks/my_quark/v1.2.3, and point the user config file to that location. If this isn’t done, it will only ever be possible to have a single configuration for a given system, since installing quarks could change the state of quark git repos. - Proper “fuzzy” semver requirements needs both dependency resolution and a lock file representing the “resolved” quark versions - these features are common to all package managers…
- Ruby has a similar package system as SuperCollider - namely, there must be one valid version of every package for a given set of requirements (unlike e.g. node, where you can have 7 verisons of a package installed for a given project, each depended on by a different other package). I would suggest copying this algorithm direct from the ruby package manager - I also implemented a version of this for quarks in this tool: https://github.com/scztt/qpm/blob/master/qpmlib/dependency_graph.py - this would need to be translated from python to sclang, but I think the algorithm is pretty clear and doing it line-by-line should be easy.
- I’m not sure what to do about the lock file - probably the “lock” could simply be the list of “final” quark paths in the config yaml file, especially since they should include version numbers in the path. This still makes it difficult to “recover” a specific quark configuration on another system - one option here might be to add a new area in the config yaml, where quarks could be specified by name-and-version rather than by include path. This way, you could specify your “quark repository path” separately and the yaml files become less system specific.
- Existing quarks should be incrementally moved to the new format, probably starting with well-known / highly used quarks. This would probably entail adding the minimal unit tests I mentioned, but since many quarks right now don’t work anyway, this would be a good way to purge broken things and point new users to well-maintained and high-functioning quarks. All legacy quarks that are moved forward should be marked with a sclang version requirement of
~3.Xwhere X is the current minor sclang version (so, they would not be installable on versions before X). Note that because dependencies can specify legacy quarks, this means we don’t need to move ALL the dependencies of a given quark forward in order to move it forward, though obviously having legacy dependencies should be avoided as much as possible. This should give us guidance as to what quarks to move forward next. - Unit tests should be added that install and test all quarks in the catalog via the install class I mentioned earlier. sclang changes can easily be tested against the entire quarks library, to detect new regressions and quarks that don’t play will with a change. Obviously, maintained quarks should be fixed, unmaintained quarks or ones where a fix is untenable could have their version pinned to an older sclang version - this should be exceedingly rare though.
- Once a critical mass of quarks is in the new system, the old quarks could be explicitly marked as “legacy” in the UI or explicitly hidden e.g. behind an “show legacy quarks” checkbox. This should eliminate a HUGE amount of new user frustration and instability.