Yes, agreed, this would be immediately useful, even if it may not solve all of the problem.
P.S. Let’s not move towards the big silo of VSCode if we can though. Don’t you think this may be feasible in the scide?
It’s also feasable in ScIDE, if somone want to add it. Though, fundamentally, ScIDE doesn’t really have an obvious path toward supporting multiple workspaces and sclang instances - obviously anything is possible, but it would entail some larger architectural reworking. But it would be easy to do things like “freezing as a little quark” via the Document class.
FWIW there is VERY little “specific” code in the VSCode extension, by design - most of it is just the general boilerplate that is required to implement a VSCode extension in the first place. This is intentional, since pushing implementation into the Language Server means those pieces will work with other clients as well (neovim, sublime, kate, etc etc).
Obviously the client that’s actually launching sclang will have to manage things like passing arguments for the conf file, so some parts of this need to be client specific. Eventually I’d like to move even this behavior to a more proper standalone language server that “wraps” sclang and can encapsulate behavior like restarts, caching, maybe a symbols database - but there are a lot of much higher priority things for the project right now.
…but perhaps this is largely because currently distributing larger projects is almost impossible!
Yes! This is important for many reasons.
Yes! This is similar the “Project as Quark” idea floated by @josh - see this thread Project as Quark - #5 by smoge
That said, I’m not so sure about the single file / yaml-in-a-comment idea - It might be simpler to for projects to simply be Quarks - that is live in an enclosing folder with a .quark file in it. All the methods you describe could still be implemented. Projects as Quarks can include multiple files, media and binaries that way and be distributable.
+1 ( SCNvim user here 
)
Ah yes, I see what you mean - I mean, this is preferable, one structure for everything. Again a thing that’s already possible now, I used to stick conf yaml’s in all my project folders for just this reason, it’s just a pain to actually use it this way.
1 Like
One idea to make it easier to move stuff between Quarks and Common: have a version number (possibly a fourth digit) that can be changed really quickly without publishing a new version. If you move something, you just push that number up by one, and thus make sure everyone is on the same level. The same probably needs to be dones ofr the Quarks of course.
2 Likes
Very much in agreement that the more we have a clear, simple procedure for this, the better off we’ll be. We’ve got to be careful about it, I think keeping version numbers meaningful (basically, what following semver provides) is also extremely critical as well. It’s important to have a very firm idea of - given a requirement and a version number, is it compatible. Strictly speaking, removing something (e.g. from Common) MUST be a major version change, because it signifies that code depending on the previous version may no longer work and would (at least potentially) need to be changed. Adding something (e.g. to a Quark) would be a minor version change, since it’s backwards compatible (or forwards, depending on your perspective 
).
Plotting this out:
- We identify
Common/Control/asScoreas something that is better as a module. - We create a new “CommonScore” quark with those files, including help and unit tests. We use some git filtering to keep the file history. This becomes the
CommonScore v1.0.0quark. - We bump the sclang version to
v4.0.0(whoo, we finally made SC4!) - We add a
CommonScore >= 1.0.0to the list of “core” quarks that probably everyone wants / come pre-installed. - [optional] We add a
CommonScore >= 1.0.0dependency to an otherwise empty quark calledCompatibility_4_to_3. If you have code for 3.0 and you want to run it in 4, this should cover removed features. - [optional] Since we’ve already made breaking changes, now is a good time to do obvious refactoring, bug fixing, and clean-up to
CommonScore. This would immediately becomeCommonScore v2.0.0(unless for some reason its backwards compatible). We want to keep our 1.0 for compatibility, but we could also consider starting our “branched” versions atv0.0.0to indicate specifically it was copied from Common. - [optional] We run a script to step through and install all quarks in the catalog and run their unit tests. Anything that compiles and tests okay is a candidate to just automatically bump it’s sclang version compatibility. It might be trivially easy to find these thing (e.g. for score stuff) or quite hard (e.g. extracting GUI). 99% (100%?) of breakage here will be fixable by simply adding a dependency to the new quark.
Obviously, bumping major versions that often is a bit disruptive for a big mega-library like the class lib - probably what we do is basically what we’re already doing: we deprecate things, let them sit in the classlib, and do a once-yearly-or-less major version bump where we remove the deprecated things to clean out.
If we want to keep the “3” as a MAJOR-major version, we can always just use the minor version for versioning the classlib - in other words, we’re now on SuperCollider 3, sclang 13.0. The meaning is the important thing. I actually prefer just bumping the major version and not clinging to our 3 - I think it’s the most clear, and it will give everyone a much needed feeling of forward progress.
Now, there’s no change in behavior for any existing quarks/projects, since they all point to 3.x. If there are new features in 4.0 you want to use, you just bump your sclang dependency to >=4.0, add the Compatibility_4_to_3 quark, and start running code. Of course, any quarks that need 3 will also need to be bumped - you can imagine that this will be annoying at first, but once the initial round of modularization happens (ideally in one major version bump) and you’ve upgraded major quarks, the remaining classlib code is very unlikely to have major changes very often, so you might never need to bump again. And, you can always opt to install the compatibility quark and do some basic testing to make sure things are okay, which should allow you to upgrade (a compatibility quark can’t solve all problems, but its easy for basic stuff like removing classes).
“Future” classlib work can also be done in a quark - for example, you could pre-branch JITLib now, and start doing major refactoring that just steamrolls the existing code by overriding existing methods. Once this is tested and stable, then we could do all of the above INCLUDING both the JITLib v0.0.0 compatibility version and your new JITLIb v1.0.0 version in one swoop.
2 Likes
I am under the impression that all the necessary elements are in place to fulfill many desires within the community.
A skillfully crafted new iteration of Quark can potentially simultaneously address a diverse array of requirements.
Quark could serve as an improved tool for language modules. It could also function as a way to distribute music pieces. A local yalm file can isolate your workspace, also working with our future LanguageServer, I imagine.
Also, when one works in a big project, one knows what quark versions are known to build together and pass the tests, at least related to that project. Being able to save this snapshot is important, since the project will be performed many times in the future and needs to be very safe.
All of this makes a lot of sense and is very exciting.
Also, I’m not sure it that’s already possible, but since there isn’t a very strict rule on git tags, one should be able to use git commit hash as version numbers too, when creating a “snapshot” for a piece.
Most package managers differentiate between dependencies and e.g. locked dependencies - where your dependencies would only have constraints (e.g; <= 3.10.0), and your final locked configuration would resolve all of those dependencies to specific git commits (ofc you can generally specify a git commit directly, but this is super inflexible since that dependency is unlikely to match anyone elses usage of it!).
If you were giving someone a piece for performance, you’d probably give them the lock file / locked dependencies, since these are a 100% deterministic snapshot. But you’d want to work locally with the version dependencies, because git commits are too inflexible - it would make it impossible to reasonably upgrade to install other quarks.
Probably for us, a lock file == sclang_conf.yaml, especially if we tracked quarks there in a slightly more generic ways than a local file path.
Incidentally, you can do this now with the existing Quarks system, Quarks.load and Quarks.save - iirc it just doesn’t work very well
1 Like
In the practical context of SuperCollider, this would be the primary utility, but even if it is the only one, it is crucial. A piece can sit for more than a year, up to many years, and it would be nice to have a deterministic environment to perform it without any headache. (as long as the sc version still runs…)
But overall, Nix has that philosophy in general. And some tools like stackage (a haskell tool) work based on snapshots with git hashes. Of course, are much more complex environments than sc.
https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/21/8.yaml
–
edit: Just to complement the point, I remember Joel Ryan telling me that he never took code he hadn’t been familiar with for a year or so to the stage (he used sc at the time, I believe he still does). There is a difference between performing on a stage with other musicians you respect, and a very different situation where we free play with ideas and all sorts of things. It is perfectly possible to reconcile the two situations in the same system.