Quark Versioning / Dependency Management

Development
10

This is the first time I realise that this is a consideration. Then the pattern system or the GUI classes may be candidates? Would it be okay not to presume anything for now?

11

With regards to the above, there is one issue with the quark system and I wonder if there could be improvement in this: Moving classes from a Quark to the core library and back is really a hassle, because it is very easy to get duplicate class errors. Having class extensions in often used Quarks will silently build up a dependency on that extension, and when it is gone, it will break code. It may be hard in further future to reconstruct the dependencies. Organising this in larger communities is even harder. Can we learn from other communities (like Haskell maybe) how to best achieve this? Or is something already planned that would solve it?

12

I think this is where having a concrete sclang version dependency (4.1 above). Suppose that a quark specifies a dependency on sclang v3.20 - this means that it has been fully tested and supports this version. Supposing that we allow breaking changes in minor sclang versions, which is SORT OF allowed now (e.g. 3.21 can remove things, rather than moving to 4.0). What does an update like this look like?

  • sclang 3.21 removes ClassX.
  • MyQuark 1.1 expects ClassX to exist, and it also depends on <=sclang 3.20
  • With no work, MyQuark will not be installable in sclang 3.21, because it does not list support this version.
  • When sclang 3.21 is being released, there are several things that could happen (assuming here that we DON’T KNOW that this quark necessarily depends on ClassX):
    • The sclang classlib team installs and runs basic unit tests for MyQuark as part of the release process for 3.21. This should immediately reveal 99% of missing-class-related problems, since these are checked at compile time (it’s also possible to detect potential missing classes at runtime, I’m doing this in QuarkEditor.quark). They could then choose to propose a fix to the quark themselves, or notify the maintainer.
    • The maintainer of MyQuark removes references to ClassX from the code, and releases a new version with sclang_version: "<=3.21", or possibly strictly limiting to 3.21 if it won’t run with older versions. This should resolve the problem more or less transparently for users.
    • The sclang classlib team moves the removed code to e.g. ClassXDeprecated.quark. The maintainer of MyQuark releases a new version with this additional dependency, allowing it to operate exactly as it did before.

I think the class extension case is the only one thats a little wierd - generally, I’d imagine that a class extension to a class that doesn’t exist is usually NOT a fatal error. IIRC this is currently only a warning anyway, but probably it would be good to have a more formal way to specify in code that a class extension is optional, and if the class doesn’t exist, no warning/error is necessary.

I think the key thing here is that the dependency, with a revised quark system, is NOT silent - it’s expressed as a dependency on a version of sclang. Obviously this is a monolithic dependency, which can create problems - this is one reason why it would be beneficial to start slicing sclang up into modules: it allows dependencies to be specified in a more specific and focused way. The above change would instead be a change to something like Core_ClassX.quark rather than sclang as a whole, which would mean a much easier upgrade path for quarks that might depend on this piece of functionality.

13

I imagine we could endlessly slice and dice the classlib into modules - it all takes effort, so it should definitely be incremental and done according to real value. But, it’s VERY easy to map dependencies in the classlib, and dependencies between quarks and classlib functionality - most of this can be done by searching for references to classes at runtime. It’s a little harder to discover dependencies on e.g. extension methods, so modularizing code that contains class extensions takes a little more manual care.

Ultimately, I think there are probably some very obvious targets of opportunity, e.g. clusters of highly inter-dependent classes that are otherwise depended on by few external things. Jitlib and patterns are obvious ones - in fact, the directory structure of the classlib already sort of exposes what our modules would be, and I wouldn’t be surprised if these were already pretty isolated in terms of dependencies.

But yes, I think this is a purely incremental process - we SHOULD assume that this kind of refactoring would be possible and easy with a “v2” quarks system, but not assume that we would need to take any particular strategy with it…

14

(splitting conversation about sclang.conf.yaml to this thread Distributing code - relative path for sclang.conf.yaml)

15

@scztt Should Quark2 be implemented in supercollider? Are we using Quark2 to update the core library, and with it, Quark2 itself? Wouldn’t a tool that doesn’t depend on supercollider running be better?
Never really understood why Quark wasn’t a separate program to begin with as you have to recompile the class library anyway.

Some people have mentioned that install git on windows is a little more involved than elsewhere. Perhaps something standalone could be made, perhaps in go using GitHub - go-git/go-git: A highly extensible Git implementation in pure Go. ?

Thought it might be fun to try and map the dependencies within the current scclasslibrary by looking for class names directly reference in the source code.

I think splitting apart the existing class library without breaking anything will be harder that expected/require many class extensions which might make it harder to reason about the code.

The regex I’ve used will match things in comments (would be nice to have a sclang parser in sclang!), but seems pretty good. It also doesn’t matching things like thisProcess which implies a dependency on Main.
Obviously everything also depends on Object as well.

Here is files that reference other files.

sc 
~getNicePath = {|str| str.asString.split($/)[6..].reduce('+/+') };

~getMentionedClasses = {|class|
	[class.asSymbol, ~getNicePath.(class.filenameSymbol.asString)] ->
	File.readAllString(class.filenameSymbol.asString)
	.findRegexp("[ ({][A-Z][a-zA-Z0-9_]+")
	.collect({|n|
		n[1]
		.reject([$ , $(, ${].includes(_)  )
		.asSymbol
	})
	.reject({|n|
		n.asClass.isNil
	})
	.collect({|c|
		[c, ~getNicePath.(c.asClass.filenameSymbol)]
	})
	.asSet
};

~r = Class.allClasses
.reject(_.isMetaClass)
.collect({|c| ~getMentionedClasses.(c, ) })
.asEvent;

t = TreeView().front;
t.columns_(["Class", "File"]);
~r.keysValuesDo({|k, v|
	var i;
	t.addItem([k[0].asString, k[1].asString]);
	i = t.itemAt(t.numItems - 1);
	v.do({|ar|
		i.addChild([ar[0].asString, ar[1].asString])
	})
});
t.canSort = true;

1
1718×682 107 KB

Here is Classes referencing other classes

sc 
(

~getNicePath = {|str| str.asString.split($/)[6..].reduce('+/+').asSymbol };

~getFileConnections = {|class|
	~getNicePath.(class.filenameSymbol.asString) ->
	File.readAllString(class.filenameSymbol.asString)
	.findRegexp("[ ({][A-Z][a-zA-Z0-9_]+")
	.collect({|n|
		n[1]
		.reject([$ , $(, ${].includes(_)  )
		.asSymbol
	})
	.reject({|n|
		n.asClass.isNil
	})
	.collect({|c|
		~getNicePath.(c.asClass.filenameSymbol)
	})
	.asSet
};


~r = Class.allClasses
.reject(_.isMetaClass)
.collect({|c| ~getFileConnections.(c, ) })
.asEvent({|a, b| (a ++ b).asSet });


t = TreeView();

t.columns = ["File", "Count" ];

~r.keysValuesDo({|k, v|
	var i = t.addItem([k.asString, nil]);
	v.do({|c| i.addChild([c]) });
	i.setString(1, v.size.asString);
});

t.canSort = true;

t.itemPressedAction({|a|
	a.postln
});

t.front;

)

2
2871×682 127 KB

As graphs…

Directories referencing directories, the arrows means ‘references’, so an incoming arrow means ‘is a dependant’.

output
output5385×1211 994 KB

And then, just for fun, classes…

outputd
outputd1920×155 44.2 KB

1 Like
16

Here’s the same thing but using the actual compiled result of the class library. This doesn’t catch dependencies on extension methods, and doesn’t catch cases where a class name is e.g. constructed dynamically from a string, but I would argue that these may not constitute proper dependencies anyway (“missing” methods are still valid calls in many cases, and construction a class from a string should always have error handling anyway, which would make it not a hard dependency). I mocked this up quickly, so it might not be totally accurate.

There are some very obvious slices that could be made. As I suspected, JITlib is pretty isolated - there are a few places in Common that depend on JITlib details, but these would be easy to eliminate. The GUI module could also easily be extracted, the only dependencies are things that should probably be class extensions anyway. We know UnitTesting is a proper module, because it USED to be a quark. SCDoc could be separated as well, it seems like the main things that depend on it are help doc GUI things, which could just be moved to SCDoc rather than the GUI folder. Quarks is highly separable, apart from some silly extensions. Predictably, most folders in Common have complex inter-dependencies - it might look different if we broke things down in some other way than by folder, but in the end I think it would probably be best to keep Common relatively monolithic anyway.

At a high level, this might look like splitting the class library up as: [Common/Audio/Bela, Common/GUI, Common/Quarks, Common/UnitTesting, Common/Unix, Common/{everything else...}, JitLib, SCDoc, Platform]. Probably pattern things could be split off as well (this would be beneficial), but at a glace it’s a bit more challenging to do.

17

Long ago I implemented a version of the Quarks system in Python, because I thought the same thing (GitHub - scztt/qpm: qpm). But, our biggest problem overall is participation, maintenance, and keeping a pace of quality-of-life improvements. Ultimately whatever we gain by implementing a Quarks system with a “more appropriate” language like Python, or using mechanisms that are more theoretically stable/reliable (nice command line tooling) - comes back to bite us as the number of possible contributors / maintainers gets trimmed down even further.

We can increase the stability quite a lot by simply doing quark installs with a “pure” out-of-process sclang instance that doesn’t load any external files, doing some basic sanity checking and rollback in case of failure, and maybe adding a “safe mode” to the IDE. All of this stuff can be added to the current Quarks system now, I think quite easily.

Removing the git dependency is also not SO hard - essentially all quarks are either on GitHub or a comparable git hosting backend. AFAIK all of these backends have public API’s to query repo tags and download files - the quark python tool I built just queries github directly to discover quarks. We can NEARLY do all of this now in sclang, but our only mechanism for doing this is the Download class, which isn’t a complete enough HTTP query API - we’d need to add some basic functionality to this for things like headers and slightly better response handling. I think this would be much easier and better than relying on an external tool, which just puts users in the familiar rat race of “oh, to install X, I need to install Y, but first that requires I install Z…”

2 Likes
18

With “silently” I meant that assuming that certain quarks are installed for most people (like the sc3plugins are now), the use of methods from these quarks will just be standard and their dependency go unnoticed. The fact that these methods depend on the quarks does not appear in the code, but in the langauge configuration. Later on, one will have no way to figure out what extensions are necessary to run a specific piece of code.

The underlying problem of sclang is one of its features: that the typical entity that is distributed in a community is a small piece of text, rather than a file or even a folder. It is an aesthetic as well, it is a unit of a small composition.

Now when we want to modularize, we have to find a way to support this common paradigm, otherwise we will lose this feature (which is central in my opinion).

When I wrote the String.include message, I had in mind that one can stick that in front of any bit of code a little easier. Still, this becomes clumsy, if you imagine that you need a handful of dependencies and their versions. Perhaps one could bundle dependencies into a single name for this.

But then, once installed, the dependencies stick around, they are not uninstalled after you run the piece of code. This means running pieces of code practically amounts to an accretion of dependencies over time.

In turn, for one’s own code, especially as a beginner, this means that you build up “silent” dependencies in the above sense.

When thinking this through it seems to me that the best thing would be to first implement the dynamic extension of the class library. As far as I know, the current method table could be replaced by a hash table, which can be extended.

Then, every small piece of code could carry with it a backback with a little library, which gets pushed onto that table and removed afterwards.

But without this, the cleanliness of modularisation might well just end in dependency hell.

2 Likes
19

Gotcha - this is a kind of low-level friction thing that is likely annoying to every SC users, and those of us who’ve been dealing with it for a million years just have tiny calcified pockets of our brains devoted JUST to managing this :slight_smile:

Honestly, a decent solution here is pretty reachable:

  • Specify a way to call out dependencies in a floating .scd file - i’m imaging a blog of yaml in a comment at the top, that mirrors the format of the Quarks dependencies field. This would act like a standalone quark in a single file.
  • Add a method to inject your current deps list into the current scd file - or e.g. resave it as a “frozen” scd file with explicit dependencies: +File { *saveQuarklet { |path, string| /* ... */ } }
  • Add a method to thaw a frozen scd, which would require (a) grabbing dependencies, and (b) creating a corresponding sclang-conf, +File { *loadQuarklet { |scdPath, targetFolder| /* ... */ } }

The only slightly clunky thing here is that you would need to run the above in it’s own sclang instance, which requires a recompile and some fiddling in the ScIDE. This is maybe solveable via deep changes to class loading as you mentioned, but honestly I think addressing in a basic way would make the workflow 100x smoother than now and it could be done immediately.

Incidentally, this kind of multi-engine, multi-configuration setup is planned for the VSCode client - once I can finish that work, having several co-existing configs running at once, and automatically picking up config from e.g. a sidecar file or workspace folder will be automatic.

1 Like
20

Re “silent dependencies” - implicit class dependencies can be detected with pretty high accuracy straight from the code, as long as your scd is compile-able. It’s also POSSIBLE with method extension dependencies but - I tried implementing this in QuarkEditor and it gave me too many false positives / misses to really feel reliable… And anyway, it’s a priori not really possible to detect method extension dependencies because of things like doesNotUnderstand - the best that can be done is to make a good guess.

21

Yes, agreed, this would be immediately useful, even if it may not solve all of the problem.

P.S. Let’s not move towards the big silo of VSCode if we can though. Don’t you think this may be feasible in the scide?

22

It’s also feasable in ScIDE, if somone want to add it. Though, fundamentally, ScIDE doesn’t really have an obvious path toward supporting multiple workspaces and sclang instances - obviously anything is possible, but it would entail some larger architectural reworking. But it would be easy to do things like “freezing as a little quark” via the Document class.

FWIW there is VERY little “specific” code in the VSCode extension, by design - most of it is just the general boilerplate that is required to implement a VSCode extension in the first place. This is intentional, since pushing implementation into the Language Server means those pieces will work with other clients as well (neovim, sublime, kate, etc etc).

Obviously the client that’s actually launching sclang will have to manage things like passing arguments for the conf file, so some parts of this need to be client specific. Eventually I’d like to move even this behavior to a more proper standalone language server that “wraps” sclang and can encapsulate behavior like restarts, caching, maybe a symbols database - but there are a lot of much higher priority things for the project right now.

23

…but perhaps this is largely because currently distributing larger projects is almost impossible!

Yes! This is important for many reasons.

Yes! This is similar the “Project as Quark” idea floated by @josh - see this thread Project as Quark - #5 by smoge

That said, I’m not so sure about the single file / yaml-in-a-comment idea - It might be simpler to for projects to simply be Quarks - that is live in an enclosing folder with a .quark file in it. All the methods you describe could still be implemented. Projects as Quarks can include multiple files, media and binaries that way and be distributable.

+1 ( SCNvim user here :wink: )

24

Ah yes, I see what you mean - I mean, this is preferable, one structure for everything. Again a thing that’s already possible now, I used to stick conf yaml’s in all my project folders for just this reason, it’s just a pain to actually use it this way.

1 Like
25

One idea to make it easier to move stuff between Quarks and Common: have a version number (possibly a fourth digit) that can be changed really quickly without publishing a new version. If you move something, you just push that number up by one, and thus make sure everyone is on the same level. The same probably needs to be dones ofr the Quarks of course.

2 Likes
26

Very much in agreement that the more we have a clear, simple procedure for this, the better off we’ll be. We’ve got to be careful about it, I think keeping version numbers meaningful (basically, what following semver provides) is also extremely critical as well. It’s important to have a very firm idea of - given a requirement and a version number, is it compatible. Strictly speaking, removing something (e.g. from Common) MUST be a major version change, because it signifies that code depending on the previous version may no longer work and would (at least potentially) need to be changed. Adding something (e.g. to a Quark) would be a minor version change, since it’s backwards compatible (or forwards, depending on your perspective :slight_smile: ).

Plotting this out:

  1. We identify Common/Control/asScore as something that is better as a module.
  2. We create a new “CommonScore” quark with those files, including help and unit tests. We use some git filtering to keep the file history. This becomes the CommonScore v1.0.0 quark.
  3. We bump the sclang version to v4.0.0 (whoo, we finally made SC4!)
  4. We add a CommonScore >= 1.0.0 to the list of “core” quarks that probably everyone wants / come pre-installed.
  5. [optional] We add a CommonScore >= 1.0.0 dependency to an otherwise empty quark called Compatibility_4_to_3. If you have code for 3.0 and you want to run it in 4, this should cover removed features.
  6. [optional] Since we’ve already made breaking changes, now is a good time to do obvious refactoring, bug fixing, and clean-up to CommonScore. This would immediately become CommonScore v2.0.0 (unless for some reason its backwards compatible). We want to keep our 1.0 for compatibility, but we could also consider starting our “branched” versions at v0.0.0 to indicate specifically it was copied from Common.
  7. [optional] We run a script to step through and install all quarks in the catalog and run their unit tests. Anything that compiles and tests okay is a candidate to just automatically bump it’s sclang version compatibility. It might be trivially easy to find these thing (e.g. for score stuff) or quite hard (e.g. extracting GUI). 99% (100%?) of breakage here will be fixable by simply adding a dependency to the new quark.

Obviously, bumping major versions that often is a bit disruptive for a big mega-library like the class lib - probably what we do is basically what we’re already doing: we deprecate things, let them sit in the classlib, and do a once-yearly-or-less major version bump where we remove the deprecated things to clean out.

If we want to keep the “3” as a MAJOR-major version, we can always just use the minor version for versioning the classlib - in other words, we’re now on SuperCollider 3, sclang 13.0. The meaning is the important thing. I actually prefer just bumping the major version and not clinging to our 3 - I think it’s the most clear, and it will give everyone a much needed feeling of forward progress.

Now, there’s no change in behavior for any existing quarks/projects, since they all point to 3.x. If there are new features in 4.0 you want to use, you just bump your sclang dependency to >=4.0, add the Compatibility_4_to_3 quark, and start running code. Of course, any quarks that need 3 will also need to be bumped - you can imagine that this will be annoying at first, but once the initial round of modularization happens (ideally in one major version bump) and you’ve upgraded major quarks, the remaining classlib code is very unlikely to have major changes very often, so you might never need to bump again. And, you can always opt to install the compatibility quark and do some basic testing to make sure things are okay, which should allow you to upgrade (a compatibility quark can’t solve all problems, but its easy for basic stuff like removing classes).

“Future” classlib work can also be done in a quark - for example, you could pre-branch JITLib now, and start doing major refactoring that just steamrolls the existing code by overriding existing methods. Once this is tested and stable, then we could do all of the above INCLUDING both the JITLib v0.0.0 compatibility version and your new JITLIb v1.0.0 version in one swoop.

2 Likes
27

I am under the impression that all the necessary elements are in place to fulfill many desires within the community.

A skillfully crafted new iteration of Quark can potentially simultaneously address a diverse array of requirements.

Quark could serve as an improved tool for language modules. It could also function as a way to distribute music pieces. A local yalm file can isolate your workspace, also working with our future LanguageServer, I imagine.

Also, when one works in a big project, one knows what quark versions are known to build together and pass the tests, at least related to that project. Being able to save this snapshot is important, since the project will be performed many times in the future and needs to be very safe.

All of this makes a lot of sense and is very exciting.

28

Also, I’m not sure it that’s already possible, but since there isn’t a very strict rule on git tags, one should be able to use git commit hash as version numbers too, when creating a “snapshot” for a piece.

29

Most package managers differentiate between dependencies and e.g. locked dependencies - where your dependencies would only have constraints (e.g; <= 3.10.0), and your final locked configuration would resolve all of those dependencies to specific git commits (ofc you can generally specify a git commit directly, but this is super inflexible since that dependency is unlikely to match anyone elses usage of it!).

If you were giving someone a piece for performance, you’d probably give them the lock file / locked dependencies, since these are a 100% deterministic snapshot. But you’d want to work locally with the version dependencies, because git commits are too inflexible - it would make it impossible to reasonably upgrade to install other quarks.

Probably for us, a lock file == sclang_conf.yaml, especially if we tracked quarks there in a slightly more generic ways than a local file path.

Incidentally, you can do this now with the existing Quarks system, Quarks.load and Quarks.save - iirc it just doesn’t work very well :slight_smile:

1 Like