Development Group: class library / sclang

Great that there’s some enthusiasm for the pitch class idea.

This thread/channel is very general and with many ideas interleaved. I was going to suggest, as a matter of process, that it’s totally reasonable and probably more productive to start a new thread on any topics introduced here so they can gain some momentum.

2 Likes

Pitch Class Discussion moving over here: Note Names / Pitch Classes

I also added the tag class-lib for threads spawned from class lib dev group discussions - when we get around to the threads for refactoring and Quark system improvements lets do the same perhaps.

3 Likes

Just came from the main Dev Group Meeting where it was decided to consolidate the general and group dev meeting announcements in one thread. To avoid conflicts, there will be weekly meetings, interleaving the group focuses with more general meetings.

Meeting polls wil be in this thread: Developer meeting polls and the final meeting times announced here: Dev meetings schedule - #111 by MarcinP so do track those!

Next weekend’s meeting will focus on the Class Library - I’ll cross post the poll when it arrives. Look forward to seeing you all next week.

4 Likes

Just to be clear, maybe there was a misunderstanding. Given the fact that JITLib has become a de facto core component, I am in favour of integrating the functionality into the common library, not of moving it out.

If anyone likes to help with improving and factoring it, I am very happy to give advice.

3 Likes

Sorry if I haven’t understood correctly what has been proposed here but I want to add my +1 to NOT removing JitLib functionality from core. NodeProxys are my preferred style and I know others for which this is the case. Having to use a Quark for something like that would be really inconvenient…

Thanks Julian! I certainly have no objection to that! :slight_smile: I guess the problem is the old one of deciding what is really ‘high’ or ‘low’ level functionality, and what is ‘general’ vs idiosyncratic stuff for a broad swathe of the user base.

One could take a number of different views on this, and I’d be curious what people would say, but one obvious subset of functionality to consider is the way buses and node ordering is largely abstracted away. This for me is significant enough to be a game changer relative to the low level Node stuff, in the sense that it encourages richer ways of working because it makes an annoying problem easy! If that could be abstracted into a middle layer that was clearly part of core, I think it would provide both a more sensible basic way of working than the current node style, and a useful foundation for a variety of bespoke higher level approaches in user code. Hope that makes some sort of sense!

2 Likes

Hello,
At the last developer meeting we decided that the next dev meeting will be focused on the class library. Here’s the cross link to the meeting poll. Sorry for not sending this earlier!
Marcin

2 Likes

Hello again,
Let’s have a meeting on Sunday!
Cheers,
Marcin

1 Like

I am not sure if I can be woken up at the meeting time.
(2:00 am in my geographical location)

In the hope of seeing you at the meeting, or in the hope that my ideas will be mentioned by those attending the meeting, I would like to propose the following two things:

All the systems I have made support 48 EDO for the Lilypond-style pitch naming convention (note: the octave numbers follow the scientific pitch notation) and 96 EDO for their own pitch naming style. My SPN and PitchClassSet seem to be ready to use, but Notator needs more work. I will test it more this year.

While writing the code below, I extended my thought to make a DynamicMark class to get amplitude and dBFS from dynamic markings. I think this will make it easier for beginners to get started with algorithmic composition:

(
	s.waitForBoot {
		var dynamic, pitch, rhythm, legato;
		dynamic = { 
			var dynamics = (
				ffff: 104, fff: 94, ff: 84, f: 74, mf: 64, 
				mp: 54, p: 44, pp: 34, ppp: 24, pppp: 14, 
				sf: 90
			);
			dynamics[dynamics.keys.asArray
				.wchoose(([1, 2, 3, 4, 5].mirror2 ++ [4]).normalizeSum)
			]
		};
		pitch = {
			var pitchClass, alteration, octave;
			pitchClass = [\a, \b, \c, \d, \e, \f, \g].choose;
			//21.midispn; //108.midispn;
			alteration = [\s, \n, \f].choose;
			//octave = (0..8).wchoose((1, 3 .. 9).mirror.normalizeSum);
			octave = (2..6).wchoose((1, 3 .. 5).mirror.normalizeSum);
			(pitchClass ++ alteration ++ octave).asSymbol;
		};
		rhythm = {
			2.pow((0..4)).reciprocal.wchoose([1, 3, 5].mirror.normalizeSum)
		};
		legato = { [1.01, 0.9, 0.5, 0.2].choose };
		
		100.do { |index|
			var thisPitch, thisPitchMIDI, thisDuration, thisLegato, thisDynamic, dynamicChange,durationChange;
			thisPitch = ({ pitch.() }!(1..4).wchoose([2, 1, 1, 1].normalizeSum))
			.asSet.asArray.sort;
			thisPitchMIDI = thisPitch.collect { |item| item.midi };
			thisDuration = rhythm.();
			thisLegato = legato.();	
			thisDynamic = dynamic.() / 127;
			dynamicChange = 1.rand2;
			durationChange = 1.rand2;
			(1..5).wchoose([5, 1, 1, 1, 1].normalizeSum).do { |repetitionIndex|
				var thisPitchChange, thisDynamicChange, thisDurationChange;
				thisPitchChange = (-0.03, -0.02 .. 0.03).choose;
				thisDynamicChange = (-15, -14.5 .. -12).choose.dbamp * (dynamicChange * repetitionIndex * 3).dbamp;
				thisDurationChange = (durationChange * repetitionIndex / 64);
				(index.asString ++ "-" ++ repetitionIndex).postln;
				(
					instrument: \default,
					amp: thisDynamic * thisDynamicChange,
					midinote: thisPitchMIDI + thisPitchChange,
					dur: thisDuration + thisDurationChange,
					legato: thisLegato
				).play;
				(thisDuration + thisDurationChange).wait
			}
		}
	}
)

@semiquaver The idea of the function pitch above is related to your following idea in other post:

With the current SPN:

[\a, \c, \e,].collect { |item| (item ++ 4 ++ \e).asSymbol }
// even though rhythm is not here.

[\a, \c, \e].collect{ |item| (item ++ 4.rrand(6)).asSymbol }

Maybe SuperCollider Quarks Web Index and Add infos about each quark on website · Issue #117 · supercollider-quarks/quarks · GitHub helps?

As an occasional pd-lurker I found that the split from pd-vanilla and pd-extended to very unfavorable for the userbase of pd, see https://forum.pdpatchrepo.info/topic/12797/pd-vanilla-vs-pd-extended-why-such-a-mess - there should be a good reason to do such fork (b/c many will not follow if it breaks their code) and I fail to see a real justification for such a discontinuation - maybe so it is “nicer to maintain” - but at the same time is it? After 10+ years of software development I found monorepos the easiest to maintain - especially since sclang has no rock-solid dependency and test management system. In my experience the best way to manage a big code base is to have a really big test-suite, something where sclang could definitely improve.

Also - what would one consider core and what not? I found the programming languages with a big standard libraries (e.g. Python) to have a way more stable eco-system than something with essentially no standard library (e.g. JavaScript, the mess with jQuery, leftPad etc).
I also find Quarks quiet sophisticated, you need git installed and in my everyday “sc-teaching” I found that many people don’t have git installed and installing git, especially on Windows, is not a trivial procedure for users and many will not use any Quarks because of it.

I think it is important to keep in mind that many SC users do not have a focus on programming, so once you break their hard written code they may not follow a transition if they can’t see any gains in it. I would also not switch because there are many extensions I use which have not seen updates in 8+ years and I doubt they would be all compatible with such breaking changes.

2 Likes

very cool!

But it would also be good to be able search the contents of Quarks - you may come across a piece of code with DWGReverbC1C3 in it and want to get the associated Quark. If we had a Quark object database the IDE could prompt you to install.

This could also help with the annoying problem of choosing names for classes and methods and having them collide at compilation time after adding a Quark. discussion over here: Searchable "object list" for Quarks

This is a discussion worth having. It would be good to be able to distribute binaries with code for one thing - if we are downloading binaries directly perhaps we may as well do that with the code as well. at the last dev meeting @MarcinP asked whether there is an off the shelf solution that we might migrate to - @Spacechild1 suggested that we look at ~pd’s deken - I don’ know if the thought there was to adapt deken or generate something similar ?

Hello,

We had a “class library” dev meeting today.
@mike inquired about where should the meeting notes be posted. I personally think that notes from all meetings should still be posted on our wiki (though it could also be reorganized), but for visibility we might also post them here.

I think the most actionable items are to discuss the search paths management (for both sclang and scsynth), then the package manager solutions, and then the session/project management. See below for more complete notes from the meeting.

2023-10-01 meeting minutes

In the meeting: James S, Marcin P, Simon D, Michael W, PyoungRyang K

  • discussion on rebranding the developer tracks as proposed by Mike McCrea
    • too soon to change the name
    • table the discussion until another meeting
  • From the previous meeting:
    • idea of trimming down class library
    • would need package management system
      • various proposals floated before
        • sharing the whole class library
        • search paths
      • new package management system
    • musical notation
  • Discussion on package management
    • two separate features:
      • package managemen
      • session management
    • the packer manager needs to provide:
      • conflict resolution
      • binary files
      • searchability
        • classes
        • help files
      • versioning!
        • SC compatibility
        • OS compatibility
      • additive configuration - installing requested extensions on top of “home” extensions (e.g. for local hardware etc)
      • need to solve the patch management between sclang and scsynth
    • the session manager needs to provide:
      • transferring project to another machine
      • switching extension sets (overlap with package manager?)
      • need a format to combine sc code and requirements
      • “lock” file that lists extensions
      • discussion on how to store that data
        • change .scd format to include metadata
        • or use multiple files in a particular directory structure
      • save/recall the current state of the project
    • discussion on existing solutions: deken
    • security concerns of a possible solution
    • what are the bedrock improvements
    • note Josh’s the idea of using quarks as a surrogate for the project
    • Class library not booting when conflicts are countered - could this be addressed regardless of the package manager solution? e.g. “safe mode” boot with basic class library if that happens
    • Order of actions: discuss search paths > package manager > session manager
  • Discussion on Piping functions
  • Note names
4 Likes

Discussing the Package Manager, we quickly asked ourselves how much we needed to reinvent the wheel implementing it.

Since the packages are to be interacting with SC’s particularities, we have at least to tweak the connections even if we copy from an existing project.

Pure Data’s deken has been mentioned previously as a potential source of inspiration since PD and SC share many similarities.

Thinking about if afterwards, Godot Game Engine’s “assets manager” might also be worth looking at. I’ve only used it’s UI, to download user made extensions from GitHub, so it might not differ from what the Quark system is currently proposing, thus being a dead end (except maybe for features inspiration, like UI, rating system, content moderation, etc).


Thinking about it further, I realized that video games softwares might be one of the most up-to-date source of ‘package managers’ (but I’m not really comfortable with the technical details of such technologies and I might be pointing a wrong direction here).

I’m talking about ‘modding’ interfaces that now sometimes come implemented right at software release, as an essential feature. They allow users to share and download extensions through a GUI that will modify some aspects of the game (internal algorithms for optimisation, custom 3D textures, etc).

For example, I joined a ‘Project Zomboid’ server a friend of mine had created, with mods installed I didn’t have, I just had to click ‘join’, then the game interface told me ‘you need those installed’, I clicked ‘agree’, game installed the mods, recompiled and started, without any more prompt. I think that’s the kind of ‘straightforward’ experience we’d like to provide to users ?

Two downsides :

  • video games are rarely open-source…
  • they’re developed by enterprises, their methodologies/architectures might not fit SuperCollider’s ‘small size’.

So if anybody knows an open-source code which implements this kind of functionality, it might be helpful to post it there :slight_smile: .

Thanks,
Simon

1 Like

Note : it seems that GitHub doesn’t like hosting binary files, so it’s unlikely to be the platform where future packages are stored.
EDIT : ^ the note above is all wrong, it’s the git command line which should not be used with binary files.


Erratum : didn’t use Godot recently, so I messed it up. It’s assets library isn’t a package manager (tho it could). It’s an interface to download examples from GitHub. There’s, for example, a 2D Platformer with collisions and controls already binded, so that you just have to change the character image with a picture of your cat to have it jumping around firing bullets.

This is similar to what the current Quark GUI offers (except it would download code from https://sccode.org).


So what about extensions in Godot ? Three years ago, I had to manually copy the extension inside the correct folder inside my project folder. I don’t really know how this evolved.

But apparently there’s support for native shared-libraries at run-time. And the page says that you can distribute the extensions with the assets library, thus mitigating what I said before.

There’s also the ability to create C++ modules. This looks a bit more like what we’re trying to achieve, apparently they’ve serialized a way to bind any external algorithm to the engine via a C++ interface.


As a note, Godot has a Project Manager. I don’t know if it is really dealing with extensions, as for my own usage, it was nothing more than an integrated file browser. And project manager is still two steps ahead.

As a personal note : Godot forces you to create a project to start something new. I understand the reason why. But that means you have to take some time to setup your project every time you want to do something new.

With SuperCollider I can :

  • start the IDE
  • ctrl + b
  • { SinOsc.ar(440!2, mul: 0.25); }.play;
  • ctrl + RET

This took ten seconds, I can tune my guitar and go forward. I hope this won’t change.

Apart from that, Godot is a free C++ software to create interactive programs. Does it sound familiar ?


So now, for an open-source video game integrating a modding system : Tales of Maj’Eyal . Game written in Lua, extensions written in Lua. A bit old, and not many contributors.

The game main menu has an addon menu which lists the addons you’ve got installed. You can toggle them ON/OFF. If you click a ‘get a new addon’ button, it displays a list of user made addons (description, version requirements, rating, etc).

The game’s official extensions (i.e. written by the same team as the core engine) are addons themselves. This is similar to what we’d like to implement to reduce the ‘core’ of SC.

When downloading an addon, it is added to an ‘extension folder’ inside the game’s folder. When you start a new savefile, the addons that are currently active are written in a config file tied to this savefile. You cannot load this savefile if you are missing an addon, or if you have an active addon which is not listed in the config file (except in dev mode).

There’s a bit of misunderstanding: you generally shouldn’t commit binaries to your git repository (unless absolutely necessary) because git is design for text files. In particular, you shouldn’t commit build artifacts.

However, hosting platforms, such as GitHub or GitLab, allow you to host binaries in the form of assets, typically as part of releases. For example: Releases · supercollider/supercollider · GitHub. This could be a viable solution if a centralized hosting server is not desirable. (After all, someone needs to run and maintain that server.)

If I remember correctly, jrsurge said he once pushed a binary file on GitHub and that there was no way to remove it, that he had to email GitHub for them to remove it.
Maybe this asset solution is indeed the right way to do so (I’m a total beginner in GitHub, just trying to relay informations as much as I can here). I think that depending on a server should be avoided at all cost.

Be sure not to confuse git (= the version control system) with GitHub (a git hosting platform)!

Once you commit a binary file to git, it is part of the history; you would need to rewrite the whole history to remove it.

On the other hand, GitHub assets are just arbitrary files on a webserver, they can be freely added, removed and replaced.

The nice thing about assets is that you can just download them; they are completely independent from the git software.

Ok thank you for the clarification. I’m still confused with these. On top of this, I’ve subscribed to GitHub to be able to contribute to SC, and I think this have changed recently, as most of the commands on the wiki are using git, but to connect to GitHub with git, I had to create a token or whatever, so I didn’t go through this process. I used gh instead, the GitHub command line equivalent of git, which is easy to setup. I think this should be transparent in term of contributing, as long as I figure out the equivalent of the wiki’s commands. This out of context, but might help other contributors getting started.

Again, git is the underlying version control software, GitHub is just the (most popular) hosting provider.

You can certainly connect to GitHub from the command line and use regular git commands. GitHub offers a dedicated desktop app (GitHub Desktop) and CLI (gh); some people may find them convenient, but they are not necessary.

but to connect to GitHub with git, I had to create a token or whatever, so I didn’t go through this process.

IIRC, GitHub removed simple password authentication in favor of “access tokens”. However, you can also connect to GitHub via SSH (which I would personally recommend).

I think the wiki should stick with regular git commands, but we might need to add some further setup instructions. The wiki does point to some GitHub links, but they are not easy too follow if you’re not already familiar.

agree!

The issue here was with git-lfs which has been proposed elsewhere as a way to extend the Quark system to handle binaries (and which I have also found to be a nightmare!)

1 Like