Development Group: class library / sclang

Development
37

sclang/ClassLib Dev Meeting 2023-08-06

  • Attendees: Josh, Michael, Mike, Mike, Stefaan, Jordan

Thanks to all who attended! Please raise any issues that weren’t captured in these minutes and I’ll update here and on the minutes archive wiki.

Dev process

Started out discussing the Dev process. At a high level, we’d like to see:

  • Clear development cadence for a sense of momentum and that SC is a live project.
  • Communicate the dev process more clearly so people new to contributing have a sense of the process, and where they can start.
  • Focal “projects” to rally around, that can also drive release milestones.

Focal projects

Ideally these would consist of small working groups (or very motivated individuals) that do the varied work of large changes:

  • E.g. state the problem clearly to the community, field their ideas in response, triage and prioritize requests, set the milestones and provide periodic summaries to keep the community updated and solicit targeted feedback.
  • A couple examples of focal projects might be:
    • Cleaning up the quark system: systematically identifying and removing pain points
    • Refactoring the class library
    • Outreach and documentation around dev processes
  • An example of a project suited for a personal initiative:
    • Make the Quark library more deeply searchable (keywords, class names, etc.)
  • Not everyone wants or has time to be involved with larger focal projects, we should still be actively supporting individual initiatives for those with passion projects (more exploratory, open-ended, or less deadline-driven)

Refactoring / trimming the Class Library

Some initial discussion took place…

  • The existing directory structure in SCClassLibrary already suggests what might stay and what might better be a Quark.
  • Any group implementing this would, in the course of testing, come up with a “Vanilla SC install”, akin to pd-vanilla. This is essentially the goal of the project: to make SC itself “vanilla”.
    • So as a thought experiment: in lieu of consensus or of a successful refactor, what about the possibility of an officially maintained fork: sc-vanilla.
      • there were murmurs of skepticism, w.r.t. counterproductively growing the maintenance base, but… worth considering?

Low-hanging fruit

How do we move through “low-hanging fruit” on the ever-growing Issues list?
Meanwhile we have people eager to become new contributors!

  • New Contributors are confused on how to get started and on the review process.
    • What’s the review process? Who are reviewers?
    • How do we get more people to join the review process?
      • We have “good first issue” for contributors, how about a “good first review” for reviewers?
  • We can try to make more effort to usher in and follow through with “Good first issue” contributors.
  • This has been discussed many times on various forums, it’s worth revisiting those posts…

Documentation

The never-ending Project

Self-advocacy

We recognize dev efforts are spread thin, if you have an open issue or PR, you may need to follow through with pings/reminders.

  • Don’t be afraid of being annoying, showing enthusiasm and commitment is generally a good thing :wink:

“Projects” as swappable quark packages

A kind of meta quark that would install all required quarks for your project

  • Temporarily remove non-project quarks?
  • Would help when sharing/migrating your project to new systems.
  • There is partial support for this already, but could definitely be smoothed out.
  • Examples:
    • Your art project has x, y, z dependencies. You sit down to work on it, when opening that project you can expect all the quarks to be loaded/pulled down, and none you didn’t need.
    • For contributing to the SC, you want to make sure you’re not accidentally including Quark-dependent methods, so you load “SC Dev” Project that removes/disconnects all quarks so you can be sure you’re just compiling the default classlib.

Namespaces

…aren’t technically supported (maybe SC 4 X-P), but for Quark authors, a prefix on class names is helpful. Ex: Josh’s CTK quark.

  • keep in mind that if you modify your existing quarks, it can break for users that currently use it. Make use of the quarks versioning feature so users can check out old versions!

This brings up the question: do we have a Tutorial on publishing quarks?

  • Does it highlight all the features? In particular, versioning/tagging. #GoodFirstIssue

Search-ability of Quarks

Many quarks are black boxes… Are the current metadata fields enough to learn about the quark at-a-glance?

  • Should we add more metadata fields?
  • Should they be displayed differently in the list view?
  • Many Quark authors haven’t made use of the existing metadata features, most importantly the description! Should PUNISH them??? Well ok, no… but we can
    • do a public service campaign to encourage them to put at least a description
    • sort the Quark list in the GUI so that those with descriptions are listed first

Pitch class

There’s been a spirited discussion around inclusion of a Pitch class to SC.

  • There are lots of conventions to consider, with variations in, e.g., scales and nomenclature varying across culture.
    • We won’t capture all the possibilities… but that shouldn’t stop us from having some solid foundations for people to get up and running without having to build their own.
    • And we can keep adding in the future of course with community feedback!
    • Better yet, if the class is easily extendable so composers can easily build their own wild tonalities.
  • By one view, this is a perfect candidate for a Quark. On the other hand, the functionality is close enough to the heart of SC that a case can be made for it to be in the Core.
  • As an example of a class that is well underway to this end: CTK’s PitchClass https://github.com/joshpar/Ctk/blob/master/Ctk%20classes/PitchClass.sc
    • This is NOT a good example of how to (not) document a class (Josh!!! :stuck_out_tongue_winking_eye:)

Threading in sclang

  • Could there be ways to handle threading in the language more gracefully?
  • Examples discussed would, in some cases, alleviate the need to call the mysterious s.sync.
    • @jordan feel free to elaborate or post your Issue if you file one. EDIT: I see a related thread may be underway already.

Meta-resource: Development-meetings:-procedure-and-templates

3 Likes
38

This dovetails with some of what was discussed in the meeting just now. It’s a clear source of friction for newcomers. Specifically, I proposed in the notes above that we have a style guide for SCDocs, and a “standard way to do things” would certainly be helpful in SCDocs revisions.

wrt Patch/Def some in the meeting also expressed concerns for learners with the abstracted layers, busses in particular. So hopefully that discussion can continue here.

39

Curious about the Pitch class idea. Can someone explain what is intended here? A lot of what seems to be implied is already covered by the Scale and Tuning classes, but I think from the example linked what’s meant is a PitchClass class? Interesting to think about how to do that in a non-culturally specific way. Worth looking at Khyam Allami’s work perhaps, or even asking him to advise on this.

40

I believe the goal was to use notenames to specify pitches rather than midi numbers as most users come from a music background. One idea was to extend PitchClass, which uses lilypond names for 24edo, to use all of lilypond’s other language formats which can be switched between (perhaps with a classvar). I don’t think the goal was for this to replace Scale & Tuning, but to offer a more familiar interface with a limited number of features.

1 Like
41

Okay interesting! Will require some careful thinking in terms of how this might interact with Scale and Tuning (maybe need a PitchClasses class to define a naming system). Would be really great to support Tuning/cultural specific naming schemes!

1 Like
42

I think this was going to limited to 24EDO scales (I was there, but it isn’t my proposal).

Perhaps, before this is attempted, a standard interface could be agreed upon and actually made in supercollider (which hasn’t been the norm for some reason) something like…

AbstractNamedPitches {
   toFreq {|note| subClassResponsibility(thisMethod) }

   // moveable 'roots' (Do) can't do this
   *toFreq {|note| subClassResponsibility(thisMethod) } 

   toPseq {|repeat ...notes| subClassResponsibility(thisMethod) } 
   * toPseq {|repeat ...notes| subClassResponsibility(thisMethod) } 

}
// which might be subclassed like...
WesternNote.language = \english;

WesternNote.toPseq(inf, \a, \b, \ctqf)

// Don't know anything about Raga, just pulled a name off wikipedia so its probably wrong
~bilawalA = BilawalRagaNote(rootFreq: Notes.toFreq(\a)); // assuming 'Sa' is moveable

~bilawalA.toPseq(inf, \sa, \re, \ga)

…making it clearer how this could be implemented for (perhaps culturally) specific naming schemes/tunings such that they might be ‘hot-swapped’ at will.

But perhaps this conversation should break away from this thread.

1 Like
43

Great that there’s some enthusiasm for the pitch class idea.

This thread/channel is very general and with many ideas interleaved. I was going to suggest, as a matter of process, that it’s totally reasonable and probably more productive to start a new thread on any topics introduced here so they can gain some momentum.

2 Likes
44

Pitch Class Discussion moving over here: Note Names / Pitch Classes

I also added the tag class-lib for threads spawned from class lib dev group discussions - when we get around to the threads for refactoring and Quark system improvements lets do the same perhaps.

3 Likes
45

Just came from the main Dev Group Meeting where it was decided to consolidate the general and group dev meeting announcements in one thread. To avoid conflicts, there will be weekly meetings, interleaving the group focuses with more general meetings.

Meeting polls wil be in this thread: Developer meeting polls and the final meeting times announced here: Dev meetings schedule - #111 by MarcinP so do track those!

Next weekend’s meeting will focus on the Class Library - I’ll cross post the poll when it arrives. Look forward to seeing you all next week.

4 Likes
46

Just to be clear, maybe there was a misunderstanding. Given the fact that JITLib has become a de facto core component, I am in favour of integrating the functionality into the common library, not of moving it out.

If anyone likes to help with improving and factoring it, I am very happy to give advice.

3 Likes
47

Sorry if I haven’t understood correctly what has been proposed here but I want to add my +1 to NOT removing JitLib functionality from core. NodeProxys are my preferred style and I know others for which this is the case. Having to use a Quark for something like that would be really inconvenient…

48

Thanks Julian! I certainly have no objection to that! :slight_smile: I guess the problem is the old one of deciding what is really ‘high’ or ‘low’ level functionality, and what is ‘general’ vs idiosyncratic stuff for a broad swathe of the user base.

One could take a number of different views on this, and I’d be curious what people would say, but one obvious subset of functionality to consider is the way buses and node ordering is largely abstracted away. This for me is significant enough to be a game changer relative to the low level Node stuff, in the sense that it encourages richer ways of working because it makes an annoying problem easy! If that could be abstracted into a middle layer that was clearly part of core, I think it would provide both a more sensible basic way of working than the current node style, and a useful foundation for a variety of bespoke higher level approaches in user code. Hope that makes some sort of sense!

2 Likes
49

Hello,
At the last developer meeting we decided that the next dev meeting will be focused on the class library. Here’s the cross link to the meeting poll. Sorry for not sending this earlier!
Marcin

2 Likes
50

Hello again,
Let’s have a meeting on Sunday!
Cheers,
Marcin

1 Like
51

I am not sure if I can be woken up at the meeting time.
(2:00 am in my geographical location)

In the hope of seeing you at the meeting, or in the hope that my ideas will be mentioned by those attending the meeting, I would like to propose the following two things:

All the systems I have made support 48 EDO for the Lilypond-style pitch naming convention (note: the octave numbers follow the scientific pitch notation) and 96 EDO for their own pitch naming style. My SPN and PitchClassSet seem to be ready to use, but Notator needs more work. I will test it more this year.

While writing the code below, I extended my thought to make a DynamicMark class to get amplitude and dBFS from dynamic markings. I think this will make it easier for beginners to get started with algorithmic composition:

(
	s.waitForBoot {
		var dynamic, pitch, rhythm, legato;
		dynamic = { 
			var dynamics = (
				ffff: 104, fff: 94, ff: 84, f: 74, mf: 64, 
				mp: 54, p: 44, pp: 34, ppp: 24, pppp: 14, 
				sf: 90
			);
			dynamics[dynamics.keys.asArray
				.wchoose(([1, 2, 3, 4, 5].mirror2 ++ [4]).normalizeSum)
			]
		};
		pitch = {
			var pitchClass, alteration, octave;
			pitchClass = [\a, \b, \c, \d, \e, \f, \g].choose;
			//21.midispn; //108.midispn;
			alteration = [\s, \n, \f].choose;
			//octave = (0..8).wchoose((1, 3 .. 9).mirror.normalizeSum);
			octave = (2..6).wchoose((1, 3 .. 5).mirror.normalizeSum);
			(pitchClass ++ alteration ++ octave).asSymbol;
		};
		rhythm = {
			2.pow((0..4)).reciprocal.wchoose([1, 3, 5].mirror.normalizeSum)
		};
		legato = { [1.01, 0.9, 0.5, 0.2].choose };
		
		100.do { |index|
			var thisPitch, thisPitchMIDI, thisDuration, thisLegato, thisDynamic, dynamicChange,durationChange;
			thisPitch = ({ pitch.() }!(1..4).wchoose([2, 1, 1, 1].normalizeSum))
			.asSet.asArray.sort;
			thisPitchMIDI = thisPitch.collect { |item| item.midi };
			thisDuration = rhythm.();
			thisLegato = legato.();	
			thisDynamic = dynamic.() / 127;
			dynamicChange = 1.rand2;
			durationChange = 1.rand2;
			(1..5).wchoose([5, 1, 1, 1, 1].normalizeSum).do { |repetitionIndex|
				var thisPitchChange, thisDynamicChange, thisDurationChange;
				thisPitchChange = (-0.03, -0.02 .. 0.03).choose;
				thisDynamicChange = (-15, -14.5 .. -12).choose.dbamp * (dynamicChange * repetitionIndex * 3).dbamp;
				thisDurationChange = (durationChange * repetitionIndex / 64);
				(index.asString ++ "-" ++ repetitionIndex).postln;
				(
					instrument: \default,
					amp: thisDynamic * thisDynamicChange,
					midinote: thisPitchMIDI + thisPitchChange,
					dur: thisDuration + thisDurationChange,
					legato: thisLegato
				).play;
				(thisDuration + thisDurationChange).wait
			}
		}
	}
)

@semiquaver The idea of the function pitch above is related to your following idea in other post:

With the current SPN:

[\a, \c, \e,].collect { |item| (item ++ 4 ++ \e).asSymbol }
// even though rhythm is not here.

[\a, \c, \e].collect{ |item| (item ++ 4.rrand(6)).asSymbol }
52

Maybe SuperCollider Quarks Web Index and Add infos about each quark on website · Issue #117 · supercollider-quarks/quarks · GitHub helps?

As an occasional pd-lurker I found that the split from pd-vanilla and pd-extended to very unfavorable for the userbase of pd, see https://forum.pdpatchrepo.info/topic/12797/pd-vanilla-vs-pd-extended-why-such-a-mess - there should be a good reason to do such fork (b/c many will not follow if it breaks their code) and I fail to see a real justification for such a discontinuation - maybe so it is “nicer to maintain” - but at the same time is it? After 10+ years of software development I found monorepos the easiest to maintain - especially since sclang has no rock-solid dependency and test management system. In my experience the best way to manage a big code base is to have a really big test-suite, something where sclang could definitely improve.

Also - what would one consider core and what not? I found the programming languages with a big standard libraries (e.g. Python) to have a way more stable eco-system than something with essentially no standard library (e.g. JavaScript, the mess with jQuery, leftPad etc).
I also find Quarks quiet sophisticated, you need git installed and in my everyday “sc-teaching” I found that many people don’t have git installed and installing git, especially on Windows, is not a trivial procedure for users and many will not use any Quarks because of it.

I think it is important to keep in mind that many SC users do not have a focus on programming, so once you break their hard written code they may not follow a transition if they can’t see any gains in it. I would also not switch because there are many extensions I use which have not seen updates in 8+ years and I doubt they would be all compatible with such breaking changes.

2 Likes
53

very cool!

But it would also be good to be able search the contents of Quarks - you may come across a piece of code with DWGReverbC1C3 in it and want to get the associated Quark. If we had a Quark object database the IDE could prompt you to install.

This could also help with the annoying problem of choosing names for classes and methods and having them collide at compilation time after adding a Quark. discussion over here: Searchable "object list" for Quarks

This is a discussion worth having. It would be good to be able to distribute binaries with code for one thing - if we are downloading binaries directly perhaps we may as well do that with the code as well. at the last dev meeting @MarcinP asked whether there is an off the shelf solution that we might migrate to - @Spacechild1 suggested that we look at ~pd’s deken - I don’ know if the thought there was to adapt deken or generate something similar ?

54

Hello,

We had a “class library” dev meeting today.
@mike inquired about where should the meeting notes be posted. I personally think that notes from all meetings should still be posted on our wiki (though it could also be reorganized), but for visibility we might also post them here.

I think the most actionable items are to discuss the search paths management (for both sclang and scsynth), then the package manager solutions, and then the session/project management. See below for more complete notes from the meeting.

2023-10-01 meeting minutes

In the meeting: James S, Marcin P, Simon D, Michael W, PyoungRyang K

  • discussion on rebranding the developer tracks as proposed by Mike McCrea
    • too soon to change the name
    • table the discussion until another meeting
  • From the previous meeting:
    • idea of trimming down class library
    • would need package management system
      • various proposals floated before
        • sharing the whole class library
        • search paths
      • new package management system
    • musical notation
  • Discussion on package management
    • two separate features:
      • package managemen
      • session management
    • the packer manager needs to provide:
      • conflict resolution
      • binary files
      • searchability
        • classes
        • help files
      • versioning!
        • SC compatibility
        • OS compatibility
      • additive configuration - installing requested extensions on top of “home” extensions (e.g. for local hardware etc)
      • need to solve the patch management between sclang and scsynth
    • the session manager needs to provide:
      • transferring project to another machine
      • switching extension sets (overlap with package manager?)
      • need a format to combine sc code and requirements
      • “lock” file that lists extensions
      • discussion on how to store that data
        • change .scd format to include metadata
        • or use multiple files in a particular directory structure
      • save/recall the current state of the project
    • discussion on existing solutions: deken
    • security concerns of a possible solution
    • what are the bedrock improvements
    • note Josh’s the idea of using quarks as a surrogate for the project
    • Class library not booting when conflicts are countered - could this be addressed regardless of the package manager solution? e.g. “safe mode” boot with basic class library if that happens
    • Order of actions: discuss search paths > package manager > session manager
  • Discussion on Piping functions
  • Note names
4 Likes
55

Discussing the Package Manager, we quickly asked ourselves how much we needed to reinvent the wheel implementing it.

Since the packages are to be interacting with SC’s particularities, we have at least to tweak the connections even if we copy from an existing project.

Pure Data’s deken has been mentioned previously as a potential source of inspiration since PD and SC share many similarities.

Thinking about if afterwards, Godot Game Engine’s “assets manager” might also be worth looking at. I’ve only used it’s UI, to download user made extensions from GitHub, so it might not differ from what the Quark system is currently proposing, thus being a dead end (except maybe for features inspiration, like UI, rating system, content moderation, etc).

Thinking about it further, I realized that video games softwares might be one of the most up-to-date source of ‘package managers’ (but I’m not really comfortable with the technical details of such technologies and I might be pointing a wrong direction here).

I’m talking about ‘modding’ interfaces that now sometimes come implemented right at software release, as an essential feature. They allow users to share and download extensions through a GUI that will modify some aspects of the game (internal algorithms for optimisation, custom 3D textures, etc).

For example, I joined a ‘Project Zomboid’ server a friend of mine had created, with mods installed I didn’t have, I just had to click ‘join’, then the game interface told me ‘you need those installed’, I clicked ‘agree’, game installed the mods, recompiled and started, without any more prompt. I think that’s the kind of ‘straightforward’ experience we’d like to provide to users ?

Two downsides :

  • video games are rarely open-source…
  • they’re developed by enterprises, their methodologies/architectures might not fit SuperCollider’s ‘small size’.

So if anybody knows an open-source code which implements this kind of functionality, it might be helpful to post it there :slight_smile: .

Thanks,
Simon

1 Like
56

Note : it seems that GitHub doesn’t like hosting binary files, so it’s unlikely to be the platform where future packages are stored.
EDIT : ^ the note above is all wrong, it’s the git command line which should not be used with binary files.

Erratum : didn’t use Godot recently, so I messed it up. It’s assets library isn’t a package manager (tho it could). It’s an interface to download examples from GitHub. There’s, for example, a 2D Platformer with collisions and controls already binded, so that you just have to change the character image with a picture of your cat to have it jumping around firing bullets.

This is similar to what the current Quark GUI offers (except it would download code from https://sccode.org).

So what about extensions in Godot ? Three years ago, I had to manually copy the extension inside the correct folder inside my project folder. I don’t really know how this evolved.

But apparently there’s support for native shared-libraries at run-time. And the page says that you can distribute the extensions with the assets library, thus mitigating what I said before.

There’s also the ability to create C++ modules. This looks a bit more like what we’re trying to achieve, apparently they’ve serialized a way to bind any external algorithm to the engine via a C++ interface.

As a note, Godot has a Project Manager. I don’t know if it is really dealing with extensions, as for my own usage, it was nothing more than an integrated file browser. And project manager is still two steps ahead.

As a personal note : Godot forces you to create a project to start something new. I understand the reason why. But that means you have to take some time to setup your project every time you want to do something new.

With SuperCollider I can :

  • start the IDE
  • ctrl + b
  • { SinOsc.ar(440!2, mul: 0.25); }.play;
  • ctrl + RET

This took ten seconds, I can tune my guitar and go forward. I hope this won’t change.

Apart from that, Godot is a free C++ software to create interactive programs. Does it sound familiar ?

So now, for an open-source video game integrating a modding system : Tales of Maj’Eyal . Game written in Lua, extensions written in Lua. A bit old, and not many contributors.

The game main menu has an addon menu which lists the addons you’ve got installed. You can toggle them ON/OFF. If you click a ‘get a new addon’ button, it displays a list of user made addons (description, version requirements, rating, etc).

The game’s official extensions (i.e. written by the same team as the core engine) are addons themselves. This is similar to what we’d like to implement to reduce the ‘core’ of SC.

When downloading an addon, it is added to an ‘extension folder’ inside the game’s folder. When you start a new savefile, the addons that are currently active are written in a config file tied to this savefile. You cannot load this savefile if you are missing an addon, or if you have an active addon which is not listed in the config file (except in dev mode).