Additively pass Quarks to supercollider (both `sclang` and `scide`) from the command line?

Questions
#1

Hi folks!

Is there a way to additively provide Quarks to the command line invocations of sclang and scide? Perhaps environment variables or command line arguments that can be used for this?

Motivation

I’m attempting to write a Nix flake for declaratively configuring reproducible supercollider environments that not only provide supercollider with a desired set of plugins (nixpkgs already handles this), but a desired set of Quarks in a ready-to-go manner too.

In order to achieve a fully declarative approach, I’d like to avoid mutating the user’s existing sclang_conf.yaml configuration and instead provide them directly to the running supercollider instance somehow, whether that be environment variable, command line argument, or applying some patch to supercollider itself (less ideal).

Existing (lacking) approach:

So far for sclang I’ve been using the -l flag to pass a custom sclang_conf.yaml configuration that declares the location of the provided Quarks. However, this is not ideal for a couple of reasons:

  1. It overrides the user’s existing sclang_conf.yaml configuration. I’d prefer not to do this in order to allow the user to provide Quarks using supercollider’s traditional approach too if they wish.
  2. The flag is not available for scide.

Any tips or tricks on how to go about additively providing Quarks to both sclang or scide invocations would be greatly appreciated!

#2

The IDE itself is responsible for choosing an sclang_conf.yaml, so it feels a bit weird to select one as a command line argument. I believe this is stored in the preferences for the IDE, so it’s possible you could provide a preferences file that specifies the expected yaml - users can always repoint in the IDE of course.

I think scang_conf.yaml is really the most correct way to specify a reproducible configuration for sclang. If we wanted the ability to have cascading quark configurations (e.g. an “environment” one, and a user one on top of that), one way to start would be to simply allow multiple yaml’s and merge them when we load. This would be straightforward to implement via allowing multiple -l flags, but probably what we’d want is an additional environment variable to specify an “environment” yaml file - something like SCLANG_ENVIRONMENT_YAML=/path/to/sclang_config.yaml. This would play better with the various SuperCollider IDE’s and editing environments.

1 Like
#3

Agreed, having the ability to merge sclang configurations sounds nice in theory.

Taking a closer look however, it’s not clear how a merge operation might behave without any sense of order/priority. E.g. what do we do in the case that one configuration specifies an exclude directory, but another specifies the same directory as an include directory?

Proposal: SCLANG_CONF_PATH

One solution that might help to clarify merge behaviour and offer more flexibility to users could be to accept a SCLANG_CONF_PATH environment variable.

Conflicts between include and exclude paths could be resolved by having the order of priority follow the order of the configurations specified in the SCLANG_CONF_PATH. The non-conflicting content of each config listed in the path would be merged.

Example

In the following:

SCLANG_CONF_PATH=/foo/sclang_conf.yaml:/bar/sclang_conf.yaml

Priority is given to /foo/sclang_conf.yaml when resolving conflicts, and the rest is merged.

Supporting Existing Behaviour

The existing behaviour in SC_LanguageConfig.cpp is:

  1. Check for config file passed via CLI.
  2. If it doesn’t exist, check user config.
  3. If it doesn’t exist, check “global” config at /etc.

Solution 1

If we were to support a SCLANG_CONF_PATH, we could get close to this behaviour by prepending the CLI argument configuration and appending the user and global configurations SCLANG_CONF_PATH. E.g. if the user passes:

SCLANG_CONF_PATH=/foo/sclang_conf.yaml

Internally, we could resolve the path as:

SCLANG_CONF_PATH=<cli-arg-conf>:/foo/sclang_conf.yaml:<user-conf>:<global-conf>

That said, this would technically be a breaking change, as the user and global configurations would be merged, whereas currently the user config overrides the global one.

Solution 2

A more accurate, non-breaking approach would be to update the behaviour to the following:

  1. Check SCLANG_CONF_YAML and retrieve the list of config paths (possibly empty).
  2. Check for config file passed via CLI.
    • If it exists, prepend it to the list.
    • Merge the list to produce final config.
  3. If CLI arg didn’t exist, check for user config.
    • If it exists, append it to the list.
    • Merge the list to produce final config.
  4. If user config didn’t exist, check for “global” config at /etc.
    • If it exists, append it to the list.
    • Merge list to produce final config.

@scztt thoughts? If 2 sounds OK, I might have a go at doing up a PR.

#4

I made a patch for SuperCollider to pick up the configuration file defined in SCLANG_CONF_YAML (if it exists). It does not yet merge with the user configuration, but I think it is a good starting point
EDIT: I managed to make it load the environment config first, then merge it with the user configuration:

The way I managed to set the includePaths might also be of interest. Quark derivations write their corresponding extension path, as well as the extension paths of their dependencies(which I pass as propagatedBuildInputs) into a file inside the nix-support.

Then, when generating the language configuration file, each Quark will have its path written inside the includePaths field.

This allows for the dependencies to be propagated without having to copy or link them to another place, avoiding the issue of duplicates.

This functionality seems to work just fine alongside other tooling such as scide, and fits well with my usage, but I am still open to suggestions.

#5

To bring this forward I created a PR on GitHub: Adding SCLANG_CONF_YAML environment variable by mraethel · Pull Request #6049 · supercollider/supercollider · GitHub.
It implements Solution 2 of @mindtree Proposal.

1 Like
#6

Hey @mraethel this looks awesome.

I noticed over on GitHub that your PR changes sclang.conf.yaml to resolve relative paths! - This would be oh so helpful for distributing code / standalones etc.

Would it be straightforward to make a second PR that includes only this change so that it can happen while ppl make up their minds about merging configs?

#7

Let’s think through the consequences of having relative paths and implementing standalones/code distributions without a build option or environment variable for passing configs.
For all the different kind of systems requiring custom default paths we need to hardcode those paths somewhere in the actual source code (probably lang/LangSource/SC_Language_Config.cpp) and have switch cases depending on the systems were compiling for. Notice that we can’t use the systems global config for that as without the ability to merge configs. A user config would override the default paths. To me this seems to be the devil in disguise as we wouldn’t gain modularity at all from it.
Now you could ask why we can’t have build options or environment variables to just specify include/excludePaths. The answer is we can, however I consider it redundant to add another option to our build system which is already handled by our config. Imo we should embrace config files as they provide modularity as well as being a way to customize SuperCollider to everyone using/packaging/developing it.
That’s why I think the solution to standalones/code distributions is not only tied to the ability to resolve relative paths but also the ability to handle configs well.
Do you agree?