Current priorities in improving the documentation?

Hi all,
(and sorry about the wall of text; it’s mostly meant to give an account of the issues I’m already aware of and thus to save people time pointing them out again.)

I’ve been meaning to contribute to sc in some way for a while and now at last I seem to have some time on my hands during the next few months (unless I miraculously get a full-time job, get evicted, or the revolution finally comes). I’m not originally a techy guy but a so-called “mUsIc ThEoRiSt” (soon with doctoral degree and all, thanks neoliberal university that shall not be named), so I guess the best way to make myself useful is to work the contents of the documentation/help (contents as opposed to, e.g., implementing more HTML tags or formatting in the SCdoc).

Obviously I’ve looked at the github issues and I’ve browsed the forum a bit (the “learn to contribute” tag in particular) to get a general sense of where development is at. My impression is that there isn’t some grand general roadmap, but rather (apart from select projects like the treesitter grammar or the LSP) just a large number of small-ish or medium-sized things, and that is to some extent the nature of the unfunded open-source beast.

Nonetheless, especially as I don’t have experience contributing to projects like this, I don’t just want to start randomly submitting PRs out of the blue, even if it’s “only” for help pages. (There are of course simple things like occasional typos but I would also think they are mostly irrelevant.)

So I did want to check in with maintainers/devs as to what the current priorities are with regards to the help.

Two relevant bits I’ve found so far:
@prko’s old thread from 2023

and the associated PR with some more recent activity

and also, I’ve found this bit by @jamshark70 enlightening:

the amount of help you get from the documentation is (partly) proportional to the size and depth of your mental map of the documentation. After 20+ years, my mental map of SC help is pretty thorough. But, if I have to look up something about LilyPond, if a web search fails, I’m likely to end up asking their mailing list because I don’t have even a tenth of a sense of where things are in their manual.
Improving documentation structure and searchability can make it faster to build the mental map, but it always takes time. In that sense, documentation will never be transparent enough.

There’s somewhat of a double constraint implies here imho because on the one hand, any change to the docs shouldn’t upset users’ existing mental maps too much, but on the other hand, there are some idiosyncrasies or near-rendundancies in the way the doc is currently organized that make it tough to build that map in the first place.
For instance, there seem to be two distinct pattern tutorials: The practical guide and the Streams-Patterns-Events tutorial. They do not cross-reference each other, and the former is in the Streams-Patterns-Events category and the latter in the Tutorial collection.

Realistically, I imagine it might help to a) introduce more crossreferences between, e.g., different help files (“for a slightly different way to do this, see x”; “this is different from x as explained in y”), and b) to supersede the “categories” system (where presumably a page can’t be part of both the tutorial and the pattern category) with something more like tags (where it can). That way it becomes easier to introduce new organization without breaking older layers of organization.

Finally, I recall that when I started out learning sc I had little experience with other programming languages (sc is what got me into computer stuff, essentially), and a lot of it makes more sense now that I do; and learning resources for other languages like python and cpp are obviously abundant. This makes me wonder if some of the more conceptual things about supercollider that may pose a hurdle for not-so-tech-inclined musicians might benefit from references to relevant non-sc resources, even wikipedia. (I recall having trouble wrapping my head around what an environment is, for instance).

I don’t know how this sounds to folks with more experience doing this sort of work; that’s why I’m asking.
Cheers!

2 Likes

Actually, it can: from SCDocSyntax help – " CATEGORIES:: list – A comma-separated list of document categories" – if it’s restricted to only one category, then there’s no reason to allow comma-separation.

Yes!

One of the biggest weaknesses in SC help is a lack of documentation focusing on crucial topics: use cases, how-to’s, etc. We tend to focus on class and method documentation, which splits up topics of interest into a little piece here and another little piece over there. One example might be: how to map MIDI (or OSC, or GUI) controllers onto synth parameters, when the MIDI values might change at a time when there aren’t any synths. In general, (IMO) control buses are usually the best way, but it’s hard for incoming users to see that right away. This is a common recurring question on the forum, which means the documentation isn’t providing sufficient guidance that’s easy to locate.

Here, I’m not sure which is “better.” The Practical Guide (PG, by me), I think, tries to go a bit slower and to choose entry points that are more directly relevant to most users’ questions. Streams-Patterns-Events (SPE, by James McCartney)… I reopened it just now and it makes sense to me… but I’m already fairly well expert in patterns and streams. I remember feeling overwhelmed by the pace when I was starting out. (Perhaps a different writing philosophy is at play: SPE warns readers not to skip ahead; I guess my thought is, instead of providing a warning, why not organize the material and pace it so that readers are less tempted to skip ahead? :wink: Anyway, it’s frequently the case that developers are not the best authors of documentation – I’m not a real developer, not really – an above-average hacker, yes, and probably a better tech writer.)

The Categories could be revisited here, sure.

hjh

2 Likes

In case it’s helpful for you, I’ve been thinking about what’s in my current “mental map” of helpfiles, particularly crucial information not contained in class documentation. (I’m not a developer/maintainer but a somewhat long time user.) Of course everyone’s priorities are different, but here are the helpfiles that have come up for me so far:

Language
Symbolic notations
Syntax shortcuts
Operators
Adverbs for binary operators
J concepts in SC
Control structures
List comprehensions
Partial application
Literals
Tour of Special Functions
Multichannel expansion
Classes
Writing classes

Server
Client vs server
Scheduling and server timing - could use an update, does not include preferred s.bind notation
Server command reference
Non-Realtime Synthesis (NRT) - we need better NRT guides in general, or I haven’t yet found the good ones
Tour of UGens

Events, patterns
recursive_phrasing
Event, Pbind often I need to check both helpfiles to find the right information
Event Types helpfile is actually less complete than Pattern Guide 08: Event Types and Parameters
In general, always look at Pattern Guide first

GUI
List of GUI classes
Resize behavior

We could also use better guides for MIDI and OSC, I think. Two of the most powerful and compelling reasons to use SC, even beyond sound processing/synthesis, as a conduit to/between other time-based systems.

I would be interested to know what helpfiles other people have on speed dial.

And now I’m finding a concrete failure of the Categories system - I couldn’t tell you for sure which category many of these help files live in, even though I literally just looked them up…

Also in browsing the help system just now, I encountered a lot of helpfiles I’ve never seen before, such as User FAQ (which contains a helpful section on using ‘if’ style logic in a SynthDef) that could be very useful if they were more complete.

And in general from my experience I agree that there are a lot of redundancies (the two series of pattern tutorials is the prime example) with lack of cross-referencing, also a lot of older and incomplete pages. There is so much information to be organized, and so many different potential readers of that information (beginners, experts, programmers, musicians, composers, …), and also so much historical significance (there’s a compelling reason to keep JMC’s pattern tutorial even though the Pattern Guide is probably the one new users should read; I appreciate the “SuperCollider 3 versus SuperCollider 2” page even though I’ve been using SC for 15 years and never touched SC2) – but maybe it makes sense to quarantine the historical documents and update/consolidate/streamline the relevant information for a modern audience. (although it will be endlessly up for debate what that actually means…)

Since the pace of new SC developments/paradigms is currently pretty slow (not meant as a bad thing), I imagine this effort would stay relevant for quite a while.

Sorry for rambling :slight_smile:

1 Like

Also, just thinking, maybe it’s instructive to include in the documentation examples of translating patches from other ecosystems… for example these are some things that might feel more intuitive to do in Pd than SC, and could be gateways to several important best practices / cookbooks

I used to think the same. But some people encouraged me to do so. So I picked a random small issue and got through my first PR. I learnt a lot. Made less mistakes on my second PR. Then I felt ready for ‘bigger’ changes.

I highly encourage you to pick something simple and submit it (like: there’s no documentation page for this topic, I’ll redact one, or this Class has undocumented methods, I’ll document them).

You’re not alone on that, current maintainers are kind and will discuss with you during the PR. You’ll gain a lot of experience in no time, which I think you wouldn’t get otherwise.


I’m also interested in updating the documentation. I’ve redacted a GUI tutorial (which is available in current develop branch, which is currently broken :slight_smile: ), see GUI: Tutorial Introduction.

I was impressed by Godot’s tutorial which I found very beginner friendly (it might have changed since then though). I think SuperCollider’s technical documentation is ok, but its ‘step-by-step documentation’ is a bit messy, especially for beginners. IMO, having a clear ‘learning path’ available directly from the help browser would be a huge plus. But it isn’t trivial to design.

2 Likes

Sometimes, if well planned, a task that is too big for one person could be more easily done by a few people as medium or several people for even-less demanding tasks.

Maybe we just need a little organization and should use the brains we already have here. Well-planned, a big task can become several small tasks.

2 Likes

Thanks for all the input so far, keep it coming!

Alright, on my end I’ll focus on the categories and crossreferences things to get started. Seems like a relatively low-hanging fruit, especially if pages can belong to multiple categories. Also might help me get my own “mental map” up to speed. I’ll probably start with pattern related stuff because that’s what I’m currently trying to understand better anyway.

It just occurred to me that it would be useful to have a graph of all (or a subset of) the help files and how they interconnect, kind of in the way that folks made graphs for some subsections of wikipedia. I suspect that it isn’t all that difficult to do, if one leaves out the bells and whistles… but what do I know (Wiki Graph).
On the visualization end, I can probably figure out how to draw this in python’s matplotlib without spending an excessive amount of time, but I don’t know the networkX library that people use to prepare the data, nor how forthcoming the scdoc format is with respect to a parsing in terms of its links. Doesn’t look particularly scary though.

I also think more content on interaction with / translation from/to other systems is a good idea. The OOP intro to me seems like a candidate for an article that can benefit from a reference to tutorials for other languages. (I remember that a few years ago, when I encountered the OOP-related help files, I just didn’t really understand what the point of it was; so motivation / use cases would be good here.)

EDIT: oh wait a second, I should look at @Dindoleon’s gui tutorial first, maybe that addresses some of those issues.

Yes! But this reminds me of another beginner hurdle with sc, namely that the GUI isn’t nearly as straightforward as in pd/max, can’t as easily make a slider, connect it to stuff and go; at least the general tutorials don’t mention it. Maybe it would be good in some of the tutorials to use some brutally simplified subclass of the current sliders and checkboxes, at least (with a lot of things preset, including the controlspec and .action function, etc.?)

1 Like

Such as… EZSlider perhaps. This underscores the need for GUI documentation – we already have a simplified slider class – you just couldn’t find it easily. (Slider help doesn’t link to EZSlider.)

An important point that GUI topic documentation needs to stress – well, this is kind of a matter of opinion, but I have good reasons for it (and, following this principle avoids the “operation cannot be called from this Process” problem) –

Your code should be able to do everything that it needs to do in code, without GUI. Then the GUI is a shell on top of that. I.e., don’t use valueAction – because valueAction means that essential logic is locked up inside a GUI object. Or, “how can I get the value of a slider in a pattern?” If you have to get the value of a slider in a pattern, then the logic isn’t independent of the GUI = design mistake. The GUI should represent a value, not serve as storage.

I’d love to see a GUI tutorial series that starts with best practices for simple cases, and builds up to MVC.

As an aside… I find it’s usually better with GUIs to think in terms of building a composite behavior using one or more of the provided GUI widgets. Subclassing is (IMO) usually not the right way to go for GUIs. (Oddly enough, we’re very familiar with the concept of a “pseudo-UGen” – a class that uses several existing UGens to get a more complex behavior – but for GUIs, it seems we have a mental bias toward subclassing. But “pseudo-widgets” – like EZSlider! – have strong advantages.) So perhaps that tutorial series could demo a composite GUI object.

hjh

1 Like

@girthrub and everybody interested in improving the documentation.

Having a graph of documentation files/architecture would indeed be helpful. I can help with that if needed.


See this comment about improving SC documentation. In short:

  • do we improve current documentation by replacing ‘short help files’ by bigger tutorials? Do we remove previous ‘short doc’ afterwards?
  • what’s the difference between ‘guides’ and tutorials’, i.e., more generally, should we define a clear and documented terminology that defines help files ‘types’ (class helpfile, concept overview, concept guide, how to do this music thing guide, etc), which would allow us to say things like ‘hum, Patterns have a good ‘concept guide’, but we should redact a concise ‘concept exemplification’’ more easily?
  • How does the help file tree appear on the help browser home page?

I’ve released a widget quark which tries to address some of current GUI issues. Every class method is documented if not internal. When needed, help file starts with paragraphs about every distinct concept/issue the widget might have (see GMTapTempo). Help files end with examples, and, if relevant, copy-paste templates (see GMFaderSlider). There’s also help files for usage and key concepts (see FM Drone Example and Sequencer Interface example).

This is the methodology I tried to apply to make the quark as usable as possible. Maybe this could be a reference point for improving SC’s documentation ?


Last, getting back to Godot’s tutorial, I tried to mimic it to provide a SC guide for newbies. This is redacted in french, but I think you can figure out how it goes looking at code example or using an automatic translator. You can find it here. It is an introduction to OOP and covers ScIDE’s interface, audio warnings, interpreter usage, Synth & SynthDef, etc, so that beginners can make sound as quickly as possible.


Sorry for the wall of text.

2 Likes

Slightly OT but there is something about the EZ classes that does not work with LineLayout (VLayout or HLayout) or maybe I am missing something?

// works as expected, you get 4 sliders in one view
v = View().layout_(HLayout(*4.collect{ Slider() })).front

// does not work as expected, 4 sliders in 4 different views
v = View().layout_(HLayout(*4.collect{ EZSlider() })).front
1 Like

Yep, EZGui is broken with containers. If you want even more challenge, try inserting a Plotter within a Layout :slight_smile: .

2 Likes

about that “brutally simplified” gui object (I wrote subclass, but it can really be whatever):

I think I meant that (originally) in the context of a beginner’s tutorial, and then got confused and connected it to the gui tutorial. I imagine that in the context of a beginner’s tutorial, it’s justifiable to have some training wheels that violate a number of design principles. VERY naively, I was thinking of something like a gui object that can quite simply serve as an argument to a UGen, where it returns not itself but the current value of the GUI element; more or less like a slider in pd when connected to an input of something. EZGui, broken or not, isn’t quite that EZ.

Anyway, I don’t think this is terribly important.


I might get the graph thing together today or tomorrow.


My two cents: I can imagine to have a “short doc” at the beginning of the file, followed by a tutorial and more in-depth examples; as long as these remain clearly labeled, so users know what they need and don’t need to look at. (Again, the idea behind this is to not break any “mental maps”). Even better if that (e.g., short doc then tutorial then in-depth examples) is a format shared by lots of help files, so readers would be familiar with that.

I think as far as tutorials/guides are concerned, the terminology and type may be less important than the intended audience. For instance, flagging things as e.g. “beginners”, “advanced”, “cs background”, “music background” etc…
One thing I am noticing, though, is that, unlike some other languages, there seems to be no clear distinction here between “help” and “reference”. I’m not sure if that distinction is very clear cut elsewhere, but the general trend is probably for the reference to be more concise; then “reference” in sc would just be the class documentation whereas “help” would refer to everything else, more or less? Also I don’t think this particular distinction is necessarily very important, but it does have to do with different types of help files, and aligns with practices in other languages. (I think lilypond does this, although it remains confusing… and numpy/matplotlib/etc.)

2 Likes

Some high level organization does seem wanted!

I can remember as a learner just poking through the Browse tree in the help reading everything one at a time and never being able to remember which bits I had already read! And still unsure where to look for pages I remember seeing.

Right now if you browse for “Reference” you get:

Which seems like a pretty random presentation!

for ref here is the TOC for pd’s Manual:
http://msp.ucsd.edu/Pd_documentation/

Lilypond:

With my quarks, I’ve been doing this for about 20 years now.

s.boot;

g = GenericGlobalControl(\freq, nil, 440, \freq);
g.gui;  // this depends on crucial-lib; I should probably change that

a = { SinOsc.ar(g.kr, 0, 0.1).dup }.play;

a.release;
g.free;

It never quite entered the common lexicon though.

hjh

1 Like

I have a similar construct N() that I use to tune synth values, with an emphasis on ease of use, for example

{ SinOsc.ar(N(440), 0, 0.1).dup }.play

will allocate a bus and show a small window with a GUI number box to edit.

If you make several at once they will tile, and you can name them so you can tell what’s what:

{ SinOsc.ar(N(440, "freq"), 0, N(0.1, "amp")).dup }.play

It would be a nice touch to add a slider with a control spec…

Here’s my attempt to replicate the Pd examples I posted before in vanilla SC

// example 1, frequency slider
(
~window !? { ~window.close };
~window = Window("Oscillator").onClose_({ ~synth.free }).front;

~slider = EZSlider(~window, 
  bounds: Rect(10, 10, 380, 30), 
  label: "freq", 
  controlSpec: \freq.asSpec,
  action: { |view|
    ~synth.set(\freq, view.value);
  }
);

~synth = { |freq = 440, amp = 0.1|
  SinOsc.ar(freq).dup(2) * amp;
}.play;
)



// example 2, slider showing real time value of oscillator
(
~window !? { ~window.close };
~window = Window("Phase viewer").onClose_({ ~synth.free }).front;

~slider = EZSlider(~window, 
  bounds: Rect(10, 10, 380, 30), 
  label: "value", 
  controlSpec: [0.0, 1.0].asSpec
);

~synth = {
  var sig = LFSaw.ar(1).range(0, 1);
  var trig = Impulse.ar(30); // 30 fps
  SendReply.ar(trig, '/value', sig);
  nil; // silent
}.play;

OSCdef(\value, { |msg|
  //msg.postln;
  defer { ~slider.value = msg[3] };
}, "/value");
)

Trying to be as clear and compact as possible, and so not really MVC, although hopefully avoiding the more egregious pitfalls. Also still far less elegant than the Pd patches, which is why we make our own solutions for ourselves… But I wonder, is there a “better” / more recommended way to do these things in vanilla SC, without adding complexity?

That’s interesting ! I’ve re-redacted your first example:

(
// Declare the variables we need
var window, slider, synth;

// Create our window
window = Window(
	"Oscillator",
	Rect(
		0, 0,
		380, 100
	)
);

// Free synth when the window is closed
window.onClose_({ synth.free });

// Create our slider
slider = EZSlider(
	window, 
	bounds: Rect(10, 10, 380, 30), 
	label: "freq",
	action: {
		var value = slider.value;
		
		// Map sliders value, which is between 0 and 1 on a linear scale,
		// between 20 and 20000 (hertz) on an exponential scale
		value = value.linexp(0, 1, 20, 20000);
		
		// Update our synth
		synth.set(\freq, value);
	}
);

// Set our slider starting value (this time, mapping is inverted)
slider.value_(440.explin(20, 20000, 0, 1));

// Load our synthdef to the server:
SynthDef(\oscillator, { |out = 0, amp = 0.1, freq = 440|
	var snd = SinOsc.ar(freq, mul: amp);
	
	// Duplicate our mono signal to convert it to stereo
	snd = snd.dup(2);
	
	// Send the signal to the soundcard
	Out.ar(out, snd);
}).add;

// Wait for the synthdef to be loaded, and play the synth.
// This needs to be forked:
fork {
	// Synchronize with the server
	s.sync;
	// When synced, synthdef has been loaded,
	// play the sound
	synth = Synth(\oscillator);
};

// Close the window if user uses CmdPeriod
CmdPeriod.doOnce({ window.close; });

// Finally, bring the window front
window.front;
)

This is how I would’ve redacted the example if it was to appear within a help file. I think this is an opportunity to discuss example redaction usage, and to identify some problems that might arise due to SC’s complexity.

Of course, we lack context. I assumed reader is comfortable with PD, but he might be a beginner in terms of code.


~window !? { ~window.close };

I think those kind of lines shouldn’t appear within the documentation (unless the help file is explicitly labelled as advanced). Imagine you’re a beginner, with no SC knowledge, trying to figure out what it does. The fact that ~window is a global variable is implicit. The fact that !? is a shortcut for notNil is implicit. The fact that this line is a safety measure when user re-evaluates the code is implicit. I think this adds too much complexity to the example, and only makes sense if reader knows how sclang works, leading to the hen & egg situation. Of course, this is good code. But if the reader is a beginner, I think he’s already lost.


~window = Window("Oscillator").onClose_({ ~synth.free }).front;

I think one method call per line, with comments, is preferable:

// Create our window
window = Window("Oscillator");

// Free synth when the window is closed
window.onClose_({ synth.free });

// Bring the window front
window.front;

controlSpec: \freq.asSpec,

Back to the implicit issue. I find ControlSpec awful, and I would’ve used a Slider instead of an EZSlider just for the sake of avoiding to have to explain this… On my GM Quark, controls implement specs directly : GMFaderSlider().min_(20).max_(20000).scale_(\exp).


~synth = { |freq = 440, amp = 0.1| SinOsc.ar(freq).dup(2) * amp;}.play;

This is an important question. Should the documentation present this shortcut? The server/client distinction is super important, and a big source of confusion. I think that the tedious add a SynthDef > play a Synth process makes it apparent. This duo is at the heart of SC usage, especially with Patterns. As such, I think it should be preferred within the documentation, even if it adds more code.



So I don’t know. There’s a lot of confusing things within SC, that only makes sense afterwards. I tried to make a contrast between beginner friendly code (my code) and good SC code (yours). I think we need to find a way so that readers are somehow first presented easy and detailed code, up until they’re presented powerful ‘real life’ code.

3 Likes

Very good points about syntactic shortcuts, one method per line with explanations etc.

So here’s a preview of the graph thing, still somewhat a work in progress (on the visual/interactive side) … the screenshot is of the busiest nodes in jamshark’s Pattern Guide. The data is all there but it isn’t quite as insightful as I hoped, it’s just a bit much to digest at once :slight_smile:
I suspect the matplotlib drawing might not be the fastest; my computer isn’t exactly new but the wikipedia webgraphs ran ok on it, whereas just zooming in the mpl interface is slow. I guess it might help to not draw all of the nodes and edges right away but successively (both for responsiveness and for insightfulness).

Might as well dump the code as it stands here:

import re
from pathlib import Path

import matplotlib.pyplot as plt
import networkx as nx

# insert absolute path to supercollider
# omit leading /
# initially limit this to a subfolder of helpsource, because getting the whole help (for now) results in an unreadable mess of a graph
helpsourcesubfolder = Path("supercollider/HelpSource/Tutorials/A-Practical-Guide")
helpsourcepath = Path("supercollider/HelpSource/")

pathlist = list(helpsourcesubfolder.rglob('*.schelp'))
# pl is small: there are only 1120 schelp files!

# relational terminology: (not sure if consistent w graph theory):
# outnode is the help file that contains a link TO innode; outedge of a node is a link in the helpfile TO somethign.
# innode is a helpfile TO WHICH something links; inedge is a link FROM something to a helpfile.

# build the following dict: {<outnodename>: {<innodename>: weight} }

edgedict = {}
for x in pathlist:
    outedges_w_methods = re.findall(r'link::(.*?)::', x.read_text())
    outnode = str(x.parent.relative_to(helpsourcepath) / x.stem)  # without the schelp extension
    edgedict[outnode] = {}  # make a dict
    for m in outedges_w_methods:
        innode = re.split(r'#', m)  # returns list of len 1 or 2, depending
        # innode[0] is always a helpfile's name (without extension)
        if innode[0] not in edgedict[outnode]:
            edgedict[outnode][innode[0]] = 1
        else:
            edgedict[outnode][innode[0]] += 1

# now turn it into an edgelist of tuples (outnode, innode, weight) that can be passed to nx
edgelist = []
nodelist = []
for outnode, innodes in edgedict.items():
    nodelist.append(outnode)
    for i, w in innodes.items():
        edgelist.append((outnode, i, w))
        if i not in edgedict and i not in nodelist:  # i.e., i is only linked to, doesn't link out to anything
            nodelist.append(i)  # we'll need it in the list nonetheless


G = nx.MultiDiGraph()
G.add_nodes_from(nodelist)
G.add_weighted_edges_from(edgelist)
weights = nx.get_edge_attributes(G, 'weight').values()

ax = plt.subplot()
pos = nx.spring_layout(G)
nx.draw_networkx_nodes(G, pos, nodelist, alpha=0.5)
nx.draw_networkx_edges(G, pos, edgelist, width=list(weights), arrows=True, edge_color="red", alpha=0.5)
nx.draw_networkx_labels(G, pos, alpha = 1)
plt.ion()
plt.show()
2 Likes