SCDoc: Events/patterns, and searchability of non-class docs

Development
1

Referring back to Is there a way to send the freq value from a pbind to Processing? - #12 by jordan – this post raises several issues about SCDoc vs the requirements of documenting streams-patterns-events.

One major issue is that SCDoc indexing is heavily biased towards class/method documentation, while the default event prototype is just a data structure and not indexable this way.

I’ve argued for a long time that a significant gap in SC documentation is what I’d call “topic” documentation. IMO the linked post is calling attention to this, but reaching (again IMO) the wrong conclusion. (“A Practical Guide to Patterns” was an attempt at topic documentation… but… “if it doesn’t come up as a result of the search in the help browser when looking for specific terms (Event, Event Types, Pbind, callback…), it might as well not exist.”)

It’s a good thing that the default event prototype is a “soft” structure – you can extend or modify at runtime, without recompiling the class library. The suggestion in the linked post is that we should remove this flexibility, and force all event types to be hard classes, because of SCDoc’s poor indexing of non-class documentation. Well, I’m going to have to disagree with that. If the problem is that it’s difficult to search for documentation that is not about classes or methods, then the solution is to improve the search features!

We should definitely not be making programming-interface design decisions based on the documentation system’s strengths and weaknesses. The documentation system should serve the language, not the language serving the documentation system.

So one purpose of this thread is to brainstorm ideas to improve SCDoc indexing, so that topic documentation can be as easily accessed as class documentation.

One simple approach that wasn’t even considered is – thorough linking. We already have a document that exhaustively lists the keys in the default event prototype (but “it might as well not exist” because it isn’t searchable).

What if every help file related to the default event prototype (Event, Pbind etc.) contained a prominent link to the existing document?

I’ll have to wrap up shortly, but I’d like to finish with one observation from my experiences a few years ago preparing some scores in LilyPond. LP’s documentation is exhaustive, thorough and generally excellent – and it takes a long time to learn where things are in it. From this, I came to suspect that there may simply be no way around the problem of building a mental map of the documentation. Improving search features can help to alleviate this problem, but they cannot eliminate this problem. Agree with Jordan that there are things we can do to make it easier.

hjh

2

I agree with @jordan that this info needs to be findable. I’ve seen the /callback key used but would also have no idea where to look for help of proper use.

Here’s a thought: - what if the event types were stored in an EventTypeLib object instead of just hanging out in Event’s classvars? that seems like.a more usual pattern in SC and also could make them findable in the help system.

3

I also agree – where we disagree is how to make them findable.

The argument to class-ify event types and keys is: because SCDoc’s class/method search is good, and search for other types of keywords is relatively poor, then we should push things into classes and methods for no reason other than documentation.

My point is that if we don’t have a good way to search for non-class, non-method entities, then this weakness in the documentation system should be addressed in the documentation system. With apologies but I believe what I’m saying is simple common sense.

The benefit here would be to unify event types and event type parents (where the current storage design is a bit of a hack). In that sense, yeah, I’m interested.

But how would this address the search problem? The keys are still just data. There’s no compelling reason to make methods for these data keys (except for documentation, but I’ve argued and will continue to argue that documentation should follow the software’s design and that we would be setting a very bad precedent if we start making software design decisions because “this way is better for the doc system”). And, method-izing the keys runs the risk of placing arbitrary limits on usage of what is supposed to be an agnostic name space. Think very very carefully about this before proposing things that will make events worse and which may not solve the doc problem.

How would you search for “callback” if you didn’t know in advance that it’s called “callback”? (Similar problem: Python calls it “map,” while SC calls it “collect.” Hypothetical forum question: “I’m coming to SC from Python and I searched for ‘map’ and didn’t find the right thing.”)

You would discover “callback” by the comprehensive listing being linked in from somewhere, and by reading through it.

While I understand what’s being said about search results being the #1 way to find things, I’m not certain that search will fix everything here.

hjh

4

WRT the idea of making methods for event keys –

The key concept is that the data in an event have no meaning until the event is played. (There is one exception: the event stream player needs to know the event’s delta, in order to reschedule. This is independent of Event:play.)

So it would be possible to apply an event type at play time, and at this time, extract values from storage and put them into the instance variables.

Here, I haven’t worked everything out – the all storage is muddled – but it shows a shape of what it might look like.

EventType {
	classvar <all;

	var <>name, <>parent;
	var <>event;

	*initClass {
		all = IdentityDictionary.new;
	}

	*add { |name, parent|
		var new = super.newCopyArgs(name, parent ?? { Event.default.parent });
		all.put(name, new);
		^new
	}

	*new { |name|
		^all[name].copy
	}

	// playable instance with data from an event
	*fromEvent { |name, event|
		var new = this.new(name).dump;
		if(event.parent.isNil) { event.parent = new.parent };
		^new.performWithEnvir(\fromEvent, event)
	}

	fromEvent {
		^this.subclassResponsibility(thisMethod)
	}
}

EventTypeNote : EventType {
	var <>delta, <>dur, <>stretch, <>sustain, <>legato,
	<>degree, <>note, <>midinote, <>freq, <>detunedFreq,
	<>ctranspose,
	<>root, <>octave, <>stepsPerOctave, <>gtranspose,
	<>scale, <>mtranspose,
	<>db, <>amp;

	*initClass {
		Class.initClassTree(EventType);
		EventTypeNote.add(\note);
	}

	fromEvent { |delta, dur, stretch, sustain, legato,
		degree, note, midinote, freq, detunedFreq,
		ctranspose,
		root, octave, stepsPerOctave, gtranspose,
		scale, mtranspose,
		db, amp|

		this.delta_(delta)
		.dur_(dur)
		.stretch_(stretch)
		.sustain_(sustain)
		.legato_(legato)
		.degree_(degree)
		.note_(note)
		.midinote_(midinote)
		.freq_(freq)
		.detunedFreq_(detunedFreq)
		.ctranspose_(ctranspose)
		.root_(root)
		.octave_(octave)
		.stepsPerOctave_(stepsPerOctave)
		.gtranspose_(gtranspose)
		.scale_(scale)
		.mtranspose_(mtranspose)
		.db_(db)
		.amp_(amp)
	}

	play {
		// not going to replicate all the logic today
	}
}

Then:

e = (degree: 0, dur: 2);

f = EventType.fromEvent(\note, e);

f.dump;
Instance of EventTypeNote {    (0x5610b1d5aed8, gc=D0, fmt=00, flg=00, set=05)
  instance variables [22]
    name : Symbol 'note'
    parent : instance of Event (0x5610b1a9f998, size=5, set=3)
    event : nil
    delta : nil
    dur : Integer 2
    stretch : Float 1.000000   00000000 3FF00000
    sustain : instance of Function (0x5610ae928cf8, size=2, set=2)
    legato : Float 0.800000   9999999A 3FE99999
    degree : Integer 0
    note : instance of Function (0x5610ae928828, size=2, set=2)
    midinote : instance of Function (0x5610ae928898, size=2, set=2)
    freq : instance of Function (0x5610ae928978, size=2, set=2)
    detunedFreq : instance of Function (0x5610ae928908, size=2, set=2)
    ctranspose : Float 0.000000   00000000 00000000
    root : Float 0.000000   00000000 00000000
    octave : Float 5.000000   00000000 40140000
    stepsPerOctave : Float 12.000000   00000000 40280000
    gtranspose : Float 0.000000   00000000 00000000
    scale : instance of Array (0x5610af0f6c80, size=7, set=3)
    mtranspose : Integer 0
    db : Float -20.000000   00000000 C0340000
    amp : instance of Function (0x5610ae928f98, size=2, set=2)
}

Well, OK… it’s all there.

But I have to be fully honest that I’m not sure to what extent this is a proof of concept vs a reductio ad absurdum. I’m not convinced that it’s an improvement to have 19 instance variables (at minimum) for the most commonly-used event type. performWithEnvir is not implemented as a primitive; this would introduce overhead (current way: get an Event from Pbind, call play; hypothetical way: get an Event from Pbind, loop over 19 variable names to copy stuff into instance variables, then play – watch one-grain-per-event granular synthesis suffer). That long list of .delta_(delta).dur_(dur) strikes me as a red flag. Replicate this Event.eventTypes.size = 53 times (some of these may be from my quarks though). All in all, it seems like a long jog around the block, for… documentation?

So I think it underscores the point that the way to fix the document-search problem is to improve document search, not to hack up something ugly only to exploit method searching.

But… maybe someone will clean it up and convince me, who knows?

hjh

5

That isn’t true.

I thought I’d give a short implementation of what I suggested.

PrEventBase {
	var rollingEvent;
	*new { |...kvs|
		^super.newCopyArgs(kvs.asEvent);
	}
	asEvent { ^rollingEvent }
	add {|k, v|
		rollingEvent[k.asSymbol] = v;
		^this
	}
	dur {|v| ^this.add(\dur, v) }
}

NoteEvent : PrEventBase {
	var rollingEvent;
	*new { |...kvs|
		^super.newCopyArgs(kvs.asEvent ++ (\type : \note));
	}
	asEvent { ^rollingEvent }
	instrument { |v| ^this.add(\instrument, v) }
}

…and how it should be used …

Pbind(NoteEvent(
	\instrument, \synthA,
	\dur, 1, 
	\freq, Pwhite()
))

It is just a wrapper, same syntax.

This way you can add the documentation for each key in the event and there are no restrictions on what else can be added, the methods are really only there to provide examples of what the default keys do. There should also be a ++ operator which gobbles the keys on the left, expect the \type key. There would have to be a little preamble explaining that, whilst the methods can be used, the intended purpose is that they are supplied in the constructor.

Also, no need for any other implementation, whilst the implementation of \callback (for example) should be mored into this class, it doesn’t have to be immediately. This makes it compatible with the existing version and allows for it to be implemented gradually.

If you consider the class system as a type system instead, I think this makes even more sense given the key that varies is \type.

Is this still ugly?

Sorry, could you elaborate on what the difference is? I don’t understand.

Except from all the scheme functions which are often left out or missing argument descriptions. It think this is a similar kind of problem here where there is a different type of code/data that the docs where not originally designed for and has been neglected as a result. The only way to write custom callbacks in lilypond is to search through the source code to understand what scheme functions exist and how to use them.

6

Sure.

Here’s an event.

e = (degree: 0);

Now: What does degree mean?

Most of us probably say that it means the root note of the scale.

But what if I’ve created an event type that sets drawing parameters for a UserView, and degree indicates rotation? This is a valid usage – it’s a critical part of the event system that it should be completely user-definable. Nothing in the default event prototype is mandatory.

This means that the data in an event quite literally have no meaning at all until they are interpreted by the event type. An event is just a collection of arbitrary data, until the moment that it’s played.

From this perspective, it’s conceptually muddled to bind the act of populating data to a specific interpretation of the data (Pbind(NoteEvent(...))). Pbind only prepares data. Pbind should not in any way, shape or form assign meaning to the data. (You could say that \type, \note already assigns meaning but I could replace the default event prototype with a different prototype, with a different event type system, which is quite different from hard-binding it to an object.)

But the core of my skepticism is this: The problem is in the help system. All of this seeks to avoid fixing the problem in the help system. It’s like, I hit my head but asked the doctor to put my arm in a cast. Fix the problem where the problem is. (If people are frightened of doing dev work on SCDoc, this is understandable but it does not in any way invalidate the idea that it’s better to fix the problem where the problem is.)

hjh

1 Like
7

I’d say it has no meaning, or is at least ambiguous, and is therefore, probably a mistake - or at least unclear without context. Implicitly converting data like this is usually a cause of headache later on. A lot of programming languages have moved from raw units to typed units, e.g., in c++ you’d write time as std::chrono::seconds{1} rather than just a dimensionless 1, and then you can covert implicitly between any type of temporal duration - the type of the unit is encoded in the type system. Then if you want to convert you do so explicitly.

It isn’t, NoteEvent assigns the meaning, same as (\type, \note).
You could easily change the type like this… MIDIEvent.fromEvent(note_event_instance) … here it would grab all keys except the type key. You could also get the raw array out easily … midi_event_instance.asEvent.

Sure, but if you are doing that then all standard ways of interpreting keys go out the window and the help documentation is meaningless - so you might as well use the existing way (the two systems can co-exist).

We disagree about this, I do not think the problem is the help system, but that a type of thing has not been encoded in the type system, and therefore, does not show in the docs.

What I’m suggesting is that when you use NoteEvent you are joining the preparation and the performance, now if you wish to uncouple the data from its interpretation you fall back to the ways things are now, and when you need to specify an interpretation, you do so like … NoteEvent.fromEvent(e).

Also, how should the help system turn itself into searchable documents? Originally, I suggested the headers (or sub headers…) could appear in the search results. But this might require rewriting several help docs to make them return useful results. Perhaps instead it could be some command where you specify tags? Either way, this seems like a lot of work to rewrite these documents. Alternatively, it could just do a raw search, but then this will create a lot of false positives.

8

I agree with @jamshark70. SC’s event system is dynamic on purpose. Creating classes just for improving the documentation seems backwards to me.

There is already an overview Event types, but it is a bit rudimentary. Why not extend it and add prominent links?

1 Like
9

… and the ideal solution to this is to improve the help system so that this type of thing is encoded.

hjh

10

If it is a type, shouldn’t it be a type (class is sc lingo)? Even if they are just factory functions for an event.

Is it possible to improve the help system without parsing the code? Aren’t these things just implemented as symbols or even ‘global variables’ inside a class? How would you know when to apply such a heuristic? That sounds like a big job to do automatically, and even if all the documentation for each event type and their corresponding keys were added manually to the documentation, perhaps in Event Types, it would still needed to be updated every time something changed.

11

The prob with using a class is that SC can’t add classes without recompiling (uggg)

It bugs me though that Events default to type \note when you call play on them and that the keys live in Event.parentEvents.default. Its super tough to discover this.

A more consistent interface might have been
Event( \note, [ freq:400, instrument:\default ])
like Synth. Since you can’t call play on an Event without a type (one is invisibly assigned if you don’t set it) - type should be a parameter. Plus then the IDE would prompt the user for the type parameter.

Then we could store the Event Types in an object EventTypesLib that could the parent/partials in a sensible way and would have its own help page. Inbuilt types could have convenience methods that could show up in the docs like the convenience methods for Env.

side note: I still have no idea where the \callback key is implemented and search doesn’t help

12

It’s in Event.makeParents or something… in the Event.sc file… It’s a 1000 line monster.

13

ok Event:makeParentEvents sets partialEvents to an Event which contains, in turn, an Event at the playerEvent key which has the ~finish and ~callback keys in its play function…

14 
Event.partialEvents.playerEvent[\play].postcs
1 Like
15

Keep in mind that Event is just a dynamic object with support for prototypical inheritance, like Object in JavaScript or a (meta)tables in Lua. Its use is not limited to sequencing in any way. Maybe you actually meant Pbind?

EDIT: Actually, we cannot even assume that every Pbind will be played. The \type key is only relevant in the context of an EventStreamPlayer, but (Event) Patterns can be used for many other things.

16

Thanks for the reminder @Spacechild1 - yes as you say Event is a dynamic object with support for prototypical inheritance… but a system of types that determines how it responds to .play is bolted on. In particular calling .play sets the parent invisibly when type key is nil. Its nice and fast at the start ().play and magic - a cheezy piano note! - but ultimately confusing and tricky to extend and understand. I’m not a sophisticated enough programmer to see the way forward here but I’m guessing there are improvements that could be made!

17

This is a good idea. Event.eventTypes was more or less fine when an event type was only a function. Then we added event-type parents, to supply default values with an event type. So now there’s Event.eventTypes and Event.parentTypes (which isn’t a “parent type” – it’s really a “parent for an event type”).

If the concept is that a play function is bound with a set of default values, then yes, I fully agree that this should be represented as an EventType class, with a global collection.

However… Part of the design of Event is that you are not forced to use the default event prototype – the built-in event type system should be optional. It is so commonly used that we often assume it’s the only way, but it has never been enforced.

So Event( \note, [ freq:400, instrument:\default ]) is wrong for two reasons: 1/ as spacechild1 said, you can’t assume every Event will be played and 2/ this would make it mandatory to use this particular event-type system. (Although, it would be OK to have a subclass of Event which implements event types in this way.)

There is a very big difference between making a mechanism so convenient that everybody just uses it vs making it impossible to implement a different mechanism.

I mean… perhaps this is not fancy enough, but if I open Event.sc and Edit > Find “callback”… there it is.

Yes, it takes some sleuthing sometimes to find things. But, for another instance, there is no table of contents for the C++ sources. I find UGen sources by doing, in Emacs, alt-x rgrep and searching within the server/plugins directory, or, I find sclang primitives by rgrep-ping lang/LangPrimSource.

-*- mode: grep; default-directory: "~/share/supercollider/lang/LangPrimSource/" -*-
Grep started at Thu Jan  5 08:00:33

find . -type d \( ... deleted... -name \*.c\+\+ \) -exec grep --color -nH --null -e _ArrayReverse \{\} +

./PyrArrayPrimitives.cpp\02543:    definePrimitive(base, index++, "_ArrayReverse", prArrayReverse, 1, 0);

Simple, dumb text search isn’t a one-key jump-to-the-place solution like method lookup in the IDE, but it’s always available and shouldn’t be neglected.

My opinion here is that there is no substitute for writing documentation.

There already exists documentation for keys in the default event prototype. So the issue that was raised initially is that you can’t search for a default-event-prototype key and get to this document – fair enough, that’s a legit issue. But now we’re not just talking about searchability – we’re talking about a much higher-complexity solution = autodoc. I mean, sorry but, where does this end?

But that doesn’t matter because a mechanism already exists. I didn’t know about this until I went looking for it just now (I found it, incidentally, by “Find in page” in the SCDoc syntax reference – text search FTW again!) but… SCDoc already has a “KEYWORD::” tag, which creates a named anchor in the page, and the keyword itself is searchable. I tried it.

subsection::Timing control

KEYWORD:: sustain
code::\sustain:: -- Number of beats to wait before releasing a Synth node on the server. The SynthDef must have a code::gate:: argument for the explicit release to be sent; otherwise, the pattern assumes the note will release itself using a timed envelope. code::\sustain:: is calculated from code::~dur * ~legato * ~stretch:: if not given directly.

Help → search → “sustain” →

Tutorials
Pattern Guide 08: Event Types and Parameters - Describes the event types defined in the default event, and the parameters they expect

EDIT: Found later, it isn’t even necessary to change the document structure – counterintuitively, but usefully, keyword:: is allowed like this :exploding_head: – fully parses and renders!

definitionlist::
##
keyword:: xyz
xyz || Definition of xyz
::

It’s a large manual edit, but you would do it once. (Yes, it’s true that “it would still needed to be updated every time something changed” but 1/ the default event prototype is pretty stable and 2/ this is true of 99.99% of SC documentation – so let’s not throw out the baby with the bathwater.)

So there is something that could be done in a matter of hours to make the keys help-searchable, without any code changes in SCDoc, and without any API changes in the event system. IMO this is the most practical way forward.

At the same time – @Jordan, if you want to pursue the approach of classes for event types, you could do this in a quark without waiting for anyone’s permission. A working prototype of that would be more persuasive than anything else.

hjh

2 Likes
18

Related to this: here’s a use case to consider.

MIDIClient.init;

m = MIDIOut(0);  // Linux: also .connect something

(
p = Pbind(
	\instrument, \default,
	\degree, Pn(Pseries(0, 1, 8), inf),
	\dur, 0.125
);

q = Pbindf(p, \type, \midi, \midiout, m, \amp, 0.5);

r = q.play;
)

That is, “I have a ‘note’ pattern – now I want to play it as ‘midi’.” Super-easy with \type, .... I’d expect complication with an EventTypeNote() object. Just mentioning it so that a prototype could be feature-complete.

hjh

19

Winter holiday semester break, can mess around with stuff:

That “large edit” actually took < 5 minutes, with regexp search-replace in emacs.

The one problem is that where keywords appear multiple times, only the first can be reached by linking.

But now the document does exist in help search results.

hjh

5 Likes
20

That’s great!

I just tried to add keywords to my own documentation. It generally works, but I noticed an issue: The search code does not recognize the special use of backslashes for Symbol notation. For example, it would interpret \vst_set as \x0bst_set… You can get around this issue by selecting the part after the backslash before hitting Cmd+D, but this is pretty awkward…