POC of simplified responders

SuperCollider 4: First Thoughts - #151 by Rainer cites “making GUI’s is difficult” as a pain point.

That got me thinking… what would make it easier to make GUIs? (Or, really, anything involving a callback function.)

The most common question I recall hearing about this is “how do I get a value out of my GUI or OSCFunc or MIDIFunc”? Then we have to explain that the GUI or other responder is just a conduit and isn’t really storage, or proxy, for a value. This is a concrete area where usage is not intuitive. Users think (especially for MIDI and GUI) that the object represents a control, and the control has a current state.

What if responders did maintain state? At any time, you could ask for its current value, by the same interface across GUI, MIDI, OSC etc.? (MIDI and OSC don’t know their value. GUIs do, but only in AppClock, making GUIs no easier than MIDI to use in patterns, for instance.)

As a proof of concept, I tweaked View so that it connects a second method to the Qt-side action() signal. This method updates the variable belonging to the View (currentValue – this is an awkward name, but I don’t want to go through all the view subclasses and changed the value method just now). This could be implemented for MIDIFunc/defs and even OSCFunc/defs – OSC is trickier because of messages’ free format, but there are likely some standard formats e.g. [cmdPath, value] where it would be easy to extract a value.

So then every type of external input responder could be polled for a value.

Going a step further – the updateCurrentValue method can also broadcast to dependents. So you can attach some object to forward the value to any place you like. Some standard types of connections could be automated using mapToSomething methods. If this were implemented consistently across Views, OSC, MIDI, HID, then the common user request, say, "I want to control a synth parameter by GUI would require only:

  • make the synth
  • make the GUI
  • mapToNodeControl

… and a lot of the glue that users are currently responsible for managing by hand gets hidden.

For example: Apply these commits –

Then:

(
s.waitForBoot {
	a = { |freq = 440, ffreq = 2000, rq = 0.1|
		var sig = Saw.ar(freq * [1, 1.008]);
		BLowPass4.ar(sig, ffreq, rq) * 0.1
	}.play;

	v = Slider2D(nil, Rect(800, 200, 400, 400)).front;

	// GUI --> freq (50 - 600)
	// because this is a 2D slider, extract 'x'
	v.mapToNodeControl(a, \freq, outSpec: [50, 600, \exp])
	.translate_(_.x);

	// GUI --> ffreq (225 - 8000)
	// because this is a 2D slider, extract 'y'
	v.mapToNodeControl(a, \ffreq, outSpec: [225, 8000, \exp])
	.translate_(_.y);

	// ideally the user wouldn't have to 'didFree' the model
	// but I don't want to do that much surgery in this POC
	v.onClose = { v.changed(\didFree); a.release };
};
)

There’s probably something that I’m missing, but I tried to design the POC so that it could apply in general to just about anything where we currently use a callback function. I could anticipate issues about translating various data formats, though that could be handled by plug-n-play translator objects.

It does directly express the idea of “connect this control source to that thing.”

For comparison, the current canonical way. It seems shorter, but I’m not sure it’s clearer – e.g. a new user might ask, why do you need vars up front for the specs?

(
s.waitForBoot {
	var freqSpec = [50, 600, \exp].asSpec;
	var ffreqSpec = [225, 8000, \exp].asSpec;

	a = { |freq = 440, ffreq = 2000, rq = 0.1|
		var sig = Saw.ar(freq * [1, 1.008]);
		BLowPass4.ar(sig, ffreq, rq) * 0.1
	}.play;

	v = Slider2D(nil, Rect(800, 200, 400, 400)).front;

	v.action = { |view|
		a.set(
			\freq, freqSpec.map(view.x),
			\ffreq, ffreqSpec.map(view.y)
		)
	};

	v.onClose = { a.release };
};
)

Anyway… thinking out loud for now. (I think Miguel(?) had done some work on functional reactive programming in SC, didn’t quite get traction either.)

hjh

1 Like

This is all very impressive.

So, when a slider is mapped to a node control, will changes to the node control that occur elsewhere also reflect in the Slider’s view?

So that change slider → affects node
as well as change node → affects slider

Not implemented in this POC.

That would be straightforward for, say, mapToControlBus but for nodes, consider a counterexample:

g = Group.new;

a = Synth(\default, [freq: 200]);
b = Synth(\default, [freq: 400]);

Now, if mapping something to g’s freq control, which value should sync back to the GUI, 200 or 400? … You can set controls on Groups, but there’s no control to get back. (The example is a bit contrived, because, if you’re mapping a data stream to a control name on a group, all synths inside will take the same value for that control. But it illustrates that you have no guarantee of consistent values within a group.)

Syncing a GUI is simple. Syncing OSC isn’t, because you can’t be sure of the return message format (or even if there is a return message). MIDI probably can’t.

hjh

I think the only thing that makes GUIs simpler is to introduce a functional-reactive / dataflow layer. You will want to get rid of writing any sort of callbacks altogether. I also wouldn’t paste GUI code into a SynthDef, it’s always useful to think of GUI as wrapping around, so you can “turn it off” when you don’t need it.

My UI language in Mellite does this to a certain degree, although it’s not yet particularly polished. For instance, if you have a sine oscillator

val sig = SinOsc.ar("freq".kr(441)) * "amp".kr(0.2)
Out.ar(0, Pan2.ar(sig))

you can access that from a widget and control function like so

val r  = Runner("proc")
val sl = Slider(min = 0, max = 127)
val lb = LoadBang()
lb --> r.runWith("freq" -> sl.value().midiCps)
FlowPanel(Label("Freq:"), sl)

So here sl.value() gives you a dataflow variable (reactive expression) that can be used as a synth argument. Runner is kind of similar to an Ndef in this case.

Then, if you wish to, you can make the UI optional by declaring an explicit dataflow variable:

val r      = Runner("proc")
val sl     = Slider(min = 0, max = 127)
val pitch  = Var(72)
// should really be `sl.value <-> pitch`,
// but I haven't implemented that yet, so two
// links are needed
sl.value <-- pitch
sl.value --> pitch
val lb     = LoadBang()
val freq   = pitch.midiCps
val rnd    = Random().range(0, 127)
val bRnd   = Button("Random")
bRnd.clicked --> Act(
  rnd.update,
  pitch.set(rnd),
)
val freqS  = "%1.1f Hz".format(freq)
lb --> r.runWith("freq" -> freq)
FlowPanel(Label("Freq:"), sl, Label(freqS), bRnd)

Now the slider and the randomise-button can update that variable, but you could also cut away that bit, and/or encapsulate the part that doesn’t include widgets.

Screenshot from 2021-08-13 13-03-50

(yeah, there is an issue with the baseline alignment of the slider)

Exactly what I was trying to do: view.mapToSomething(parameters) is essentially a dataflow relationship (again, unless I’m missing something). Agreed that your .value is more flexible – in my own work, I have a GenericGlobalControl that syncs a value between the client and a server-side control bus. Then I can gc.asPattern and it can do any processing available to patterns, or gc.kr and use any UGen-style ops, transparently (and they can be watched to get server-side value changes too). But since it depends on a control bus, it may not be quite general enough for the broader topic.

hjh

FWIW here’s the same example with the Connection quark:

(
s.waitForBoot {
	~c = ~c ?? { ControlValueEnvir() };
	~c.use {
		~freq.spec = [50, 600, \exp].asSpec;
		~ffreq.spec = [225, 8000, \exp].asSpec;
	};
	
	~synth = { 
		|freq = 440, ffreq = 2000, rq = 0.1|
		var sig = Saw.ar(freq * [1, 1.008]);
		BLowPass4.ar(sig, ffreq, rq) * 0.1
	}.play;
	
	~slider = Slider2D(nil, Rect(800, 200, 400, 400)).front;

	// Convert .action into signals for \x and \y, because .value doesnt work for Slider2D
	ViewActionUpdater.enable(~slider, \action, \x, \x);
	ViewActionUpdater.enable(~slider, \action, \y, \y);
	
	[
		// Connect to node
		~c.connectToSynthArgs(~synth),
		
		// Connect TO slider
		~c[\freq].signal(\input).connectTo(~slider.valueSlot(\x)),
		~c[\ffreq].signal(\input).connectTo(~slider.valueSlot(\y)),
		
		// Connect FROM slider
		~slider.signal(\x).connectTo(~c[\freq].inputSlot),
		~slider.signal(\y).connectTo(~c[\ffreq].inputSlot),
		
		// Logging
		~c[\freq].connectToUnique({ |...args| args.postln }),
		~c[\ffreq].connectToUnique({ |...args| args.postln }),
		
	].freeAfter(~slider)
}
)

There’s a conceptual challenge with the code @jamshark70 is proposing, which is that there is no single, canonical location for the value of our parameters. These values are somehow shared between the internal value of the Slider2D (and is thus lost if the UI is closed), and the running Synth (lost of the Synth is free’d, and additionally not accessible without a complex async call to the server).

It’s a long term goal to pivot Connection.quark to something closer to a redux design pattern, though there are some reasons why this might not be a very good fit for simple UI building use-cases in SuperCollider.

And, an example of adding a “randomize” and “reset” functions is just:

	View().front.layout_(VLayout(
		~slider = Slider2D(nil, Rect(800, 200, 400, 400)).minSize_(300@300),
		ToolBar(
			MenuAction(
				"randomize",
				{
					~c.do {
						|cv|
						cv.input = 1.0.rand;
					}
				}
			),
			MenuAction(
				"reset",
				{
					~c.do {
						|cv|
						cv.value = cv.spec.default;
					}
				}
			),

		)
	));

That’s an excellent point.

The GenericGlobalControl that drives my performance interfaces serves as the Model in MVC, so I don’t have that problem in my own work.

The conflict between user expectation and good design practice is tricky. If incoming user is thinking “I want a slider controlling a synth parameter,” it’s kind of a tough sell to explain that, no, this isn’t really what you want – what you want is an object representing the parameter value, which is connected separately to the GUI and the synth. If you’ve already been down that road and wound up with some variant of MVC, then that will make sense right away. But this may be confusing at first (especially, perhaps, for users coming from Pd/Max where it’s literally [slider] → [ugen inlet]).

That’s why I was considering – what if the GUI/MIDI/OSC responder object itself holds the value? Users could eventually get around to MVC, when they run into problems.

Agreed, though, that synth.set() vs gui.value_() would be messy to handle. This is not at all thought through in this POC.

Not necessarily – if you load my first example above, then:

v.currentValue;
-> Point( 0.55555555555556, 0.65608465608466 )

v.remove;  // close the view

v.currentValue;
-> Point( 0.55555555555556, 0.65608465608466 )

… because currentValue is deliberately not stored in the Qt side, so the value is available as long as the GUI object is reachable.

I guess that’s one of the tricky things about discussing design changes – we still carry old assumptions, when the point is to challenge or replace those assumptions. Another example is that parameter values are not available in Synth. Well, Synth is a very thin abstraction for a server-side structure. Why are we still restricting ourselves to that? What if we had a “has-a” adapter for Synth, which does maintain every parameter value in a dictionary? No more asynchronous query needed, just look it up. What if the help recommended this higher-level class for routine use, instead of Synth? Then there is an opportunity to take a lot of these “oh but you can’t do that with Synth” problems and actually solve them… the assumption that “Synth is the way” ends up hurting us.

hjh

Random thought: “I want a slider controlling a synth parameter” sounds like a chapter in a tutorial about “making supercollider user interfaces”.

Second random thought: users may also wonder about how to do two-way stuff, e.g. a slider that can both send control changes to a midi synth, as well as observing/visualizing incoming control changes from that midi synth (extra bonus: midi learning to dynamically attach a two-way control to a given synth parameter). Maybe a nice idea for a quark containing “smart” controls?

(Edited to add: adding these thoughts is not intended to try to get someone to do that work, although of course, if someone felt inspired, go ahead :slight_smile: )

I know this is about implementation, but in terms of notation SuperCollider already has a rather nice way of writing simple user interfaces, namely:

{arg freq=220, amp=0.1, pan=0; Pan2.ar(SinOsc.ar(freq, 0), pan, amp)}

Given an editor command that runs this as Ui.playFunctionWithNdefUi(...) (as below) you get sliders, a randomise button, presets and a preset morph slider.

Sometimes the nicest interfaces are the ones you don’t need to write at all. (Thanks JitLib people!)

Best,
Rohan

Ui {
    *playFunctionWithNdefUi {
        arg aBlock;
        var aSymbol = ("_gensym_" ++ Date.getDate.stamp).asSymbol;
        Ndef.new(aSymbol, aBlock).mold(2).play;
        Ndef.new(aSymbol).gui;
        ProxyPresetGui.new(NdefPreset.new(aSymbol)); /* JITLibExtensions */
    }
}

Ps. In Kyma the notation is equally concise (ie. !freq &etc.) but the interfaces are perhaps a bit fancier, and can also be “unlocked” and edited graphically. Editing interfaces once they exist can sometimes be nicer than trying to work out how they should go in advance!

Pps. Having naming conventions so that (for instance) freqMin,freqMax is drawn as a range slider, and freqX,freqY as an area slider and muteSwitch as a toggle and restartTrig as a button &etc. can make the interfaces nicer also.

Sure, JITLib GUIs are helpful. My thought is not only about GUIs though.

GUIs, MIDIFuncs and OSCFuncs have all proven tricky for new users because callback functions aren’t an especially elegant representation of the relationship between a source of external input and a parameter being controlled.

Because they all use callbacks, they could all be connected to targets by the same mechanism.

Hm… An idea just now… currentValue could hold a proxy for a value, which would serve as the MVC model. The currentValue getter method would unwrap and return the plain value. Then two-way GUI / synth sync would be done in the normal MVC way (and probably a lot of the glue could be hidden), and MVC makes it extensible to, say, value <–> GUI and OSC control on a tablet.

hjh

Yes, but also the notation {arg freq=220; SinOsc.ar(freq,0) * 0.1} doesn’t say what the control source is.

The notation arg just says that freq is a control source with an initial value of 220.

It’s very declarative.

The JitLib system can read the notation and make a nice interface.

I think it counts as one kind of answer to the question “what would make it easier to make GUIs?”, ie. it still counts even if it already exists!

And I think Kyma is interesting because it shows how nice an implementation of this kind of notation can be.

Best,
Rohan

Ps. I also learned my favourite notation for event control from Kyma!

It’s also rather declarative.

{Splay.ar(Event.voicer(16, {arg e; SinOsc.ar(e.p.midicps, 0) * e.w * e.z}))}

p is the events notion of pitch, z it’s notion of pressure, w it’s notion of gate, &etc.

(I know one letter names are not to everyone’s taste!)

Delegating event processing to the system makes the notation independent of the source of the events.

(Or the other way around, making the notation independent of the source of the events requires delegating all the event processing to the system!)

Of course!

But the next question from an incoming user is, how do I connect a MIDI CC to one of the parameters? And then we’re back to the beginning.

The Connection quark is pretty cool for this.

There is one weakness to JITLib GUIs in the main class library. The only way to specify a range for a parameter is globally, by e.g. Spec.add(\ffreq, [150, 8000, \exp]) and this would apply to every \ffreq everywhere, even if a different NodeProxy needs a different filter range.

The JITLibExtensions quark solves this problem, though it’s a little bit inconvenient in that you need to do myNodeProxy.addSpec(\name, spec, \name, spec...) and myNodeProxy = { ... some source ... }. My JITModular interface uses this, and from experience, I find that it gets annoying after a while.

That’s interesting – still thinking about it. I’m not sure I grasp all the implications yet (which suggests that it’s a cool idea.)

hjh

I’m not sure “how do I connect a MIDI CC to one of the parameters?” does have to lead back to the beginning.

One thing is that the interface would be a very nice place to set this!

(I don’t think Ndef.gui will do it, though I’m no expert, but it could.)

But also if you want to write it directly in the text that needn’t be awkward.

arg freq = 220.cc(5) could get input from continuous controller five (or six, if you like!)

arg freq = 220.range(150, 8000) could set the range.

You could do both.

The arg notation just marks the names of the objects in the block that are to be treated as controls.

What to do with them is left up to the consumer of the block (which is left up to the editor command).

Best,
Rohan

Ps. Writing the .cc & etc. at the declaration is nicest (since you can refer to the object many times) but of course you’d not want to actually give such a block any arguments!

{arg x = 9.sqrt; x}.value // => 3.0
{arg x = 9.sqrt; x}.value(9) // => 9

Thinking about this more carefully, the notation above could require an awkward amount of meta-programming…

(I wrote it that way because I was too embarrassed to admit what I actually do, which is to put this information in a comment before the graph and parse that!)

I wonder if there is a nice way of notating this properly without jumping through too many hoops?

This already works:

(
~def = {
	|db, freq|
	
	freq.spec = [20, 4000].asSpec;
	db.spec = \db.asSpec;
	
	db.dbamp * SinOsc.ar(freq)
}.asSynthDef();

~synth = ~def.play;

~def.specs.postln;
)

For me, adding MIDI mapping details to a SynthDef is not ideal, as it locks that SynthDef to a single MIDI configuration, making it very hard to adapt to e.g. different controllers, or other situations where you may not control things via MIDI.

I think this is a case where you absolutely want to provide opinionated tooling / libraries / interfaces, that encourage users to do the right thing (e.g. have a proper centralized “model” for values). IMO not doing this can result in all kinds of subtle problems and confusion that are more difficult for an inexperienced user to debug.

One concrete problem is maintaining parameter values across multiple servers, and between server restarts (which is effectively the same thing as multiple servers). The simplest possible use-case: I plug in a controller and set some values with knobs. When I restart my server and my synth graph, I expect that all the same values will be set - this is very basic expectation that is quite hard to guarantee without separating value storage from a particular server instance. One of the major goals of the Connection quark was to build UI’s and controller interfaces that were persistent across Cmd+Period, server restarts, etc.

Yes, but the notation mentioned won’t work, c.f.

{arg freq = 220 * 2; SinOsc.ar(freq, 0) * 0.1}.play

(Some variation on which is probably how many of us learned what .prototypeFrame does, even if some of us (i.e. me) seem to keep forgetting…)

It might be nice if blocks could answer .initialisedArgumentValues, but that seems like a lot of work.

(Or perhaps the arg notation only seems like a nice idea if it’s easy to implement.)

And yes, I didn’t mean to extend SynthDef, only that an interface builder or such could look into the block arguments for things it wanted to know, but really it can’t.

There are so many different ways to do things!

Actually, this is completely convincing to me: Don’t encourage ill-advised shortcuts in the first place.

So – bad idea to put values in the responders.

Good idea to provide a simple way to represent values as models, and connect them to audible and visible outputs.

The Connection quark clearly has the plumbing that is required. I worry that the example is verbose enough that it would frighten incoming users.

I wonder if there’s a way, for instance, to do a bidirectional connection in one step? That is, the user could be reasonably expected to think “1. I have a value; 2. Connect to synth; 3. Connect to GUI” but they have to write “1. Value; 2. Connect to synth; 3. Connect freq to GUI x; 4. Connect ffreq to GUI y; 5. Connect GUI x to freq; 6. Connect GUI y to ffreq” which is discouraging. GUI connections could be assumed bidirectional, unless specified otherwise.

c = RepositoryOfModels.make {
	... specify controls...
};

x = Synth(...);
v = Slider2D.new(...);

c --> x;  // can we actually get it down to this?

c.mapTo(
	\freq -> v ... something for slot X,
	\ffreq -> v ... something for slot Y
);

MIDIdef.cc(\x, nil, 11, 0).mapTo(c, \freq);

// or?
MIDIDevice(...).mapCCsTo(c,
	1 -> \freq, 2 -> \ffreq
);

… which in that very loose brainstorming format might not handle specs…? Just trying to think of ways to get the syntax down to something that wouldn’t scare people away.

For OSC, it would be trickier to define a bidirectional messaging protocol (which is one reason why I started with a Slider2D – because it exposes a difficult case).

hjh

Yeah, I explicitly designed Connection with very few “auto-magical” APIs or convenience functions - I wanted it to stick to clear, 1:1 APIs where ever possible. It’s verbose because every connection or semantic relationship takes a line of code or method call. My hope was that it would provide a base layer for more “automatic” UI generation tools - definitely for my own local use, I have a lot of UI-building factory functions that make this much more straightforward.

WRT bidirectional connections - yes, it’s definitely possible to make a simple bidirectional connection util function. But, when you consider the various kinds of UI widgets and the things they can connect to, it turns in to quite a number of combinations… I still havent thought of an elegant way to handle this in the core library.