Get notified when Routines are done

[amt]

Is there a possibility to get a signal or get notified when a Routine is done iterating? The scenario is having 2 routines nested like this:

Routine {
	arg arrs = Array.rand(10, 48, 72).dup;
	arrs.do {
		Routine {
			arg arr;
			arr.do {
				arg midi;
				Synth(\help_sinegrain, [\freq, midi.midicps]);
				rrand(0.1, 1).wait;
			};
		};
		// How can I know the time I should wait to use the next array?
		?.wait
	}
}

where I want to wait in the outside iteration until the inner iteration is done (the wait times for inner iteration are randomly chosen on the fly). I know I could just create two arrays beforehand for the inner wait-times, but is there a mechanism for doing this without by getting notified from the inner Routine?

[jamshark70]

@amt

Is there a possibility to get a signal or get notified when a Routine is done iterating?

With Routine itself, you’d have to write the notification into the Routine body, after the loop.

If that isn’t ok, you could do it with Task. Task does send notifications automatically. Routine doesn’t.

hjh

[jordan]

Here is how you can do it with CondVar. You could pull latch out into a class quite nicely if needed.

r = Routine{
	var arrs = Array.rand(10, 48, 72).dup;
	
	var latch = CondVar();
	var thread_counter = arrs.size;
	var try_open_latch = {
		thread_counter = thread_counter - 1;
		if (thread_counter <= 0, {latch.signalAll});
	};
	
	arrs.do{|pitches|
		Routine.run({ 
			pitches.do{|m| 
				m.postln; 
				rrand(0.1, 1).wait;
			};
			try_open_latch.();
		})
	};
	
	\waiting.postln;
	
	latch.wait;
	
	\done.postln;
};

r.play

[amt]

Do you know a link to how to register dependents for a task?

[jamshark70]

Just noticed something – if you intend to play the outer routine, then arg arrs is incorrect.

(
r = Routine {
	arg arrs = Array.rand(10, 48, 72).dup;
	"I expect this to be an array of arrays:".postln;
	arrs.postln;
}.play;
)

I expect this to be an array of arrays:
745.04022751

A routine/task/EventStreamPlayer that’s being played uses the thread’s input value for the clock time. You aren’t allowed to use it for anything else.

If you want to pass arrays into the routine, then you can only use next on it, not play. But then your ?.wait would be invalid, so I suspect this isn’t what you mean. (And then, you would have to be aware of how to pass in the second, third etc. values by nextInputValue = returnValue.yield – this is discussed in the pattern internals reference – if you’re not clear on this, then… now’s not the time for this way.)

If you want to parameterize the arrs and play the Routine, then you would have to wrap it in a function:

f = {
	arg arrs = Array.rand(10, 48, 72).dup;
	Routine {
		...
	}
};

Then… the further peculiarity is: If the requirement is to stream out one array’s worth of notes, and in sequence, stream out the next, and so on, then the simplest way is: Don’t use an inner Routine at all.

Routine {
	arg arrs = Array.rand(10, 48, 72).dup;
	arrs.do {
		arg arr;
		arr.do {
			arg midi;
			Synth(\help_sinegrain, [\freq, midi.midicps]);
			rrand(0.1, 1).wait;
		};
		// if you just use a nested loop,
		// then the 'wait' problem here
		// completely disappears!
	}
}

That is, if possible, it’s better to avoid complication – fewer moving parts = fewer opportunities for bugs.

Jordan’s approach is valid but assumes that all of the arrays will be running at the same time, which looks like a different requirement. (That one could be simplified a bit by using a predicate in CondVar.) BTW I can guess why Jordan went that way – because, in the original problem statement, there’s not really a good reason to introduce an inner Routine unless you want concurrency.

If you want a sequence of phrases, and you’re absolutely sure you must have an independent inner player for each phrase (I doubt that it’s necessary, but let’s roll with it), probably the clearest way is with a thread-sync object like CondVar:

(
r = Routine {
	var arrs = Array.rand(10, 48, 72).dup;
	var cond = CondVar.new;
	arrs.do {
		arg arr;
		Routine {
			arr.do {
				arg midi;
				Synth(\help_sinegrain, [\freq, midi.debug("note").midicps]);
				rrand(0.1, 1).wait;
			};
			cond.signalAll;
		}.play;
		cond.wait;
	};
}.play;
)

That is, where “is there a possibility to get a signal or get notified” suggests passive receipt of an automatically generated signal, it’s clearer IMO to produce the end-signal explicitly in the code (cond.signalAll).

I did also work up a Task version using the automatic notifications, but this ends up being worse, because you still need the CondVar logic, just invoked indirectly rather than directly = more code = more confusing relationships = harder to maintain. So I won’t even post that one.

hjh

[jordan]

@jamshark70

but assumes that all of the arrays will be running at the same time

That was the only reason I could think of for needing nested routines.
Also, I assumed the cond should wait until all of the routines had finished. I don’t think your final one does that, but waits until only the fastest finishes?

@jamshark70

That one could be simplified a bit by using a predicate in CondVar.

Could you elaborate? Want I wanted to write was something equivalent to C++'s latch or barrier, which would look like…

var latch = LatchCond(size: 4);
latch.size.do{ Routine{ 10.rand.wait; latch.arrive() }.play };
latch.wait; // waits for all the routines to arrive at the latch before continuing

[jamshark70]

@jordan

I don’t think your final one does that, but waits until only the fastest finishes?

No, it’s not about the fastest one because none of my examples should have any concurrency at all. cond.wait means the outer routine won’t advance and trigger the next inner routine until after the previous inner routine ended.

Now, agreed, the use of an inner routine for a sequence of phrases is unnecessary and confuses the intent. (Evidence of that confusion is that you read my example as concurrent playback.) It’s really a bit of reductio ad absurdum, showing that you can do a sequence using an inner player, but it isn’t an improvement to do so.

@jordan

@jamshark70

That one could be simplified a bit by using a predicate in CondVar.

Could you elaborate?

r = Routine{
	var arrs = Array.rand(10, 48, 72).dup;
	
	var thread_counter = arrs.size;
	var try_open_latch = {
		thread_counter = thread_counter - 1;
		latch.signalAll;
	};
	var latch = CondVar();
	
	arrs.do{|pitches|
		Routine.run({ 
			pitches.do{|m| 
				m.postln; 
				rrand(0.1, 1).wait;
			};
			try_open_latch.();
		})
	};
	
	\waiting.postln;
	
	latch.wait { thread_counter <= 0 };
	
	\done.postln;
};

r.play

Haven’t tested; I think it’s like this.

It’s a minor style difference – try_open_latch can always try to open it, and leave it to the CondVar to ignore the signal when the condition isn’t met. (One advantage is that it would prevent premature firing if some other part of the code is also signaling.)

hjh

[jordan]

@jamshark70

Evidence of that confusion is that you read my example as concurrent playback.

Yup, I thought the cond.wait was one indent to the left.

Thanks for explaining how the predicate works. I think its really confusing (at least in this example) as the curlies look like they should be executed only once, not every time the signal is called, and only advances when true and signalled. Might better be named wait_for_pred_on_signal{...}.

[Spacechild1]

PSA: always use CondVar.wait with a predicate, either as an explicit while loop or as a function passed to wait.

CondVar behave like pthreads/C++ condition variables in the sense that signals are lost if they are sent before the other end calls wait, hence the need for the while-loop. In this respect, CondVar.wait behaves very differently from Condition.wait (which acts more like a semaphore)!

// BAD
(
var c = CondVar();
var message = nil;

fork {
	message = "meow";
	c.signalOne;
};

fork {
	1.wait; // force to run after
	c.wait; // wait without predicate - will wait forever!
	message.postln;
};

)

// GOOD
(
var c = CondVar();
var message = nil;

fork {
	message = "meow";
	c.signalOne;
};

fork {
	1.wait; // force to run after
	c.wait({ message.notNil }); // correct
	message.postln;
};

)

I think the help file should be more explicit about this common pitfall!

[jamshark70]

@Spacechild1

I think help file should be more explicit about this common pitfall!

The help file does recommend writing while explicitly when waiting without a predicate. In simple sync cases, you can get away without the predicate but it’s a bad habit.

@jordan

I think its really confusing (at least in this example) as the curlies look like they should be executed only once, not every time the signal is called, and only advances when true and signalled. Might better be named wait_for_pred_on_signal{...}.

While I see your point, I’m not sure I agree that a more cumbersome name would help that much. It’s a very specific traffic-control behavior, which the language needs to represent symbolically. Between the details of the behavior and the symbolic representation, there’s a gap. Programming languages in general can use careful design to reduce the gap (one of the common complaints about sclang is that the design often doesn’t reduce the gap) but they can’t eliminate it altogether (except maybe FORTH which failed to catch on widely because the expression is too close to machine concepts and not close enough to human concepts).

When struggling with Pure Data a couple years ago, I noticed fallacious thinking in myself which I’ll call the “I know what I’m doing” fallacy. “I’ve been doing this a long time, I know what I’m doing, so if something confused me, then it must be the language or environment’s fault (and therefore, better design would have saved me the confusion).” Sometimes this is true. Sometimes it’s just… hard to grasp, and a “better” design would only shift the confusion somewhere else. So I’m slowly trying to train myself to just accept it when I don’t get something right away. But that’s difficult…

hjh

[jordan]

Ah, so the predicate get called when the signal fires, and when wait is first called!

While your examples are waiting on a variable to be set (which is much nicer using @scztt’s Deferred), the code at the start of this thread doesn’t have an obvious returned value.

Are you suggesting it would be better with Condition?

Deferred example

fork {
	var d = Deferred();
	fork {
		d.using({ 40 });
	};

	1.wait;
	d.wait;
	
	d.value.postln
}

[jordan]

Honestly, I think what you are trying to do is really common and Supercollider supporting this in an easier way would be better…

So I made a quark.

Quarks.install("https://github.com/JordanHendersonMusic/Barrier")

Its called Barrier, has two methods, forkAll and forkN.

r = Routine {
	var barrier = Barrier.forkAll(
		{1.0.rand.wait; \a.postln},
		{1.0.rand.wait; \b.postln},
		{1.0.rand.wait; \c.postln},
		{1.0.rand.wait; \d.postln}
	);

	\waiting.postln;
	barrier.wait;
	\done.postln
};

r.play;

r = Routine {
	var func = {|n| 1.0.rand.wait; n.postln};
	var barrier = Barrier.forkN(func, 10);

	\waiting.postln;
	barrier.wait;
	\done.postln
};

r.play

So your use case would look like…

r = Routine{
	var pitches = Array.rand(10, 48, 72);
	var func = {
		pitches.do{ |m| 
			m.postln; // or the synth stuff
			rrand(0.1, 1).wait;
		};
	};
	var barrier = Barrier.forkN(func, 2);
	
	\waiting.postln;
	barrier.wait;
	\done.postln;
};

r.play

It also has all the documentation.
Wraps up all the synchronisation stuff, and I THINK its impossible to use wrong and accidentally block the thread.

Edit: typo

[Spacechild1]

@jamshark70

The help file does recommend writing while explicitly when waiting without a predicate.

To be fair, the help file for CondVar does not claim that you can omit the predicate:

This leads to two general ways of using CondVar:

• Manually checking your condition
  …
• Have CondVar check your condition for you:
  …

My complaint is that it does not make it clear why the predicate is always necessary and how CondVar differs from Condition.

[Spacechild1]

Ah, so the predicate get called when the signal fires

I’m not sure I understand. The predicate is simply checked before .wait, to avoid waiting (forever) in case it is already true. You can check the predicate manually in a while loop, or pass it as a function to the .wait method.

While your examples are waiting on a variable to be set … the code at the start of this thread doesn’t have an obvious returned value.

In that case you need to use a boolean variable, e.g. var done = false;, and the predicate would simply be { done }. This is how “real” condition variables work. To be clear: I am not advertizing the use of condition variables in user code (see below), I’m just trying to explain how they are supposed to be used.

Are you suggesting it would be better with Condition?

Condition is the traditional solution. CondVar has been added very recently (2021) and while the commit message (Add CondVar · supercollider/supercollider@3ea8a09 · GitHub) advertizes it as a replacement for Condition, I am a bit sceptical. “real” condition variables are low-level synchronization primitives (the help file even acknowledges this!); they can be used to implement various useful high-level constructs, such as futures, channels, etc., but IMO they should not be casually used in user code. Condition apparently has some problems (I don’t remember the details), but Deferred looks much better as a replacement for Condition than CondVar.

Actually, another option is FlowVar:

(
var f = FlowVar();

fork {
	2.wait;
	f.value = "meow";
};


fork {
	1.wait;
	f.value.postln; // this waits until 'f' is set
};

)

Again, the value can also be a simple Boolean, if you just need to wait for completion.

IIRC, FlowVar had been broken for years, which is probably why nobody seems to use it, but now it appears to work just fine.

[jamshark70]

@Spacechild1

Condition is the traditional solution.

One issue is that Condition (in its original version) couldn’t handle timeouts. It looked like it could, but thread timing was borked if the condition was signaled successfully. I proposed a fix for this, but it was decided that this was structurally inadequate and CondVar was added in place of that fix. The specific reasons for this, I don’t recall but they’re still all in the github issue/PR history.

Edit: One good reason to have put CondVar in place is that a structurally sound condition variable would be a better foundation for futures etc than Condition… but that development effort didn’t happen, at least not in the main library. So we just started using it directly.

In any case, thread sync is a bit of a rough edge. Since many SC users aren’t computer scientists, I think it would be helpful if we could decide on one general purpose approach and document it in a guide or tutorial. The nuances of futures vs channels vs the-next-fancy-thing-that-will-be-forgotten-two-years-later may be unnecessary complication.

Jordan’s Barrier reads nicely! But I have a bit of feedback… One of the main thread sync cases in SC is waiting for buffers to read from disk, which requires an explicit signal (rather than an implicit signal from reaching the end of a routine’s function)… s.sync pushes the sync job off to the server, and is probably adequate most of the time (why would one use Barrier for this when s.sync is easier?) but it raises the thought that there may be other sync jobs that need explicit signaling per sub-thread. If the only lock-release mechanism in Barrier is to exit the function, then this function needs to know when to signal, which just moves the sync problem to another part of the code. It might be a good idea to keep an IdentitySet of threads, and provide a public method release { |thread| set.remove(thread); condVar.signalAll } and the predicate would be { set.isEmpty }. (A countdown doesn’t account for a user accidentally releasing the same thread twice – they shouldn’t, of course, but you have no way to detect that case without keeping a collection.) The usage style looks very convenient – maybe this adjustment would cover more cases.

hjh

[jordan]

@jamshark70

“I know what I’m doing” fallacy.

When writing in a language designed for something one isn’t familiar with, then sure. But here, SuperCollider is aimed at musicians, which should mean, if it is user code and it isn’t obvious, it is wrong. The alternative is to require people spending years and years learning software; which no one wants to do and requires great privilege to afford the time to do so. That doesn’t mean people don’t have to learn, just that it should be accompanied by an ‘ahhh, that is so much easier’ moment, which is absent here. I think the whole narrative of music tech being filled with people ‘making do’ is a myth and completely ignores our agency in creating technology, and omits all the tools people from as far back as the 50s made.

Here, it seems to be agreed that the synchronisations that exist aren’t really aimed at the user but as a backend tool, so with the idea of agency in mind I’m gonna make a new thread and try to change this. I think that will be a better place to discuss feedback on what I wrote, and perhaps we might be able to come up with a few more that make using multiple processes easy and, most importantly, impossible to get wrong.

[jamshark70]

@jordan

But here, SuperCollider is aimed at musicians, which should mean, if it is user code and it isn’t obvious, it is wrong.

For what it’s worth, I 100% agree that this is a worthy ideal; I just don’t think it’s practically achievable. We can push the gaps further and further into the distance through careful design, but the gaps can’t be eliminated. (To be clear, I’m absolutely not saying that one shouldn’t try to improve clarity and tranquility – just that the ideal of a perfectly transparent and impossible to mess up language is utopian. To whatever partial degree the situation can be improved, that’s a good thing.)

My favorite example is from Max, which aims to demystify programming logic and make it accessible to non-programmers.

[456(
|
[append 123]

Is the result 123 456 or 456 123?

I have to think about it. Every time. About 30-50% of the time, my first instinct for append vs prepend is wrong. And I’m not convinced it would really be that much clearer with something like [append-argument-after-input].

It’s a delicate balance between tokens expressing the full meaning (cumbersome) and representing much more complexity than the token itself states directly.

There’s a lot of room for improvement in SC, and I hope the thread sync discussion bears fruit (since this has been a pain point for a long time). Just pointing out that the ideal solution is likely to have a gap, just in a different location from where it is now.

hjh

[semiquaver]

@jamshark70

Just pointing out that the ideal solution is likely to have a gap, just in a different location from where it is now.

Let’s not disparage improvements in advance!

[jamshark70]

@semiquaver

Let’s not disparage improvements in advance!

I’ve probably beaten it to death so I’ll lay off after this post, but… none of anything I said should be taken to discourage improvements. About life in general, nobody would say it’s pointless to try to improve anything because perfection is unattainable. I think the same for this topic. Absolutely, yes, make a better way! (I quite like a lot of Barrier, with the suggestion noted above.) But also, be realistic that programming language text doesn’t explain everything (and if it did, it would probably be cumbersome to use): to use the gap to best advantage rather than seeing gaps as fatal flaws.

Very curious to see the outcome of the other thread

hjh