Is it possible to implement more user interaction Ugens?

[prko]

Hello,

This question suddenly came to mind while reading [Best way to trigger a sample to play].

We have the following Ugens for user interaction:

• KeyState.kr
• MouseButton.kr
• MouseX.kr
• MouseY.kr

They seem to work without latency between client (sclang) and server (scsynth or supernova). Am I correct?

Could the following also be implemented as Ugens for user interaction?

• MIDI In
• HID
• SerialPort
• (Some of GUI elements: .interactionView of Plot, Button, CheckBox, Knob, NumberBox, RangeSlider, Slider) ← I think they cannot be implemented.

This will solve many of the latency problems of client-server interaction.

[jamshark70]

@prko

This will solve many of the latency problems of client-server interaction.

I don’t think it will, actually.

The server processes hardware buffers all at once: if you have a 512-sample hardware buffer, and 64-sample control blocks, then there are 8 control blocks per hardware buffer. Assuming 48000 kHz (for a bit easier arithmetic):

• At time 0 ms, process control blocks 0 1 2 3 4 5 6 7 as fast as possible, and give the 512 samples to the audio driver.
• At time 10.6666 ms, control blocks 8 9 10 11 12 13 14 15.
• and so on.

Currently, we get MIDI/HID/OSC in the language and send messages to the server. Local UDP transmission is extremely fast, like a tenth of a ms, so this is negligible. If the OSC message arrives in the server at, say, 12 ms (i.e. 1.4444 ms after the hardware buffer boundary)… it’s too late for that hardware buffer. The next opportunity is the next hardware buffer, 21.3333 - 12 = 9.3333 ms later.

If, instead, MIDI goes directly into a MIDI UGen and bypasses the language, you have… the same problem. DSP time already moved on. (Well, conceivably the UGen could report the data as soon as they’re available, and if the server is loaded up with a ton of UGens, this could be in the middle of a hardware buffer. But because control block timing is not monotonic, there’s no guarantee of correct timing of the MIDI message’s effect. Indeterminate timing would be worse than the current situation, where you can be sure it will be the next hardware buffer, or according to timestamp.)

The latency problem was never the network or the split between client and server, at least not on a local machine.

So comprehensive MIDI and HID UGens would be a large engineering effort to fail to solve the problem, which I think is unlikely to attract developer attention.

hjh

[semiquaver]

Just out of curiosity, what does the story look like for control blocks of 2 samples… or 1?

[jamshark70]

That’s not relevant, because the timing of server DSP is organized around hardware buffers.

The audio driver calls into scsynth and scsynth needs to deliver the hardwareBufferSize samples, as quickly as possible. “As quickly as possible” means that it does not space out the control blocks so that each one takes ControlDur.ir seconds. If you have a 512-sample hardware buffer, and 1-sample control blocks, then the first hardware buffer will process 512 control blocks as fast as possible, and then wait for the next hardware interrupt to do the next 512 control blocks as fast as possible.

(FWIW, I thought for years that control blocks were monotonically spaced out… but that was incorrect.)

hjh

[VIRTUALDOG]

@jamshark70

Currently, we get MIDI/HID/OSC in the language and send messages to the server. Local UDP transmission is extremely fast, like a tenth of a ms, so this is negligible

Correct me if I’m wrong but I believe the larger chain is:

1. Receive external input in sclang control thread
2. Convert and store data and wait for interpreter lock (indefinite duration)
3. Acquire interpreter lock and dispatch callbacks (indefinite duration / potential GC pauses)
4. Marshal outgoing OSC messages to UDP socket

Is it really true that this doesn’t result in occasional latency issues?

@jamshark70

Indeterminate timing would be worse than the current situation, where you can be sure it will be the next hardware buffer, or according to timestamp

I think the current situation is already fairly indeterminate, isn’t it? If you handled MIDI in the server you’d have a bit less latency, wouldn’t need to use a client at all for accepting MIDI, and wouldn’t need to worry about blocking MIDI processing by hanging onto the GIL. Just my naive impression though.

[jamshark70]

@VIRTUALDOG

Correct me if I’m wrong but I believe the larger chain is:

1. Receive external input in sclang control thread
2. Convert and store data and wait for interpreter lock (indefinite duration)
3. Acquire interpreter lock and dispatch callbacks (indefinite duration / potential GC pauses)
4. Marshal outgoing OSC messages to UDP socket

Is it really true that this doesn’t result in occasional latency issues?

AFAIK the summary of the process is correct, but, in 20 years (going on 21), in practice, I haven’t experienced concrete, significant delays on the language side (at least, not more significant than hardware buffer quantization). That includes when I was using a 2002(?)-era MacBook G4 with a single core, where running 8-10 Saw UGens at the same time risked dropouts. Even then, MIDI was pretty well responsive. I never measured carefully for latency, but I could play MIDIIn → MIDIVoicerSocket → Voicer and it was usable on stage. At that time, when scsynth was busy, I did see delays in GUI refreshing on AppClock, but I don’t remember significant sequencing delays. (If that were happening, I would have been upset enough about it to switch to other software.)

Now I can easily hit 1000 Saws and have CPU time to burn (~35%!), and sclang is doing its work on another core, so these problems are now a couple orders of magnitude less severe.

My live coding system does the bulk of the algorithmic composition on barlines, so this is something of a worst case for practical sequencing. Based on benchmarking, if I’m running 3 or 4 complex sequencers and a few other simpler ones (like drums), I’d expect maybe 15(?) ms bursts where the interpreter is busy resolving the next bar’s worth of data. I haven’t been live-playing MIDI notes concurrently, so I can’t give any measurement on that. I do update controller values based on incoming OSC, and I haven’t noticed any unacceptable delays.

GC pauses are theoretically possible but I’ve never seen it in practice, not even on that old G4.

@VIRTUALDOG

If you handled MIDI in the server you’d have a bit less latency…

Sure, and controllers would be relatively easy to handle in the server. Note on/off polyphony would be very difficult. You could queue up a number of gate == 0 synths and round-robin distribute notenum, velocity and gates to them, and have the language queue up more of them as those are consumed. The complexity of this logic, however, IMO wouldn’t be worth it if sclang’s MIDI response is faster than latency due to the hardware buffer size.

Not to say that your concerns are invalid, not at all, but I haven’t seen sclang’s MIDI keyboard response to be any worse than, say, Reaper’s.

hjh

[Spacechild1]

It might be also worth noting that you can always use a dedicated sclang process just for MIDI input, in case the main process has intolerable CPU bursts.

[TXMod]

@Spacechild1

It might be also worth noting that you can always use a dedicated sclang process just for MIDI input, in case the main process has intolerable CPU bursts.

@Spacechild1 can you explain a bit more about this - or maybe show some code?
Many thanks,
Paul