Debugging processes stopped audio on the server

Dear all,

This is a general question for SC users who have had installations running for longer time periods with SuperCollider.

How do you go about monitoring and logging debug information from your running SC installation? Do you write anything to a log file? Send email notifications when something misbehaves?

I’m running into one particularly annoying issues right now… I’ve got a long-running (24 hours/7 days a week) stream generated in SuperCollider, where occasionally some ndefs stop producing audio. Why this is I’m not sure - perhaps something is exploding somewhere? The server and language continue running happily, it’s just that the synths stop producing audio. I’m at a loss for how to debug such a happening, as it happens at some time I don’t know… usually in the middle of the night or some unexpected time during the day. I’d like to figure out a way of getting some debugging info to address this issue - maybe get notified when the synth stops working and maybe some information about why it stopped.

Does anyone have any tips or suggestions?

thanks a bunch!

jchaim

Do all of them stop making audio or just some of them? If it is all of them, I imagine something is producing a nan and making infinite dc offset. This will produce silence because a nan plus anything is a nan.

Something that may work is downloading the SafetyNet quark and adding .zap to all signals going to outputs. A better solution would be trying to figure out which Synth is causing the problem and removing the issue-causing code. You could add a .poll to all output values and then you could see when the signal goes wacko.

Sam

1 Like

Hey @Sam_Pluta ~ sounds like a reasonable assessment! I’m not sure how many of them are going silent, to be honest. This is all running on a headless sclang and the problem occurs days into the running time of the installation… so I’m at somewhat of a loss for how to even monitor the status of the system in order to be able to answer your question. :frowning_face:

SafetyNet / .zap looks like a useful tool! But if I’m understanding the code correctly, .zap seems to only replace the bad values with good ones - rather than helping you debug the situation. Something that would be really useful would be to have some kind of monitor that can notify me when a nan occurs in a specific synth, along with some state information about the parameters of the enclosing synth?

var mightbebad = RLP.ar(WhiteNoise.ar(), \freq.kr(100));
MonitorForBadValues.kr(mightbebad);

When a bad signal is encountered it sends an OSC message to a language listener that logs the data, or sends a notify message over SMTP?

Hi,

A combination of CheckBadValues and SendReply (or SendTrig) could do something like this. Here’s a quick, untested starting point:

var mightbebad = RLPF.ar(WhiteNoise.ar(), \freq.kr(100));
var status = CheckBadValues.ar(mightbebad, ugenID);
SendReply.kr(status > 0, '/badValueMonitor', status);
1 Like

Thanks - I’m building out a more robust solution based on this approach. :slight_smile:

Hey @Sam_Pluta and @jpburstrom …many thanks for your thoughts on this.
I’ve started to assemble some classes based on this approach to use for remote-monitoring of installation setups.

The classes are here, just in case anyone in the future would like to make use of them: GitHub - jreus/RemoteTools: A collection of tools and utilities for running SuperCollider remotely, as a headless installation or otherwise.