Loading Many Buffers (Server Idling)

Questions
1

I’m using the following bit of code to sort through folders, find audio files, and create 1-second fragments from the files. Running this code causes the Server status to turn yellow for a long time. Maybe the issue is using SoundFile to count the number of frames in the buffer before loading it? I think I am misunderstanding something about asynchronous language here - can anyone here help get me sorted out? Cheers.

~newfolder =  PathName(...)
~newfolder = ~newfolder.deepFiles.collect{|x|
			if (
				(x.extension=="flac")||
				(x.extension=="mp3")||
				(x.extension=="wav")||
				(x.extension=="aif")||
			    (x.extension=="FLAC")||
				(x.extension=="MP3")||
				(x.extension=="WAV")||
				(x.extension=="AIF"))
			{[x, SoundFile.openRead(x.fullPath.asString).numFrames]};
};

~final = 100.collect{
		var startFrame, dur = 1, select = (~newfolder.size).rand;
	    startFrame = ~newfolder[select][1].rand;
		dur = dur * 44100;
		dur.postcs;
		Buffer.read(s, ~newfolder[select][0].fullPath.asString,
				startFrame,  dur);
	};
2

There’s not much documentation about this, but I took a quick look at the source code to find out.

Loading the buffers uses a server command, /b_allocRead. This is a “sequenced command,” meaning it’s asynchronous, and they are executed one by one in a queue. That is, you can dump a hundred /b_allocRead-s into the queue, and the server will step through the queue until they’re all finished.

The server status bar uses a command /status. This is also a sequenced command.

So I think what’s happening is:

  1. Your code dumps a hundred allocRead requests into the queue.
  2. A fraction of a second later, the IDE asks for the server’s /status. … But this request goes at the end of the queue, after all of the allocReads.
  3. So… the server has to finish reading all the buffers before answering the /status request.

One way to handle this is to accept it, and just wait for it to finish.

Another way is to chunk the buffer loads. It isn’t immediately obvious how to do that, but this is what I’d do. By sending 5 requests, then waiting, then 5 more etc., there are brief windows where the IDE can slip in its ‘/status’ request.

(
fork {
	var chunkSize = 5;
	// countdown: number of requests to send before waiting
	var requestsPending = chunkSize;
	// countdown: how many uncompleted reads remain
	var pending = chunkSize;
	var cond = CondVar.new;
	var num = 100;
	~final = num.collect {
		var startFrame, dur = 1, select = (~newfolder.size).rand;
		var buf;
		startFrame = ~newfolder[select][1].rand;
		dur = dur * 44100;
		buf = Buffer.read(s, ~newfolder[select][0].fullPath.asString,
			startFrame, dur, action: {
				// when one request finishes,
				// drop the count
				pending = pending - 1;
				// and see if it's OK to continue
				cond.signalAll;
			}
		);

		// just sent a request, so one less is pending		
		requestsPending = requestsPending - 1;
		// no more requests for this chunk
		if(requestsPending == 0) {
			// will not advance past this point
			// until after the 'chunkSize' pending requests have cleared
			cond.wait { pending == 0 };
			requestsPending = chunkSize;
			pending = requestsPending;
		};

		buf  // loop should return the buffer
	};
	"done".postln;
}
)

BTW:

SoundFile.openRead(x.fullPath.asString).numFrames

One issue here is that you’re opening files without closing them, causing file handles to accumulate in the language. Better practice is to close a file when you’re finished with it.

Actually, in SoundFile, the header properties are still available after closing the file, so you can just do SoundFile.openRead(x.fullPath.asString).close.numFrames.

EDIT: First version of the code block didn’t handle the return value from each loop cycle correctly – fixed.

hjh

1 Like