Access a subfolder from an Environment in a .scd file in a different subfolder

hey,

I would like to access a subfolder from an Environment (ProtoDef) in an .scd file in a different subfolder, where both subfolders are stored in a main folder. In the main folder is the main.scd file which i use for booting, which evaluates .scd files from some of the subfolders inside s.waitForBoot with "%/ProtoDefs/*.scd".format(thisProcess.nowExecutingPath.dirname).loadPaths;

In one of the .scd files with the Environment inside the subfolders i have:

path = PathName(thisProcess.nowExecutingPath).parentLevelPath(2);
folderPath = PathName(self.path) +/+ folderName;

The parentLevelPath(index) method from the atk toolkit is accessing the correct path. Is there a way where i could stick with the core library? The subfolder i want to access has soundfiles which i would like to process or load into buffers. I dont know if my explanation was good, have been trying my best.

thanks :slight_smile:

Isn’t parentLevelPath(2) just path.dirname.dirname?

hjh

yeah, thanks thats true :slight_smile:

Was trying to change my main bootup file to be based on ProtoDefs stored in some subfolders, but put this on ice for a moment.

There have been several problems with passing “global variables” for paths to Environments, switching s.waitForBoot for Routines with s.bootSync and some investigations with CondVar (i guess im just too dumb, I cant really understand in what case and how this has to be implemented correctly) and the difference of adding to ServerBoot or ServerTree on server boot.
Ive tried to load ProtoDefs from the subfolders and then evaluate the Prototypes with some sort of sync and add them to ServerBoot. Adding them to ServerTree did work, but then everytime i hit command period the buffers would load again to the IdentityDictionary.

I think i just forget about that and stick with just functions which i add to ServerBoot with waitForBoot and have the main path as a “global variable” in the main file.

This, I’m not clear what you mean. One solution for global environment variables is to access globals through topEnvironment:

~path = "/xxx/yyy/zzz";

e = Environment.use {
	~path = topEnvironment[\path];
	...
};

But it’s hard to be specific based on the given information.

Hm, is it necessary, though? I just looked briefly at the code for bootSync and it uses waitForBoot, so it looks to me like fork and bootSync is just a more complicated wrapper around waitForBoot + running in the implicit routine in which waitForBoot operates.

For CondVar, normally you should follow this basic pattern:

(
fork {
	var cond = CondVar.new;
	
	// now we do something that will take a little time
	b = Buffer.read(s, Platform.resourceDir +/+ "sounds/a11w1k01.wav",
		action: {
			// and the completion action signals to go on
			cond.signalAll;
		}
	);
	
	// after initiating the async action, halt the thread temporarily
	cond.wait;
	
	("buffer loaded: " ++ b).postln;  // thread continues after the signal
};
)

If you’re not sure what you’re doing with CondVar and you’re doing something significantly different from this, probably you’d run into trouble. Variations on this theme are fine.

ServerBoot is for initialization actions that should take place after the server boots, but not upon cmd-dot.

ServerTree is to initialize a node tree after cmd-dot. Because buffers are not cleared upon cmd-dot, ServerTree wouldn’t be the right place to put buffer loading.

Robust initialization is tricky in SC. The init script for my live coding environment is quite complicated, and pretty stable at this point. I’m happy to give more advice about specific problems you’re running into, but would need more details.

hjh

hjh

thanks for all your suggestions, this one is working already :slight_smile:

defining the global path in the main.scd file in the root of the folder with:
~path = PathName(thisProcess.nowExecutingPath).parentPath;

and then loading the ProtoDefs in the main.scd file from the subfolders via:
"%/ProtoDefs/*.scd".format(thisProcess.nowExecutingPath.dirname).loadPaths;

and evaluating the Prototype in the main.scd file with its beforeInit functionality:
~soundfileFunctions = Prototype(\soundfileFunctions) { ~path = topEnvironment[\path] };

and then load the waveforms into buffers via the makeBuffers method, which i have defined in the ProtoDef from an abitrary scd file:
~soundfileFunctions.makeBuffers("waveforms");

Will have a look at the other suggestions and try to set up the loading of the ProtoDefs from the subfolders and then evaluating the Prototypes and add them to ServerBoot inside s.waitForBoot once more.
The current work in progress structure looks something like this, without taking care of ServerBoot and actually using the .makeBuffers method, probably should add self.makeBuffers; into ~initDef or ~init in the ProtoDef, dont know:

(
Routine({
	
	var c = Condition.new;

	~path = PathName(thisProcess.nowExecutingPath).parentPath;
	"%/ProtoDefs/*.scd".format(thisProcess.nowExecutingPath.dirname).loadPaths;
	
	s.bootSync(c);
	"server has booted".postln;
	
	~soundfileFunctions = Prototype(\soundfileFunctions) { ~path = topEnvironment[\path] };
	
	s.sync;
	
	//~soundfileFunctions.makeBuffers("waveforms");
	
}).play;
)

My current ProtoDef(\soundfileFunctions) has different methods for either loading soundfiles into buffers or others for processing them and saving them to disc. Only the buffer loading should be done when booting the server. Here is just the makeBuffers method, could probably be improved :slight_smile:

(
ProtoDef(\soundfileFunctions) { |path|

	~init = { |self|

		self.soundfiles = IdentityDictionary.new();

	};

	~makeBuffers = { |self, folderName|

		var folderPath = PathName(self.path) +/+ folderName;

		Routine({

			if(self.soundfiles.notEmpty) {
				self.soundfiles.do{ |arrayOfBuffers|
					arrayOfBuffers.do{ |buffer| buffer.free }
				};

				s.sync;

				self.soundfiles.clear;

				"buffers are free now!".postln;

			};

			s.sync;

			folderPath.folders.do{ |subfolder|

				var arrayOfBuffers = subfolder.files.collect{ |soundfile|
					Buffer.read(s, soundfile.fullPath);
				};

				self.soundfiles.add(subfolder.folderName.asSymbol -> arrayOfBuffers);

			};

			s.sync;

			"soundfiles loaded".postln;

		}).play;

	};
};
)

The ProtoDef class also comes with server hooks, maybe these would come in handy:
From the helpfile:

(
ProtoDef(\testHooks) {
    // class hooks
    ~defOnServerBoot = { "[DEF:test] server boot".postln };
    ~defOnServerTree = { "[DEF:test] server tree".postln };
    ~defOnServerQuit = { "[DEF:test] server quit".postln };
    // instance hooks
    ~init = { |self|
        self.id = UniqueID.next;
        ServerBoot.add(self);
        ServerTree.add(self);
        ServerQuit.add(self);
        "[test] created instance %".format(self.id).postln;
    };
    ~doOnServerBoot = { |self|
        "[instance:test%] server boot".format(self.id).postln
    };
    ~doOnServerTree = { |self|
        "[instance:test%] server tree".format(self.id).postln
    };
    ~doOnServerQuit = { |self|
        "[instance:test%] server quit".format(self.id).postln
    };
}
)

Im not sure where to start here, i couldnt figure out how to load ProtoDefs from subfolders, evaluate some methods via Prototypes (for example loading Soundfiles into Buffers) with sync and add them to ServerBoot. Asking questions about an extension doesnt make life easier haha. But its a light extension should be fine for users familiar with Prototyping.

I find it weird to register the ProtoDef itself with the ~defOnServerBoot “server hooks”. This does actually work to load the buffers if i put self.makeBuffers into ~initDef, no syncing problem.
I would find it more logical to load all the ProtoDefs from the subfolders, add a sync of some sort and then evaluate the Prototypes with some methods and add those on ServerBoot maybe with ~doOnServerBoot. What do you think about that?
Would you suggest there should be a ProtoDef bootup instance, instead of using methods from different ProtoDefs on ServerBoot, for example picking the buffer loading from one ProtoDef (which organizes soundfile processing) while picking another method from a different ProtoDef for setting up groups and busses?

At face value, this seems like a good idea… I’m curious why you aren’t doing that?

I have a feeling that you’re overcomplicating the problem by perhaps trying to do too many things at one time…? That is, in your first paragraph:

… separate each of these problems:

  • how to load ProtoDefs from subfolders
  • evaluate some methods via Prototypes (for example loading Soundfiles into Buffers) with sync
  • add them to ServerBoot.

First off, “load ProtoDefs from subfolders” is completely independent of booting the server, isn’t it? So, just remove that from waitForBoot entirely. Load them before booting the server:

"mysubfolder/*.scd".pathMatch.do { |path| path.load };

s.waitForBoot {

}

Then, “add them to ServerBoot” is sync, isn’t it? So, talking about both sync and ServerBoot is redundant, and adding extra complexity that is confusing you. So, do one or the other, but not both.

So I gave it a shot.

(
ProtoDef(\demo, {
	~server = Server.default;
	~defOnServerBoot = { |self|
		self.buf = Buffer.read(self.server, Platform.resourceDir +/+ "sounds/a11wlk01.wav".debug("loading"));
	};
});
)

// then...
s.boot;

… and it does load the buffer. So I think this is what I would recommend.

  1. Do not boot the server first.
  2. Load the ProtoDefs, with defOnServerBoot methods.
  3. Boot the server.

I think here you might be getting ahead of yourself. It will probably be more efficient to work on one problem at a time. First make sure that the loading / serverboot stuff is working. Then, how to factor the resources into different ProtoDefs is a separate question.

When getting confused or overwhelmed as in this thread, break problems down into subproblems and do not move ahead until you’ve solved the current subproblem. Running in 20 directions at the same time will not get you where you want to go.

(I organize my work into “musical processes,” which include a pattern plus the resources needed to perform the pattern. Those resources include MixerChannels, any buffers, etc etc. I don’t think in terms of a buffer Proto and a bus Proto. But object design is very personal. I won’t tell you what to do here.)

hjh

1 Like

Overnight, a couple of other strategies occurred to me.

On-demand ProtoDefs

A ProtoDef is made from a function. You could store a library of functions to initialize different types of ProtoDef:

e.g., protodefs.scd:

Library.put(\protoDefs, \busManager, {
	~server = Server.default;
	~otherVariables ...
	~initDef = { |self|
		...
	}
});

Library.put(\protoDefs, \bufferManager, {
	~server = Server.default;
	~otherVariables ...
	~initDef = { |self|
		...
	}
});

This can be done anytime (before or after booting the server), because it’s not doing any of the actual work yet.

Then, after the server is booted, create ProtoDefs as needed:

ProtoDef(\buses, Library.at(\protoDefs, \busManager))

So the functions are like class definitions, and the ProtoDef calls are like making instances of those classes… loose analogy but apt. In the main class library, you don’t expect that every class is going to instantiate during language startup – they are definitions to use when you need them.

Again, the idea is to separate loading the files away from the server boot process.

CondVar how to

CondVar would allow you to create ProtoDefs at any time, but defer their initialization until after server starts.

The catch is that something needs to signal the CondVar – meaning that there has to be a shared CondVar, or collection of them. The server boot process can’t signal CondVars that it doesn’t know about.

So I’d do maybe like this:

(
~serverBootCond = CondVar.new;

~bootSignal = {
	s.waitForBoot {
		~serverBootCond.signalAll
	}
};

ProtoDef(\demo, {
	~inited = false;
	~initDef = { |self|
		fork {
			~serverBootCond.wait { s.serverRunning };
			"ok, server is up, ready to init stuff".postln;
			self.inited = true;
		}
	}
});
)

ProtoDef(\demo).inited  // false

~bootSignal.value;
... snip...
Shared memory server interface initialized
ok, server is up, ready to init stuff

// wait... then:
ProtoDef(\demo).inited  // true

The advantage of this approach is: if you add a ProtoDef after booting the server, it will go ahead with the initialization (because the condition s.serverRunning is already met).

So, three approaches in the last couple of messages:

  • Load ProtoDefs first, then boot the server, and use defOnServerBoot.
  • Load functions first, then boot the server, then create ProtoDefs from the functions (with no need to sync because the server will already be up).
  • Load ProtoDefs at any time, with reference to a global server-boot CondVar.

Use only one of these – mixing and matching will only cause you trouble.

hjh

wow, thank you very much for taking the time looking into ProtoDef and coming up with a bunch of solutions :slight_smile:
I will have a look at all three possible implementations and pick the one which makes the most sense to me (probably the CondVar version) :slight_smile:

I think I have mixed and matched approach 1.) and 3.), which is probably not the best solution. I thought for registering with ServerBoot, Tree and Quit the “server hooks” inside ProtoDef might result in a more uniform approach instead of implementing ServerBoot with CondVar and ~initDef and then having to figure out a completely different approach for ServerTree and ServerQuit. But could be mistaken.

1.) im creating globel dictionaries here and access them with self.something = topEnvironment[\something] in ~initDef, which results in parallel storing of the data, which is probably not desirable. What do you think?
2.) If have been trying to come up with a ~cleanUp function, which i would like to register with ServerQuit, here only for the buffers (actually two versions, dont know which one to pick) and what should additionally be cleaned up.
3.) for the ~makeGroups functions i would also like to store the groups in a dictionary, currently not sure how to approach that function. At some point in the past i needed to preserve the group ids after cmd period and would like to keep that implementation from Preserving groups id or reference after CmdPeriod - #5 by jamshark70

If you have any ideas let me know :slight_smile:

(
Routine({

	// 1.) create boot condition
	~serverBootCond = CondVar.new;
	~inited = false;

	~bootSignal = {
		s.waitForBoot {
			~serverBootCond.signalAll;
		}
	};

	// 2.) initialize "global" dictionaries
	~soundfiles = IdentityDictionary.new();
	~busses = IdentityDictionary.new();
	~groups = IdentityDictionary.new();

	// 3.) load ProtoDefs and register functions with ServerBoot/Quit/Tree
	ProtoDef(\demo, {

		~initDef = { |self|
			self.soundfiles = topEnvironment[\soundfiles];
			self.busses = topEnvironment[\busses];
			self.groups = topEnvironment[\groups];
		};

		~defOnServerBoot = { |self|
			self.makeBuffers;
			self.makeBusses;
		};

		~defOnServerTree = { |self|
			self.makeGroups;
		};

		~defOnServerQuit = { |self|
			self.cleanUp;
		};

		~cleanUp = { |self|

			if(self.soundfiles.notEmpty) {
				self.soundfiles.do{ |arrayOfBuffers|
					arrayOfBuffers.do{ |buffer| buffer.free }
				};

				self.soundfiles.clear;

			};
/*
			self.soundfiles.keysValuesDo{ |key, value|
				if( value.isKindOf(Buffer) ) {
					value.free;
				};
				self.soundfiles.removeAt(key);
			};
*/

		};

		~makeBuffers = { |self|
			var buffer = Buffer.read(self.server, Platform.resourceDir +/+ "sounds/a11wlk01.wav".debug("loading"));
			self.soundfiles.add(\buffer -> buffer);
		};

		~makeBusses = { |self|
			self.busses.add(\audio -> Array.fill(5, { Bus.audio(s, 1) }));
			self.busses.add(\fx -> Array.fill(3, { Bus.audio(s, 2) }));
		};

		~makeGroups = { |self|

			//self.groups.add(\main -> Group.basicNew(s, -1));
			//self.groups.add(\fx -> Group.basicNew(s, -1));

			s.bind {

				if(~mainGrp.isNil) {
					~mainGrp = Group.basicNew(s, -1);
				};

				if(~fxGrp.isNil) {
					~fxGrp = Group.basicNew(s, -1);
				};

				~mainGrp.nodeID = s.nodeAllocator.alloc;
				~fxGrp.nodeID = s.nodeAllocator.alloc;

				s.sendBundle(nil, ~mainGrp.newMsg(nil, \addToHead));
				s.sendBundle(nil, ~fxGrp.newMsg(~mainGrp, \addAfter));

			};
		};

	});

	// 4.) boot the server
	~bootSignal.();

	~serverBootCond.wait { s.serverRunning };
	"server has booted".postln;
	~inited = true;

}).play;
)

// currently stored in ~busses and self.busses
~busses[\audio][0];
ProtoDef(\demo).busses[\audio][0];

// currently stored in ~soundfiles and self.soundfiles
~soundfiles[\buffer].plot;
ProtoDef(\demo).soundfiles[\buffer].plot;

// currently trying to figure out how to restructure ~makeGroups
~groups[\main];
ProtoDef(\demo).groups[\main];