Organising large project - What do you think of my solution? what are yours?

How do you manage large projects across multiple files? I have seen many people struggle with this, including myself, so I have made a solution and would really like to know what people think!

What is the problem

I think most people’s solution to managing large project is to break it down into multiple files storing everything in global variables and then load in the files. I have found in my own work that is becomes very cumbersome and hard to manage as a project grows.
I think there are two main problems: first how do you keep track of which order the files are evaluated in, and their dependencies; and second, how to do manage changing global variables. My proposal for each is, project targets and imports/files-as-functions.

Targets

Inspired by programs like CMake where you define multiple targets and link their dependencies together I have this basic structure.

~project = Project()
.addTarget('A', func: {
	~a = 1;
})
.addTarget('B', depends: 'A', func: {
	~b = ~a * 2;
});

~project.run(\B);

First a project with a list of targets is defined, these are stages that depend on each other. Above, 'B' requires 'A' to be evaluated before it, 'B' depends on 'A'. Internally these are all wrapped inside of s.waitForBoot and s.sync is called between each target. It is also possible to do preboot targets. These cannot depend on ‘postboot’ targets, but can depend on each other.

~p = Project()
.addPreBootTarget('A', func: {
	~a = [1, 2, 3];
})
.addPreBootTarget('B', func: {|condvar|
	5.wait;
	~b = 4!10;
	condvar.signalAll;
})

All targets are run inside a forked function, and if the func parameter has an argument, a CondVar is automatically created that will be ‘waited’ upon.

Here is a slightly larger example showing multiple dependency paths.

~p = Project()
.addTarget(\B, func: { 
	2.wait;
	~b = 4;
})
.addTarget(\A, func: { ~a = 1 })
.addTarget(\C, depends: \A, func: { ~c = ~a * 2 })
.addTarget(\D, depends: \C, func: { ~d = ~c + 2 })
.addTarget(\E, depends: [\D, \B], func: {
	~e = ~d + ~b;
});

~p.run(\E);



// returning ...
** Launching target - Boot Server **
Booting server 'localhost' on address 127.0.0.1:57110.
....
** Launching target - A **
** Launching target - C **
** Launching target - D **
** Launching target - B **
** Launching target - E **
|====> ** Finished launching endpoint: E **

Import

So far I don’t think that targets are very useful, but when combined with a nicer way to import files I think it becomes very powerful.

Import is simply a wrapper around thisProcess.interpreter.compileFile(path).().
It is used like this:

at call site

~mainSynths = Import(~dir +/+ "mk_synths_main_group.scd").(
		mainGroup: ~groups[\main],
		mainBus: ~buses[\mainOut],
		revBus: ~buses[\reverb]
)

inside “mk_synths_main_group.scd”

{   |mainGroup, mainBus, revBus|
	(\testerA: Synth(\tester, [\out, mainBus, \sendBus, revBus], mainGroup))
}

What is really nice here is that there are no of the global variables being accessed inside the file, instead everything is passed in to the file and something is returned. This mean you don’t have to think about where the variable was defined if it has been defined already.
This allows all globals to live in the file where the main Project is defined so that their values and lifetimes can easily be seen all at once.

Target Extras

I’ve also added a quick GUI where you can select which target you can run. Theses targets are called endpoints.

~p = Project()
.addPreBootTarget('A', func: {
	~a = [1, 2, 3];
})
.addPreBootTarget('B', func: {|c|
	5.wait;
	~b = 4!10;
	c.signalAll;
})
.addTarget('1', ['A', 'B'], {
	~one = ~a + ~b;
})
.addTarget('2', 'B', {
	~two = 4;
	s.sync;
	~twoPointFive = 6
})
.defineEndPoints('1', '2')
.showGUI()

producing this…


… which will run either '1' or '2' on a click.

Altogether now with a real example

Where I think this structure really shines is when you need multiple configurations of your project, for example, you might need: a performance version (perhaps a large multichannel setup), a home studio monitoring version, and a playground where only the busses and effects have been create. This is all incredibly easy using targets.

(
~dir = "/home/jordan/Desktop";
~numInputs = 2;
~numOutputs = 4;
~micHardwareBus = (\vocal: 0, \guitar: 1);

Import(~dir +/+ "setup_server.scd").(ins: ~numInputs, outs: ~numOutputs);


~project = Project()
.addPreBootTarget('MkCachedData', func: {
	/* something async */
	5.wait;
	~cachedData = 1; // maybe read from a file?
})

.addTarget('DefineSynthDefs', func: {
	Import(~dir +/+ "define_synth_defs.scd").(
		vocalBus:  ~micHardwareBus[\vocal],
		guitarBus: ~micHardwareBus[\guitar],
		numOuts:   ~numOutputs
	)
})

.addTarget('DefineBuses', func: {
	~buses = (
		\mainOut: Bus.audio(s, ~numOutputs),
		\reverb: Bus.audio(s, ~numOutputs.min(2))
	)
})
.addTarget('DefineGroups', func: {
	~groups = ();
	~groups[\main] = Group(s); s.sync;
	~groups[\inputs] = Group.after(~groups[\main]); s.sync;
	~groups[\outputs] = Group.after(~groups[\inputs]);
})

.addTarget('MkMainSynths', ['DefineSynthDefs', 'DefineBuses', 'DefineGroups'], {
	~mainSynths = Import(~dir +/+ "mk_synths_main_group.scd").(
		mainGroup: ~groups[\main],
		mainBus: ~buses[\mainOut],
		revBus: ~buses[\reverb]
	)
})

.addTarget('PerformanceOutput', ['MkMainSynths'], {
	SynthDef(\perfOut, { /* something for performance */ })
	.play(~groups[\outputs])
})

.addTarget('SynthTester', ['DefineBuses', 'DefineGroups'])


.addPreBootTarget('HomeMonitorDef', func: {
	~somethingOnlyNeededAtHome = 1;
})
.addTarget('HomeMonitorOutput', ['MkMainSynths', 'HomeMonitorDef'], {
	SynthDef(\perfOut, { /* something for monitoring away from venue */ })
	.play(~groups[\outputs])
})

.defineEndPoints('HomeMonitorOutput', 'PerformanceOutput', 'SynthTester', 'MkCachedData')
.showGUI()

)
Implementation of the Classes
Import {
	*new {|path| ^thisProcess.interpreter.compileFile(path).()	}
}

PRProjectTarget {
	var <tname, <depends, <function, <isPreBoot;
	*new {|n, d, f, b|
		^super.newCopyArgs(n, d ?? {[]}, f.isKindOf(Function).if({f}, { {f} }), b)
	}

	value {
		postf("** Launching target - % **\n", tname);
		^if (function.def.argNames.size == 1,
			{ this.prValueSync.() },
			{ function.() }
		)
	}

	prValueSync {
		var c = CondVar();
		var ret;
		try { fork{ ret = function.(c)} } { c.signalAll };
		c.wait;
		^ret;
	}
}

Project {
	var targets;
	var endPoints;
	*new { ^super.newCopyArgs(()) }
	addPreBootTarget { |name, depends, func|
		var d = [ depends ?? {[]} ].flat;
		this.prvalidateDepends(d);
		this.prvalidateDepsArePreBoot(name, d);
		targets[name] = PRProjectTarget(name, d, func, true);
		^this;
	}
	addTarget { |name, depends, func|
		var d = [ depends ?? {[]} ].flat;
		this.prvalidateDepends(d);
		targets[name] = PRProjectTarget(name, d, func, false);
		^this;
	}
	run { |name, server|
		this.prvalidateTarget(name);
		this.prRun(name, server ? Server.default)
	}

	defineEndPoints { |...list|
		list.do{|l| this.prvalidateTarget(l) };
		endPoints = list;
		^this;
	}

	mkGUI {
		|parent|
		var title = StaticText()
		.align_(\center)
		.string_("Lauch Piece With Target")
		.font_(Font(Font.defaultSansFace, 64, true));
		var ends = endPoints.collect({|v, k|
			MenuAction(v, { this.run(v) }).font_(Font(Font.defaultSansFace, 24));
		});
		var toolBar = ToolBar(*ends.asArray);
		var v = View(parent).layout_(VLayout(
			[title, s: 1, a: \top],
			[toolBar, s:99, a: \top]
		));
		v.resizeToHint(toolBar.sizeHint);
		^v
	}
	showGUI { ^this.mkGUI().front() }

	prRun { |name, server|
		var launched = Set();
		var preBootList = [];
		var postBootList = [];
		var bootCond = CondVar();

		var addDepsFirst = { |n|
			targets[n].depends.do{|d| addDepsFirst.(d) }; // reccursion
			if (launched.includes(n).not, {
				targets[n].isPreBoot.if(
					{ preBootList  = preBootList  ++ [targets[n]] },
					{ postBootList = postBootList ++ [targets[n]] }
				);
				launched.add(n)
			})
		};

		var actualFunction = {
			preBootList.do( _.value() );

			postf("** Launching target - Boot Server **\n");

			server.waitForBoot {
				postBootList.do { |t| t.(); server.sync; };
				bootCond.signalAll;
			};
		};

		// build dependants tree and try to evaluate them in order.
		addDepsFirst.(name);

		fork {
			try { actualFunction.()	} { bootCond.signalAll };
			bootCond.wait;
			postf("|====> ** Finished launching endpoint: % **\n", name);
		}
	}

	prvalidateDepends {|d|
		d.do{ |dep|
			targets.includesKey(dep).not.if({format("dependent (%) cannot be found", dep).throw})
		}
	}
	prvalidateTarget {|n| targets[n].isNil.if({"target does not exist".throw}) }
	prvalidateDepsArePreBoot {|name, d|
		d.do{ |dep|
			targets[dep].isPreBoot.not.if(
				{format("the pre boot target (%) cannot have post boot targets - %", name, dep).throw}
			)
		}
	}
}

I thought I’d post this after the previous large post got several comments and seemed quite well received. What do you think of this? Would you use it? Do you have a better alternative for managing large projects - I’d love to know as I think its actually a very difficult problem in supercollider.

6 Likes

I am also very interested in this. Currently I have a large project with one main file and 16(!) additional .scd files. The main .scd doc is a big s.waitForBoot function which first calls the .scd doc holding all my synthdefs, then an initialize .scd doc defining all global variables and instantiating the proper synths, then a bunch of other documents organized by ‘topics’ which can be loaded in any order ending with the .scd doc holding the GUI. I make sure that all global variables are defined in the initialize doc so I can always see the rather long list of global variables in one place.

Being able to set different operation modes as you describe would be very handy. Where does the .addTarget method come from, is it from an extension or a quark? One issue with dividing code into multiple files is searching. I am using the SC IDE, I assume other code editors might be able to search across multiple files in a project. Which editor are you using? In relation to this I am also interested in ways of decoupling the GUI from the rest of the code. I tried reading up on MVC, dependants and SimpleController, but I have a hard time finding resources explaining it in detail and wrapping my head around the topic in general. Do you have a strategy for this?

There’s a little check box at the end of the post with the implementation in.

With this approach, since all the files are important in one, it’s usually quite simple to figure out what file something comes from. It gets harder with nested files. I’ve used vim in the past, but use the scide more and more now.

Regarding splitting gui and state, this is done with closures here, but it’s hard to give a general answer.

I’ll turn this into a quark if people like it. Looking for feedback at the moment.

This looks interesting. Right now every project of mine is organised with a main.scd file in the root of the project which loads all the utilities, SynthDefs, Patterns etc. in a s.waitForBoot function from its subfolders.
Something likes this:

(
//1. server config
s = Server.local;

//Windows-only
if(thisProcess.platform.name === \windows){
	s.options.outDevice_("ASIO : JackRouter");
	s.options.inDevice_("ASIO : JackRouter");
};

s.options.numOutputBusChannels = 16;
s.options.numInputBusChannels = 2;
s.options.numPrivateAudioBusChannels = 1024;
s.options.numWireBufs = 1024;
s.options.memSize = 2.pow(20);
s.options.sampleRate = 44100;

//2. initialize global variables
~seed = Date.seed;
~path = PathName(thisProcess.nowExecutingPath).parentPath ++ "buffers";

//3. define piece-specific functions
~makeBuffers = {
	~samples = Dictionary.new;
	PathName(~path).entries.do{ |subfolder|
		var soundfiles = Array.fill(subfolder.entries.size,{ |i|
			Buffer.read(s, subfolder.entries[i].fullPath);
		});
		~samples.add(subfolder.folderName.asSymbol -> soundfiles);
	};
};

~makeBusses = {
	~bus = Dictionary.new;
	~bus.add(\fx -> Array.fill(3, { Bus.audio(s, 2) }));
	~bus.add(\ctrl -> Array.fill(3, { Bus.control(s, 1) }));
};

~makeNodes = {
	s.bind({

		if(~mainGrp.isNil) {
			~mainGrp = Group.basicNew(s, -1);
		};

		if(~fxGrp.isNil) {
			~fxGrp = Group.basicNew(s, -1);
		};

		if(~ctrlGrp.isNil) {
			~ctrlGrp = Group.basicNew(s, -1);
		};

		~mainGrp.nodeID = s.nodeAllocator.alloc;
		~fxGrp.nodeID = s.nodeAllocator.alloc;
		~ctrlGrp.nodeID = s.nodeAllocator.alloc;

		s.sendBundle(nil, ~mainGrp.newMsg(nil, \addToHead));
		s.sendBundle(nil, ~fxGrp.newMsg(~mainGrp, \addAfter));
		s.sendBundle(nil, ~ctrlGrp.newMsg(~fxGrp, \addAfter));
	});
};

~cleanup = {
	s.newBusAllocators;
	ServerBoot.removeAll;
	ServerTree.removeAll;
	ServerQuit.removeAll;
};

//4. register functions with ServerBoot/Quit/Tree
ServerBoot.add(~makeBuffers);
ServerBoot.add(~makeBusses);
ServerQuit.add(~cleanup);

//5. boot server
s.waitForBoot({

	s.sync;

	"utils.scd".loadRelative;

    "%/Synthdefs/*.scd".format(thisProcess.nowExecutingPath.dirname).loadPaths;
	"%/Fx/*.scd".format(thisProcess.nowExecutingPath.dirname).loadPaths;
	"%/Patterns/*.scd".format(thisProcess.nowExecutingPath.dirname).loadPaths;

	s.sync;

	//6b. register remaining functions
	ServerTree.add(~makeNodes);

	s.freeAll;

	s.sync;

	t = TempoClock.new(60/60).permanent_(true);

	"done".postln;
});
)

and then i have a composition.scd where i work on a specific piece.

I’m curious, why not simply classes and methods?

It used to be that re-compiling the class library was quite slow, but now it’s rather fast, a fraction of a second or so.

Of course it’d be nicer still if the system were modified to be more dynamic, but that seems unlikely to happen.

Class variables of dictionaries can recover something of this, ie.

P { classvar <q; *initClass { q = (r: 's'); } } &etc.

Ps. “more dynamic” as in classes and methods can be modified while the system is running.

I’m not “back” permanently – just that this is a topic I’ve taken a lot of time on, and my views on it are different from what’s been discussed so far.

The gist of this thread seems to be methods for organizing global variables.

My approach is to reduce global variables, and instead, as much as possible, encapsulate into self-contained “musical process” objects.

Let’s say you want to use a pattern to play from a sample file, onto a bus.

The usual way that we model this in SC is to load the Buffer into a global variable, and allocate a Bus object into a global variable, and then reference these in the pattern.

Around 2004-2005, I noticed (after wasting tons of time preparing a piece with an awkwardly interconnected object design) that the compositionally significant level of organization is not the pattern (necessitating outside management). The pattern with its resources represents a sonic layer or musical process.

What if I could instantiate the process as one object, and get an object that is fully responsible for all of its own dependencies (and only its own dependencies)?

The other problem at that time was that it wasted a lot of time to have to recompile the class library, reboot the server, reload everything just for a minor change. So I wanted to be able to make changes to a specific process definition and reload it on the fly, and replace only the affected process objects. This ruled out classes – rdd propoosed otherwise, but I still feel very strongly that the class library should be for common elements only; definitions specific to a piece or performance should be “soft.”

Step 1, then, was the ddwProto quark. Prototype-based programming can be used in a way very similar to object-oriented modeling. You get encapsulation, polymorphism, inheritance, but not hardcoded into the class library.

Step 2 was global storage. PR(\name) is a process prototype, analogous to a class. At that time I was interested in abusing the ChucK operator => for double dispatch, so, to instantiate a process, you “chuck” it into a Bound-Process BP(\name) (which “binds” the prototype to specific data – the nomenclature… well… lacks poetry but eh, there it is – btw there is nothing inherently necessary about “chucking” – it wouldn’t be hard to define alternate methods if you don’t like that syntax). The prototype defines constructor and destructor methods to prepare everything that’s needed for the pattern (which is defined in an asPattern method). Like Pdef, you play/stop etc the BP() directly.

In my performance setup, there are a few MixerChannels created globally, in the topEnvironment – but only these. No extra buses, no buffers, just the minimal mixing framework (hwOut, master, couple of reverb FX channels). Everything else belongs to a BP. So the surface area of what I have to manage by hand is extremely small. After that, if I need a hi-hat, I just instantiate the process and the BP goes and gets the sample files for me, no fuss, no globals, no namespace collision, and I get automatic cleanup if I do BP(\hh).free. Instead of a complex process of loading a large number of resources up front, I load resources incrementally, and each process’s own initialization is relatively simple.

Targets in the OP are interesting, but if I wanted to have different studio arrangements, I could create a process prototype for the global mixers, with different configurations. (There’s no rule that a PR/BP must be a playing entity.) Or, if I wanted a global buffer pool, I’d write a PR for that too – because it’s (pseudo-)object modeling, you can do anything.

It occurs to me that this is approaching the problem from the opposite direction – rather than a top-down organization of resources, this is bottom-up – letting individual components manage themselves.

This design has not needed significant revision since 2005 (17 years), so at this point, I’m confident that it’s a successful approach.

BTW all of this is discussed in my SC Book chapter.

sigh back to job job…

hjh

4 Likes