How do you manage large projects across multiple files? I have seen many people struggle with this, including myself, so I have made a solution and would really like to know what people think!
What is the problem
I think most people’s solution to managing large project is to break it down into multiple files storing everything in global variables and then load in the files. I have found in my own work that is becomes very cumbersome and hard to manage as a project grows.
I think there are two main problems: first how do you keep track of which order the files are evaluated in, and their dependencies; and second, how to do manage changing global variables. My proposal for each is, project targets and imports/files-as-functions.
Targets
Inspired by programs like CMake where you define multiple targets and link their dependencies together I have this basic structure.
~project = Project()
.addTarget('A', func: {
~a = 1;
})
.addTarget('B', depends: 'A', func: {
~b = ~a * 2;
});
~project.run(\B);
First a project with a list of targets is defined, these are stages that depend on each other. Above, 'B'
requires 'A'
to be evaluated before it, 'B'
depends on 'A'
. Internally these are all wrapped inside of s.waitForBoot
and s.sync
is called between each target. It is also possible to do preboot targets. These cannot depend on ‘postboot’ targets, but can depend on each other.
~p = Project()
.addPreBootTarget('A', func: {
~a = [1, 2, 3];
})
.addPreBootTarget('B', func: {|condvar|
5.wait;
~b = 4!10;
condvar.signalAll;
})
All targets are run inside a forked function, and if the func
parameter has an argument, a CondVar
is automatically created that will be ‘waited’ upon.
Here is a slightly larger example showing multiple dependency paths.
~p = Project()
.addTarget(\B, func: {
2.wait;
~b = 4;
})
.addTarget(\A, func: { ~a = 1 })
.addTarget(\C, depends: \A, func: { ~c = ~a * 2 })
.addTarget(\D, depends: \C, func: { ~d = ~c + 2 })
.addTarget(\E, depends: [\D, \B], func: {
~e = ~d + ~b;
});
~p.run(\E);
// returning ...
** Launching target - Boot Server **
Booting server 'localhost' on address 127.0.0.1:57110.
....
** Launching target - A **
** Launching target - C **
** Launching target - D **
** Launching target - B **
** Launching target - E **
|====> ** Finished launching endpoint: E **
Import
So far I don’t think that targets are very useful, but when combined with a nicer way to import files I think it becomes very powerful.
Import is simply a wrapper around thisProcess.interpreter.compileFile(path).()
.
It is used like this:
at call site
~mainSynths = Import(~dir +/+ "mk_synths_main_group.scd").(
mainGroup: ~groups[\main],
mainBus: ~buses[\mainOut],
revBus: ~buses[\reverb]
)
inside “mk_synths_main_group.scd”
{ |mainGroup, mainBus, revBus|
(\testerA: Synth(\tester, [\out, mainBus, \sendBus, revBus], mainGroup))
}
What is really nice here is that there are no of the global variables being accessed inside the file, instead everything is passed in to the file and something is returned. This mean you don’t have to think about where the variable was defined if it has been defined already.
This allows all globals to live in the file where the main Project
is defined so that their values and lifetimes can easily be seen all at once.
Target Extras
I’ve also added a quick GUI where you can select which target you can run. Theses targets are called endpoints.
~p = Project()
.addPreBootTarget('A', func: {
~a = [1, 2, 3];
})
.addPreBootTarget('B', func: {|c|
5.wait;
~b = 4!10;
c.signalAll;
})
.addTarget('1', ['A', 'B'], {
~one = ~a + ~b;
})
.addTarget('2', 'B', {
~two = 4;
s.sync;
~twoPointFive = 6
})
.defineEndPoints('1', '2')
.showGUI()
producing this…
… which will
run
either '1'
or '2'
on a click.
Altogether now with a real example
Where I think this structure really shines is when you need multiple configurations of your project, for example, you might need: a performance version (perhaps a large multichannel setup), a home studio monitoring version, and a playground where only the busses and effects have been create. This is all incredibly easy using targets.
(
~dir = "/home/jordan/Desktop";
~numInputs = 2;
~numOutputs = 4;
~micHardwareBus = (\vocal: 0, \guitar: 1);
Import(~dir +/+ "setup_server.scd").(ins: ~numInputs, outs: ~numOutputs);
~project = Project()
.addPreBootTarget('MkCachedData', func: {
/* something async */
5.wait;
~cachedData = 1; // maybe read from a file?
})
.addTarget('DefineSynthDefs', func: {
Import(~dir +/+ "define_synth_defs.scd").(
vocalBus: ~micHardwareBus[\vocal],
guitarBus: ~micHardwareBus[\guitar],
numOuts: ~numOutputs
)
})
.addTarget('DefineBuses', func: {
~buses = (
\mainOut: Bus.audio(s, ~numOutputs),
\reverb: Bus.audio(s, ~numOutputs.min(2))
)
})
.addTarget('DefineGroups', func: {
~groups = ();
~groups[\main] = Group(s); s.sync;
~groups[\inputs] = Group.after(~groups[\main]); s.sync;
~groups[\outputs] = Group.after(~groups[\inputs]);
})
.addTarget('MkMainSynths', ['DefineSynthDefs', 'DefineBuses', 'DefineGroups'], {
~mainSynths = Import(~dir +/+ "mk_synths_main_group.scd").(
mainGroup: ~groups[\main],
mainBus: ~buses[\mainOut],
revBus: ~buses[\reverb]
)
})
.addTarget('PerformanceOutput', ['MkMainSynths'], {
SynthDef(\perfOut, { /* something for performance */ })
.play(~groups[\outputs])
})
.addTarget('SynthTester', ['DefineBuses', 'DefineGroups'])
.addPreBootTarget('HomeMonitorDef', func: {
~somethingOnlyNeededAtHome = 1;
})
.addTarget('HomeMonitorOutput', ['MkMainSynths', 'HomeMonitorDef'], {
SynthDef(\perfOut, { /* something for monitoring away from venue */ })
.play(~groups[\outputs])
})
.defineEndPoints('HomeMonitorOutput', 'PerformanceOutput', 'SynthTester', 'MkCachedData')
.showGUI()
)
Implementation of the Classes
Import {
*new {|path| ^thisProcess.interpreter.compileFile(path).() }
}
PRProjectTarget {
var <tname, <depends, <function, <isPreBoot;
*new {|n, d, f, b|
^super.newCopyArgs(n, d ?? {[]}, f.isKindOf(Function).if({f}, { {f} }), b)
}
value {
postf("** Launching target - % **\n", tname);
^if (function.def.argNames.size == 1,
{ this.prValueSync.() },
{ function.() }
)
}
prValueSync {
var c = CondVar();
var ret;
try { fork{ ret = function.(c)} } { c.signalAll };
c.wait;
^ret;
}
}
Project {
var targets;
var endPoints;
*new { ^super.newCopyArgs(()) }
addPreBootTarget { |name, depends, func|
var d = [ depends ?? {[]} ].flat;
this.prvalidateDepends(d);
this.prvalidateDepsArePreBoot(name, d);
targets[name] = PRProjectTarget(name, d, func, true);
^this;
}
addTarget { |name, depends, func|
var d = [ depends ?? {[]} ].flat;
this.prvalidateDepends(d);
targets[name] = PRProjectTarget(name, d, func, false);
^this;
}
run { |name, server|
this.prvalidateTarget(name);
this.prRun(name, server ? Server.default)
}
defineEndPoints { |...list|
list.do{|l| this.prvalidateTarget(l) };
endPoints = list;
^this;
}
mkGUI {
|parent|
var title = StaticText()
.align_(\center)
.string_("Lauch Piece With Target")
.font_(Font(Font.defaultSansFace, 64, true));
var ends = endPoints.collect({|v, k|
MenuAction(v, { this.run(v) }).font_(Font(Font.defaultSansFace, 24));
});
var toolBar = ToolBar(*ends.asArray);
var v = View(parent).layout_(VLayout(
[title, s: 1, a: \top],
[toolBar, s:99, a: \top]
));
v.resizeToHint(toolBar.sizeHint);
^v
}
showGUI { ^this.mkGUI().front() }
prRun { |name, server|
var launched = Set();
var preBootList = [];
var postBootList = [];
var bootCond = CondVar();
var addDepsFirst = { |n|
targets[n].depends.do{|d| addDepsFirst.(d) }; // reccursion
if (launched.includes(n).not, {
targets[n].isPreBoot.if(
{ preBootList = preBootList ++ [targets[n]] },
{ postBootList = postBootList ++ [targets[n]] }
);
launched.add(n)
})
};
var actualFunction = {
preBootList.do( _.value() );
postf("** Launching target - Boot Server **\n");
server.waitForBoot {
postBootList.do { |t| t.(); server.sync; };
bootCond.signalAll;
};
};
// build dependants tree and try to evaluate them in order.
addDepsFirst.(name);
fork {
try { actualFunction.() } { bootCond.signalAll };
bootCond.wait;
postf("|====> ** Finished launching endpoint: % **\n", name);
}
}
prvalidateDepends {|d|
d.do{ |dep|
targets.includesKey(dep).not.if({format("dependent (%) cannot be found", dep).throw})
}
}
prvalidateTarget {|n| targets[n].isNil.if({"target does not exist".throw}) }
prvalidateDepsArePreBoot {|name, d|
d.do{ |dep|
targets[dep].isPreBoot.not.if(
{format("the pre boot target (%) cannot have post boot targets - %", name, dep).throw}
)
}
}
}
I thought I’d post this after the previous large post got several comments and seemed quite well received. What do you think of this? Would you use it? Do you have a better alternative for managing large projects - I’d love to know as I think its actually a very difficult problem in supercollider.