compileFile or writeDefFile only includes last synthdef

Questions
1

I wrote this compile.scd code to create binary files that I want to load using d_loadDir. But I’ve determined by looking at the binary results that it only includes the last synthdef. Do I have to create the .scd I’m compiling differently?

Here’s how I build the binaries:

"Building ".post;
thisProcess.argv.postln;

if(thisProcess.argv.size == 0, { "No file provided.".postln; 1.exit; },
{
    try {
        r = this.compileFile(thisProcess.argv[0]);

        if(r.value == nil, 
            { 1.exit; }, 
            { r.value.writeDefFile("bin"); 0.exit; });

    } {
        1.exit;
    }
});

And then here’s a sample of which the result is a binary that only includes the last synth def, which I created so the resulting name would make sense, rather than using the last synthdef, which I suppose now I understand is why.

SynthDef(\mic, {
    arg inBus = 0, level = 1.0, wetBus = 0, dryBus = 33333, mix = 1.0;
    var micIn = SoundIn.ar(0);
    // this will be our main output
    Out.ar(wetBus, micIn * mix * level);
    // this will be our effects output
    Out.ar(dryBus, micIn * (1 - mix) * level);
});

SynthDef(\speaker, {
    arg inBus = 0;
    Out.ar([0,1], In.ar(inBus,1));
});

SynthDef(\fader, {
    arg inBus = 0, level = 1.0, wetBus = 0;
    // Ignoring outBus for now, but could be different outputs
    Out.ar(wetBus, In.ar(inBus, 1) * level); 
});

SynthDef('gl', {arg v;
    s.options.pgaGainLeft = v;
});

SynthDef('dl', {arg v;
    s.options.dacLevel = v;
});

SynthDef('test', {
    arg level = 0.5, dryBus = 0;

    Out.ar(dryBus,  SinOsc.ar(1000) * level );
});

SynthDef('l1', {
    arg inBus = 0;
    // SendTrig.kr(Impulse.kr(5), 11, Amplitude.ar(SoundIn.ar(0), 0.01, 0.55));
    SendPeakRMS.ar(SoundIn.ar(0), 20, 1, '/tr', 11);
});

SynthDef('basics', {});

Why does the resulting binary only contain basics rather than all the synthdefs?

2

compileFile gives you a function, exactly like one defined in { braces }, following all the same rules.

What would you expect the result of this function to be?

f = { |a = 0|
	a + 1;
	a + 2;
	a + 3;
};

In most programming languages, a function is a series of expressions (or instructions to perform), ending with a single return value. That is, functions can do a lot of work, and then return something simple.

Functions do not automatically accumulate all the results of every expression, and give them all back to you. (A function in SC is not like [expr] in Pure Data, where you get a different outlet for every expression.) Normally we don’t want functions to yield back every intermediate result.

If you want a single result value that contains multiple results, then you have to make a collection of them by yourself – such as, an Array:

[
	// note, COMMA, not semicolon
	SynthDef(...),
	
	SynthDef(...),
	
	SynthDef(...),
	
	SynthDef(...),
	
	SynthDef(...)
]

BTW Array should work with writeDefFile – this feature is seldom used but should be supported.

hjh

3

I tried changing my source file to the following, but it didn’t create a scsyndef file at all.

[
SynthDef(\mic, {
    arg inBus = 0, level = 1.0, wetBus = 0, dryBus = 33333, mix = 1.0;
    var micIn = SoundIn.ar(0);
    // this will be our main output
    Out.ar(wetBus, micIn * mix * level);
    // this will be our effects output
    Out.ar(dryBus, micIn * (1 - mix) * level);
}),

SynthDef(\speaker, {
    arg inBus = 0;
    Out.ar([0,1], In.ar(inBus,1));
}),

SynthDef(\fader, {
    arg inBus = 0, level = 1.0, wetBus = 0;
    // Ignoring outBus for now, but could be different outputs
    Out.ar(wetBus, In.ar(inBus, 1) * level); 
}),

SynthDef('gl', {arg v;
    s.options.pgaGainLeft = v;
}),

SynthDef('dl', {arg v;
    s.options.dacLevel = v;
}),

SynthDef('test', {
    arg level = 0.5, dryBus = 0;

    Out.ar(dryBus,  SinOsc.ar(1000) * level );
}),

SynthDef('l1', {
    arg inBus = 0;
    // SendTrig.kr(Impulse.kr(5), 11, Amplitude.ar(SoundIn.ar(0), 0.01, 0.55));
    SendPeakRMS.ar(SoundIn.ar(0), 20, 1, '/tr', 11);
}),

SynthDef('basics', {})
]

Regarding your comment on what I think your example function results in, I’m sure I’m not thinking about it right. I’m primarily a C++/C# programmer, so at first glance I think each SynthDef is a function since they are at a top level and each one is separate from each other. But instead, perhaps these are multiple statements with an invisible “main” function? Perhaps I need to relearn LISP or Smalltalk or another functional language to help to form a better basic understanding.

I’ve also noticed that sometimes I’ll forget .ar or .kr for ugens, and it doesn’t complain. I guess it has some meaning, but it doesn’t sink in what that would be useful for. As an example sin(…) vs sin.ar(…). Really just another lack of knowledge on my part.

Regarding the compileFile, I do use that in my build script which is the first code I showed above. I’m not following, why did you mention it?

Thanks for helping me learn,
Paul

4

Yes, this is what I meant by “compileFile gives you a function, exactly like one defined in { braces }, following all the same rules.”

{ a + 1; a + 2 }.def.dumpByteCodes;

"a + 1; a + 2".compile.def.dumpByteCodes;

These two print the same.

You needed it in order to detect syntax errors – I mentioned it because I remember from the other thread that you’re using it.

Your code design isn’t clear to me. Basically there are two ways to make this work:

  • One way is, your .scd file creates an array of SynthDef objects and returns it back to the caller – and the caller does writeDefFile.
  • The other way is, your .scd file creates an array of SynthDef objects and does writeDefFile, and then the caller proceeds on the assumption that the file exists.

Here is a working example, writing a scsyndef file containing two SynthDefs. The tests below the array block demonstrate that both the language (SynthDescLib) and server (d_load) are correctly retrieving both defs from the one file.

(
[
	SynthDef(\test1, { |out = 0, freq = 440, gate = 1, amp = 0.1|
		var eg = EnvGate(1, gate);
		Out.ar(out, (SinOsc.ar(freq) * (eg * amp)).dup);
	}),
	
	SynthDef(\test2, { |out = 0, freq = 440, gate = 1, amp = 0.1|
		var eg = EnvGate(1, gate);
		Out.ar(out, (LFTri.ar(freq) * (eg * amp)).dup);
	})
].writeDefFile("test", "~".standardizePath);
)

// run these one by one and watch the post window

SynthDescLib.at(\test1);  // nothing, so... not 'add'-ed, this is correct

SynthDescLib.read("~/test.scsyndef".standardizePath);

SynthDescLib.at(\test1)  // got it! so test1 is in the file

SynthDescLib.at(\test2)  // got it! so test2 is in the file

s.sendMsg(\d_load, "~/test.scsyndef".standardizePath);

(instrument: \test1).play;

(instrument: \test2).play;  // both work

These are not valid DSP operations – you should delete these. They are not going to do what you think they’re doing. (It’s possible that the file isn’t being written if there are invalid defs in the array – not sure, but, best to remove readily visible points of failure.)

hjh

5

Oh… and now I understand the confusion a bit better.

SuperCollider doesn’t have C-style functions at all.

In C, you write a function block and then it just exists everywhere below that declaration (plus “include” etc.). That’s what you assumed about SynthDefs.

In SC, there are no function or SynthDef declarations. There are expressions which return functions or SynthDefs. There is no distinction between a data object and a Function object – everything is an object, and everything is created and manipulated by expressions, including functions.

When you do SynthDef(...).add, you have first an expression that creates a SynthDef object, and then the message .add is sent to the SynthDef object. This .add instruction causes the SynthDef to be sent to the server, and stored in the global SynthDescLib collection. At this point, it seems to “just exist everywhere” like a C function, but the mechanism is completely different from C. It has to be explicitly stored.

If you write a SynthDef expression and then discard the result (by not adding it, or assigning it to a variable, or putting it in a collection e.g. Array), then it’s lost forever.

Even this –

~times2 = { |x| x * 2 };

This does not declare a function named times2. Functions don’t have names. A function has been stored in an environment variable. “Calling the function” is to retrieve the function object from the variable, and send the .value message to it.

So your initial version created n SynthDefs, and discarded n-1 of them.

hjh

6

It works for me, but then how do I make this a file that I can integrate back into my compile.scd where I call CompileFile. The idea there was to catch syntax errors. Is there a way to make it so it returns 1 if it fails to compile, and 0 if it succeeds?

Also, regarding the parens around the square brackets, what is that doing? Creating a function? Why are they required?

7

This is one of those cases where the basic idea is not that complicated, but the plumbing to interact with the world outside of SC is less obvious. So, let me just show how I would do it.

If you’re calling:

sclang script.scd basename synthdefs dir scsyndefFileLocation script writeSynthdefs.scd

(… and assuming both scd files are in the same directory…)

script.scd

This looks a bit long, but errors could happen in multiple places: processing command line args, reading the script file, compiling the def script, running the def script, and it should also handle unknown errors. The actual detection of syntax errors is easy. Making sure the script doesn’t get stuck takes some care – the long protect block is for this reason – the script must exit, no matter what error occurred.

var cmdLineArgs = (
    uncategorized: List.new,
    script: "writeSynthdefs.scd",
    dir: thisProcess.nowExecutingPath.dirname,
    basename: "synthdefs"
);

var i = 0;
var argv = thisProcess.argv;

var errFileNotFound = -1;
var errSyntaxError = -2;
var errEvaluationError = -3;
var errCmdLineTypo = -4;
var errOMG = -128;

var func;
var file, string;

protect {
    try {
        while {
            i < argv.size
        } {
            case
            { argv[i] == "dir" } {
                cmdLineArgs[\dir] = argv[i+1];
                i = i + 1;
            }
            { argv[i] == "basename" } {
                cmdLineArgs[\basename] = argv[i+1];
                i = i + 1;
            }
            { argv[i] == "script" } {
                cmdLineArgs[\script] = argv[i+1];
                i = i + 1;
            }
            { cmdLineArgs[\uncategorized].add(argv[i]) };
            i = i + 1;
        };
    } {
        Exception(errCmdLineTypo).throw;
    };

    protect {
        file = File(thisProcess.nowExecutingPath.dirname +/+ cmdLineArgs[\script], "r");
        string = file.readAllString;
    } { |error|
        file.close;
        if(error.notNil) {
            Exception(errFileNotFound).throw;
        }
    };

    func = thisProcess.interpreter.compile(string);

    if(func.isNil) {
        Exception(errSyntaxError).throw;
    };

    try {
        func.value(cmdLineArgs[\basename], cmdLineArgs[\dir]);
    } { |error|
        Exception(errEvaluationError).throw;
    };
} { |error|
    case
    { error.isMemberOf(Exception) } {
        error.what.exit;
    }
    { error.isKindOf(Error) } {
        "Something extra bad happened:".postln;
        error.reportError;
        errOMG.exit;
    }
    { 0.exit };
};

writeSynthdefs.scd

arg basename, dir;

[
    SynthDef(\test1, { |out = 0, freq = 440, gate = 1, amp = 0.1|
        var eg = EnvGate(1, gate);
        Out.ar(out, (SinOsc.ar(freq) * (eg * amp)).dup);
    }),
    
    SynthDef(\test2, { |out = 0, freq = 440, gate = 1, amp = 0.1|
        var eg = EnvGate(1, gate);
        Out.ar(out, (LFTri.ar(freq) * (eg * amp)).dup);
    })
].writeDefFile(basename, dir);

You could also omit writeDefFile and the arguments from the second script – let this script return an array, and the main script performs writeDefFile on the result of func.value.

hjh

8

@jamshark70 thanks so much for this detailed example. It will take me a bit to absorb. I feel I’m missing basics about the language structure and syntax. For instance, I see your var cmdLineArgs and it looks like a struct? Sometimes I see parens around things, but I have no idea why. Is there tutorial or documentation I can read that could help me learn about the basics of sclang syntax and structure.

Thanks again,
Paul

9

I just found this: “A Gentle Introduction to SuperCollider (2nd edition)” by Bruno Ruviaro (bepress.com). Might be what I want.

10

Hi, to answer your specific questions about parentheses:

cmdLineArgs here is an Event – Event | SuperCollider 3.12.2 Help
(and yes in this example it is just being used as a dictionary of key/value pairs although Events have lots of other functionality built in, since the syntax is so much nicer than using Dictionary [Event is a subclass of Dictionary] often people use Events just as a data structure)

And the parentheses around

(
[
...
].writeDefFile(...)
)

are for interactive interpretation, if you paste James’s whole example into the IDE you can put the cursor anywhere between those parentheses and hit cmd-enter on Mac or ctrl-enter on PC to just evaluate the code contained within. This is very idiomatic, most code shared on this forum is written assuming it will be interactively evaluated in this way, and parentheses indicate that the entire block should be evaluated together.

Also see this help file for more syntax stuff: Syntax Shortcuts | SuperCollider 3.12.2 Help

11

Also see:
https://doc.sccode.org/Reference/Control-Structures.html

(And as described in syntax shortcuts help file >
moving blocks out of argument lists (trailing-block arguments)
instead of writing:

if (x < 3, { \abc }, { \def })

you can write:

if (x < 3) { \abc } { \def }

)

Also see J concepts in SC | SuperCollider 3.12.2 Help for some of the more arcane array related syntax

12

OK, very close, thanks @jamshark70. But one issue. Seems I’m getting extra spaces and I don’t see a trim function for strings. Why does this add a space before and after?

var curdir = File.getcwd;

var buildfile = PathName(curdir +/+ "filenospace.scd");

("|" + buildfile.fileNameWithoutExtension + "|").postln;

0.exit;

Output for me is:

| filenospace |

BTW, thanks @Eric_Sluyter for the links. I read them all. Also, the link I had cleared up my questions about enclosures but interestingly enough didn’t mention the Event. Hard to find things when you don’t know the names :slight_smile:

13

Ah ++ is no space, but + adds a space. sclang is so wierd :slight_smile:

Can I get rid of all the sclang output at the start?

compiling class library...
        Found 726 primitives.
        Compiling directory '/usr/share/SuperCollider/SCClassLibrary'
        Compiling directory '/usr/share/SuperCollider/Extensions'
        Compiling directory '/root/.local/share/SuperCollider/Extensions'
        Compiling directory '/root/.local/share/SuperCollider/downloaded-quarks'
        numentries = 1110845 / 12508526 = 0.089
        4223 method selectors, 2962 classes
        method table size 8235620 bytes, big table size 50034104
        Number of Symbols 11772
        Byte Code Size 331293
        compiled 455 files in 2.45 seconds

Info: 4 methods are currently overwritten by extensions. To see which, execute:
MethodOverride.printAll
14

++ for concatenation is at least internally consistent:

[1, 2, 3] ++ [6, 5, 4]
-> [1, 2, 3, 6, 5, 4]

"abc" ++ "def"
-> abcdef

If you try to backward-extrapolate string concatenation from other languages (+) to arrays in SC, then:

[1, 2, 3] + [6, 5, 4]
-> [7, 7, 7]

… and SC (if I recall, maybe I missed something) doesn’t support char addition, so "abc" + "def" wouldn’t be analogous. So I guess one question might be, would it be weirder to use + to concatenate only strings and ++ to concatenate all other sequenceable collections?

So then I suppose JMc felt free to use + as a convenience operator for those times when you do want spaces… debatable I guess (?), but it’s not willfully bizarre.

Not to my knowledge. A verbosity command line switch for sclang would be a nice addition.

hjh

15

I just meant in all the other languages I know, + means add, and for strings that means just concatenating them. And ++ means increment. I’m guessing things like this trip up people coming from other languages.

I’m close to getting this to work. I decided to load the file as a string and then manually tack on the ".writeDefFile(…) afterwards. I think it’s weird need to tell a source file to compile itself and passing it parameters to use, also seems unnatural. This works fine for when I have an array of synthdef. But what if I have other code as in this example?

var digitalPins = [ 0, 1, 2, 3, 4, 5, 6, 7];
var triggerCount = 1;

[
    SynthDef(\trigDetector, {
...    }),

    SynthDef("sh",{ 
...    })

]

~previousSampler = nil; 

~sendFootTrigger = { | k, v |
     ~previousSampler !? {|p| p.set(\gate, 0) };   // remove old if exists
     ~previousSampler = Synth(\playSample, [\buf: k, \vol: v]); // fill with your args
 }

o = OSCFunc({ 
     arg msg, time;
     if(msg[2] == -1 && msg[3] == 40, { ~sendFootTrigger.value(msg[5], msg[4]) });
 },'/trg');

It fails at the bottom but works if I move it to the top. However, the resulting binary file is the same size if that code is there or not. I feel it’s not including this in the compiled binary. I think I need to create a single object that contains all the code, but not sure how I would do that. I tried ( ) and { }. Any ideas how to make this work?

Here’s my compile code. I removed all the try excepts and made it a little simpler, but perhaps they are needed for certain scenario. Perhaps you have some advice about that.

// Build a SuperCollider file

var buildfile; // File path to build
var filename; // File object for scd
var file; // File object for scd
var string; // Contents for file
var func; // Compiled function
var curdir; // Current directory

thisProcess.argv.size.postln;

if(thisProcess.argv.size < 1) {
    "Usage: comp file.scd".postln;
    1.exit;
}
{
    curdir = File.getcwd;

    buildfile = PathName(curdir +/+ thisProcess.argv[0]);
    // buildfile.postln;

    filename = thisProcess.argv[0];

    // ("Loading: " + buildfile.fullPath).postln;

    protect {
        file = File(buildfile.fullPath, "r");
        string = file.readAllString + ".writeDefFile(\"" ++ buildfile.fileNameWithoutExtension ++ "\",\"bin\");";
    } { |error|
        file.close;
        if(error.notNil) {
            ("Load File Error is " + error).postln;
            1.exit;
        };
    };

    // ("Building: " + buildfile).postln;

    func = thisProcess.interpreter.compile(string);

    if(func.isNil) {
        "Function is NILL".postln;
        1.exit;
    };

    try {
        func.value();
    } { |error|
        if(error.notNil) {
            ("Compile Error is " + error).postln;
            1.exit;
        };
    };

    "Done".postln;

    0.exit;
};
16

Sure, that’s fine.

There seems still to be some confusion about functions and their return values.

For convenience, let me label the scripts “outer” and “inner.”

If you’re going to obtain the array of SynthDefs from the inner script, and perform writeDefFile in the outer script, then the inner script must return the array to the caller. (Note also that the only way sclang can execute the code in either script is to compile it into a normal Function object – the inner script is a function – a “script” in a file does not offer any way around Function behavior.)

In SC, a function’s return value is always the last expression in the function.

So when you write:

var ...;

[
	SynthDef(...),
	SynthDef(...)
]

~previousSampler = nil;

… there are two problems:

  1. The semicolon after the [ ... ] expression is missing, so, syntax error.
  2. The desired function return value (the array) gets dropped early, and never returned. So the outer script does not have access to the array, and cannot writeDefFile it.
f = { |a = 0|
	a + 1;  // this expression is performed, then dropped
	a + 2;  // this expression is performed, then dropped
	a + 3;  // this expression's result is returned
};

If you reorder those statements, then the function’s return value is different.

So, if you need the SynthDef array to be returned, then you don’t have freedom to write the operations in any order. The array must come last.

You could do:

var myDefArray = [
...
];

... other stuff...

// last thing:
myDefArray

There is never any compiled binary file of sclang code. Sclang code is always compiled a/ at startup (class library) or b/ at load-time (runtime { } function definition) and retained in memory.

writeDefFile produces a scsyndef file, containing one or more SynthDefs. A SynthDef is a signal processing graph to be evaluated in the server, and only in the server. The SynthDef binary format cannot contain any non-server sclang operations. OSCFunc is a language-side object only. The server has nothing to do with OSCFuncs; therefore there is no way to write an OSCFunc definition into a scsyndef file.

Where I’m unclear is… I thought you were using the sclang script to produce the scsyndef file, and your C-something app would boot the server, load the scsyndef file, and all of the server control would be done in your app. If that’s the case, then there’s no point in writing any OSCFuncs into your build script, because sclang wouldn’t be running when you are using your app. The OSCFunc functionality should be written in Cxxxxx using some OSC library.

If, instead, you will run your app, and your app launches sclang, and sends OSC messages to tell sclang to do things on the server, then there is no need for a build-time sclang script at all. You can init the SynthDefs at sclang startup time – fraction of a second. (But elsewhere, IIRC, you’ve spoken of using sclang in a cmake script, so I think this isn’t what you’re doing…?)

Either way eliminates some complication. Trying to do both is almost certainly overcomplicating matters.

hjh

17

Success, at least to level that it’s working as it was before with the new way to load synthdefs.

I suppose I was thinking more with my c++ brain in terms of “compiling”. You are right, I run sclang on a script that set ups options and then communicate via OSC from my c++ code. So, to resolve things I just moved most of this stuff over there. At the time I was thinking since these are references, it has to be able to resovle them locally, but obviously not for the OSCFunc, and then I guess as well for the ~ environment vars, they can reference them later as the code is compiled at startup.

Now I’m able to compile the scsyndef files and load them. The loading waits, afaict, till all scsyndef files are loaded, so no more issues with wondering what new synth call will return. Also, all of these changes enable the sample playback option of using a trigger that creates a new synth for each sample. Next step is to move off my test examples and integrate into the main app and do real world testing.

One thing I will leave here for the web crawlers and AI bots is that I’ve run into the issue where compiling uses the default location of /root/.local/share/SuperCollider/synthdefs when I guess you forget to specify. This means I was getting duplicate definitions. They showed as /d_remove in the network trace, which I couldn’t find documented btw, and while this didn’t cause a problem in the end, it did cause confusion.

Thanks again, sure I’ll be back with some future related problem.

18

Indeed, I can find /d_removed in the scsynth source code, but then two issues: 1/ it’s not anywhere in the help, and 2/ it isn’t in supernova either (probably because supernova’s interface was based on the documentation, which doesn’t include this reply). That would be a good bug to log.

Sure, nice.

Maybe not necessary to do it right now, but at some point, you could probably simplify the distribution of your app by removing the runtime sclang dependency. Most of the server messages are probably /s_new, /n_set, /c_set, /n_free – these are simple messages, should be no problem to generate them in Cxx. (OSC libraries are likely to be more verbose than sclang’s array style, but basically boilerplate.) Synth(\name) is convenient but it isn’t necessary for your app to go through sclang just for s_new.

Sclang’s SynthDef builder would be impractical to replicate (some alternate language clients have done so! but it’s a lot of work), but if you’re building a set of stable SynthDefs, you could ship the scsyndef file(s) with the app. (If, on the other hand, SynthDefs are not the same from run to run, then you would need the sclang SynthDef builder.)

hjh