That’s because you haven’t provided a function - try this:
if (~x.isNil or: { ~x.a != "xx" }) { "No XX".postln } { "XX".postln };
or in your syntax:
~x.isNil.or({~x[\a]!="xx"}).if({"No XX"},{"XX"});
although I find the above version to be much clearer (it’s pretty much the standard idiom for writing conditional checks and appears throughout the standard library). Note the curly brackets/braces. Sclang also has special conditional execution operators (?
, ??
and !?
) for simple nil checks.