Copying Dictionary seems to "link" contained array

Questions
25

There’s a sort of educational / documentation / etc. piece of this as well - SC Array’s behave a lot like Python tuple’s - there’s some clunky API’s sure, but there’s a place for this kind of data structure in a language to be sure. But - Python resources (that I’m familiar with…) are never like “you should use Tuple’s as the primary list data structure when you’re writing this casual script to process some CSV files” - this would be ridiculous.

SuperCollider has “sane” mutable arrayed data structures (like List) that would alleviate most of the common gotchas, but (a) they have extra syntax that makes them annoying to use (List[1, 2, 3] vs [1, 2, 3]), and (b) SuperCollider has an extraordinarily pedantic class hierarchy for it’s collection types that makes it very hard for a beginner/intermediate user to know which one does what, and which one you should use. When List is one among 6 or 7 collection classes that show up when you look up a common list methods (and… probably the least common to show up in docs compared with the rest!), how is anyone supposed to know that it’s the type that behaves like … std::vector, Python arrays, Ruby arrays, etc etc.

26

Not really. A Python tuple is immutable. The “weird” thing about sclang’s Array – and something I haven’t seen in any other language so far – is that it is mutable by default, but if you try to enlarge it, it may need to return a new object.

The sclang equivalent to Python tuples would be Array literals (e.g. #[ 1, 2, 3])

SuperCollider has “sane” mutable arrayed data structures (like List) that would alleviate most of the common gotchas, but (a) they have extra syntax that makes them annoying to use (List[1, 2, 3] vs [1, 2, 3]), and (b) SuperCollider has an extraordinarily pedantic class hierarchy for it’s collection types that makes it very hard for a beginner/intermediate user to know which one does what, and which one you should use.

Exactly! List is exactly how Array should have behaved in the first place. I mean, it’s kind of nice that Array is so optimized that it can be efficiently used as the underlying storage type for other container classes, such as Dictionary, IdentityDictionary, Set, IdentitySet, etc., but it is really problematic as the main user-facing array type…

But enough with complaining :slight_smile:

To be fair, the help file of Array starts with the following very prominent note:

NOTE: For Arrays, the add method may or may not return the same Array object. It will add the argument to the receiver if there is space, otherwise it returns a new Array object with the argument added. Thus the proper usage of add with an Array is to always assign the result as follows:
z = z.add(obj);
This allows an efficient use of resources, only growing the array when it needs to. The List class manages the Array internally, and in many cases is more suitable [emphasis by me].

27

Strings are mutable – always have been.

a = String.new(3).add($a).add($b).add($c);

a.put(1, $d);  // mutated!

Literal strings are immutable:

b = "abc";

b.put(1, $d);  // Attempted write to immutable object.

But, for that matter, so are literal arrays:

c = [1, 2, 3, 4];  // non-literal array
c.put(1, 10);  // mutated

d = #[1, 2, 3, 4];  // literal array
d.put(1, 10);  // Attempted write to immutable object.

Of c’s non-literality and mutability:

{ [1, 2, 3, 4] }.def.dumpByteCodes

BYTECODES: (21)
  0   06 19    PushSpecialClass 'Array'
  2   2C 04    PushInt 4
  4   C2 00    SendSpecialMsg 'new'
  6   64       PushSpecialValue 1
  7   C2 08    SendSpecialMsg 'add'
  9   65       PushSpecialValue 2
 10   C2 08    SendSpecialMsg 'add'
 12   2C 03    PushInt 3
 14   C2 08    SendSpecialMsg 'add'
 16   2C 04    PushInt 4
 18   C2 08    SendSpecialMsg 'add'
 20   F2       BlockReturn

[1, 2, 3, 4] means Array.new(4).add(1).add(2).add(3).add(4).

An array constructed by method evaluation is mutable. A string constructed by method evaluation is mutable. "abc" ++ "def" is evaluating the ++ method – therefore, the result is mutable (unless you force immutability by .freeze).

In fact, many years ago, literal strings used to be mutable. But if they are mutable, then you can have disasters like this:

~removeFromString = { |str = "default", removeChar = $a|
	str.removeEvery(removeChar.asString)
};

~removeFromString.value;

Now “a” has been removed from str’s default value – but this default value belongs to a function definition, and any change to it will affect all future calls to the function. That’s terrible. So we changed it a long time ago so that string literals are immutable.

hjh

2 Likes
28

One thing that is easily forgotten, as criticism of sclang escalates in intensity, is that in twenty years, nobody has made any radical improvements. (So, maybe it isn’t that bad.)

At the same time, computer language design has moved forward and sclang hasn’t, which could explain why quirks that had been tolerated 10 years ago are increasingly felt to be grating.

I tend to think there are only three directions: 1/ fix sclang; 2/ replace sclang with a different language; or 3/ accept sclang, warts and all.

1/ requires skills that very few of us in sc-land have. I certainly don’t. For most of us, the compiler and interpreter are a black box. Radically improving sclang would require a small number of developers to invest a large amount of time.

2/ may require less in the way of skills, but the SC class library includes a ton of conveniences for music-making, and it would take a lot of time to reimplement in another language. (Even just SynthDef building… several alternate scsynth clients don’t even try.)

3/ is easier, but would lead to a slow death. (For another example, Pure Data’s GUI is firmly stuck in 1990s Tcl/Tk and one result is that the Pd forum’s traffic level is maybe 10-20% of the SC forum… and they’re just like “well, we don’t mind the GUI that much” and “we don’t dare change it because of long-term users’ muscle memory” but it means the tool is not attracting numbers of new users. Eventually an out-of-date language design would fall too far behind.)

I’ve got an opinion on this (I think SC4 needs to be based on a different language, even though this would just about kill me in terms of reimplementing my tools), but… 52-year-old me has physical problems (rheumatism, eyesight isn’t what it used to be) that 32-year-old me didn’t have. My days of intensive development are over. So for me, #3 (accept it) is the only realistic option. But that need not be for other developers who might have the right motivation.

hjh

1 Like
29

For another example, Pure Data’s GUI is firmly stuck in 1990s Tcl/Tk

That’s a good example. I have always been a vocal critic of Pd’s Tcl/Tk GUI and there is finally some serious momentum to replace the GUI backend - or at least to allow alternative GUIs. See for example toolkit-agnostic Core->GUI Communication (Draft B) by umlaeute · Pull Request #1765 · pure-data/pure-data · GitHub. This one has already been merged: Refactor TCL-specific string handling into pdgui_vmess() by umlaeute · Pull Request #1696 · pure-data/pure-data · GitHub
We’re slowly getting there :slightly_smiling_face:

30

1/ requires skills that very few of us in sc-land have. I certainly don’t. For most of us, the compiler and interpreter are a black box. Radically improving sclang would require a small number of developers to invest a large amount of time.

A less-extreme alternative could be to simplify the path for developers. As mentioned earlier, Lists in sclang aren’t as “weird” as Arrays but few people know about them and it’s a slight inconvenience to write List[1,2,3] instead of [1,2,3].

I can imagine a guide similar to “Javascript: The Good Parts”: a short and opinionated guide to modern ways to write sclang. Easier than fixing Arrays and dealing with backwards compatibility is to just tell people to use Lists if that’s the consensus on what’s generally best!

Of course, syntax improvements could help too (for example, somehow not having to prepend “List” all the time).

2/ may require less in the way of skills, but the SC class library includes a ton of conveniences for music-making, and it would take a lot of time to reimplement in another language. (Even just SynthDef building… several alternate scsynth clients don’t even try.)

SynthDef building is frankly pretty easy, especially if you’re familiar with the format (which SC4 devs would be). I wouldn’t view that as a stumbling block. Musical abstractions that don’t break the brains of existing SC users is much harder, but actually building a client that implements every scsynth/supernova command is overall straightforward even if not always trivial.

3/ is easier, but would lead to a slow death. (For another example, Pure Data’s GUI is firmly stuck in 1990s Tcl/Tk and one result is that the Pd forum’s traffic level is maybe 10-20% of the SC forum… and they’re just like “well, we don’t mind the GUI that much” and “we don’t dare change it because of long-term users’ muscle memory” but it means the tool is not attracting numbers of new users. Eventually an out-of-date language design would fall too far behind.)

This, honestly, is my reason for mainly using a non-sclang language: even if the language itself doesn’t become out of date (just as other forms of musical notation have lasted centuries), its tools will eventually lag behind. Even niche non-musical languages have orders of magnitude more development on debuggers, static analysis tools, libraries to interact with all sorts of interesting domains, etc.

A beautiful thing about SC is that it’s modular enough to pick and choose: some people use standard GUI+sclang+scsynth, but others replace the GUI with emacs/neovim/…, replace sclang with Python/Haskell/Scheme/…, and replace scsynth with supernova. If I stop to imagine SC in the year 2035 I think I’ll see some familiar parts and some that are quite different, and the parts that are different are mostly the ones where we’ve switched to an existing tool that person-centuries have been spent improving, instead of rolling our own.

I’ve got an opinion on this (I think SC4 needs to be based on a different language, even though this would just about kill me in terms of reimplementing my tools), but… 52-year-old me has physical problems (rheumatism, eyesight isn’t what it used to be) that 32-year-old me didn’t have. My days of intensive development are over. So for me, #3 (accept it) is the only realistic option. But that need not be for other developers who might have the right motivation.

hjh

Tom

31

I disagree… Python tuples are immutable. They don’t have a public notion of maximum size or even capacity distinct from size. They don’t support appending in their API.

The two closest commonly encountered data types would be a dynamic array, which is by far the common use case when SC programmers are reaching for a collection type, or a fixed capacity, dynamic size vector (like Boost’s static_vector), which would error on an attempt to grow past its capacity. The latter is a more niche design and not something you’d typically see in a high level scripting language.

I think it’s easily forgotten because it’s an appeal to tradition that doesn’t help the discussion. The many posts about “SC4” and many questions people post about the design issues with Array, indicate that the language is indeed in need of improvement. Other people don’t see this particular case as a quirk, they see it as a pain point, and rightfully so.

Fixing Array would break backward compatibility in a good way: it would make previously broken and incorrect programs behave as their writers intended. A programmer writing array.add(3) is never intending to discard the result of the append in the case that appending results in resizing of the underlying storage. They are expecting that 3 has been added to the array.

I fully agree with this. I think it’s great that people are not locked into any particular combination of tools, and I think it contributes a lot to the project’s longevity.

32

If I’m not going to be understood properly, then I’ll gladly withdraw from this discussion.

hjh

33

@jamshark70 was not trying to say that sclang is just fine. I mean, 1/ is literally called “fix sclang”. Seems like you have only read the first paragraph…

34

Anyway, I think @VIRTUALDOG is right that the implementation of Array could be changed to allow resizing without returning a new array. Array would just need to separate the object itself from the element storage (like a typical dynamical array does). As a consequence, List could just become an alias for Array. One downside is that all containers that use Array internally ( Dictionary, IdentityDictionary, Set, UnorderedSet, etc.) would become a little bit slower, but I assume that it would hardly matter on modern CPUs. Or we can find ways to optimize it.

(Since Array literals are immutable, they might still be layed out consecutively in memory for optimal performance.)

1 Like
35

I think this is a really thoughtful, well balanced comment (the entire thing, not just the quote): could not agree more. Although I’m definitively in number 2.

A newer language that scales better might make this easier than expected (though definitely not trivial).

Regarding adopting an existing language I don’t know if there is any clear leader here, curios to get others opinion. Here is my 2 cents…

  • Python might be the best for accessibility given its wide adoption.

  • Though, I’d quite like to see something like Julia considered for its (excellent llvm) JIT compilation and type system. Julia also has the benefit it can be compiled to wasm (or anything llvm supports I think), perhaps providing a counter to Cycling74’s RNBW, this might be a good long term investment? it also solves some of the performance issues sclang has, although might introduce more latency between asking code to executed and some process starting. Julia also has broadcasting which is their implementation of array programming, which sclang also borrowed from - there might be some familiarity there.

  • Fully on the fence about Haskell, it might be too much of a change.

  • Please no Javascript.

36

:roll_eyes: As you wish, your majesty.

There should be a net performance gain since during array growth the old object no longer needs to be kept around and garbage collected, no storage needs to be allocated for a new object, and no assignment operation needs to be executed. That’s my understanding anyway.

37

:roll_eyes: As you wish, your majesty.

Please refrain from needless ad hominem attacks.

There should be a net performance gain since during array growth the old object no longer needs to be kept around and garbage collected, no storage needs to be allocated for a new object, and no assignment operation needs to be executed. That’s my understanding anyway.

Yes, there would be less pressure on the garbage collector. Also, if you reallocate in place (and the memory location does not change), you don’t have to copy the existing elements.

However, dynamic vectors and hash tables are designed so that reallocation only happens rarely, since the capacity is typically increased geometrically. This means that resizing is rarely the bottleneck.

I was rather thinking of Array access. As I explained before, if the element storage is kept separate from the object itself, any element access requires an extra indirection (which might result in a cache miss). Since Array is used internally by IdentityDictionary (as the underlying array of an open addressing hash table), this would slow down every access of environment variables or event properties! Now, there are certainly ways to optimize this, but it has to be considered from the very start.

38

Since there’s mention of swapping sclang for a different language (Python, Julia, etc.), I’d like to share my 2 cents about sclang as I think it might be useful since I’m new to it (2 months in).
In random order:

  • multiple coding paradigms support at first was “confusing” (for lack of a better word) but, in few weeks it got interiorised. This is now offering different perspectives when “tackling problems”
  • sclang + IDE = a love story! Apart from when I was poking around with Jupiter notebooks (and it’s different anyway), I never saw any other language working so tight with its environment (I suppose this comes from Smalltalk). Again, this was alien to me at the beginning but, I see it now as a strength. Prototyping in SC is insanely good and, I’m finding my way also now that I’m at a point where I’m confident enough to start bigger projects
  • syntactic sugar galore! I wouldn’t understimate this. As with everything, it takes a bit to learn about all the different ways of expressing ourselves in sclang but, that’s just so unique and powerful. Again, never seen so much of it in any other language out there
  • this is more of a personal thought. I understand the appealing of Python and any other “standard” out there but, unless there are technical reasons (eg sclang underperforms, is tricky to maintain, etc.), I wouldn’t change it for anything just because that is the standard.
    I do believe that a language and its syntax (like with natural languages) affects the choices we make, the way we program and, I (personally) find quite sterile the idea of standards (in this kind of context) and the monopoly that langs like Python have (I’d go for Lua if anything!). I think a choice like that would be counterproductive in any environment, especially one like SC that is art-related. Sure, it might take a bit of time to grasp sclang if you are new to it or come from Python, Lua, etc. but, that is kind of true with anything, it’s just that people are more exposed to Python, JS, etc… Also, SC has an incredible documentation!
2 Likes
39

Also, probably all this requires its own thread

40

So, to clarify:

Where I personally am in my life, I’m not in a position to contribute significantly to an SC4. So I try as much as possible to accept the weird stuff.

I think the SC community needs an SC4 and I would be very disappointed if it didn’t happen. Retooling would be painful for me but it would be stupid to hold SC back just for my convenience. It’s simply not correct to read my post as saying “I accept SC and you should too.” It was rather, “I kinda have to accept SC but I hope other people don’t and do what’s needed.”

About complaining: It’s useful only if it results in either fixing or replacing sclang. But I see the amount of complaining going up, while the amount of concrete work towards an SC4 is remaining constant at 0. This isn’t helpful.

About languages: Two features of SC that really grease the wheels for my work are coroutines, and the (near?) complete interchangeability of numbers and arrays of numbers (or other number-like things) in math operations. SC gets math overloading from runtime dispatch (which brings with it debugging problems etc but we depend on this flexibility in ways we maybe don’t notice all the time). A language with awkward support for these would be a poor choice for SC4.

Of disputes: So I’ll be explicit: “in twenty years, nobody has made any radical improvements” does not mean, and was never intended to mean, that no improvements are possible. This should have been clear in the remainder of my post (and it was clear to others). There was no “appeal to tradition,” and what is not helpful to the discussion is to project a biased reading.

What I meant was: It’s easy to talk about the things that are wrong with SC, but it’s hard to do something about it. Someone could do something about it, but it hasn’t happened yet. It’s worth thinking about what it would take for that to happen: what kinds of skills and time are required. (For instance, we have QtGUI and SC-IDE because Jakob Leben and Tim Blechmann got a grant: they had the skills, and a grant provided the time to work full-time on it for six months.)

hjh

1 Like
41

+1 about math on Arrays and coroutines!
And multichannel expansion, etc.
I mean there is so much good stuff that one would (probably) lose moving to any other language.

1 Like
42

I think this conversation should continue here :wink:

43

just wanted to add a big plus 1 to fixing Array to be consistent whichever way supercollider goes in the future…

44

yeah I brought this up because it seems like the ide browser tutorials should have code samples highlighting some of the quirks and have better practical tips.