Also, some notes on benchmarking.
Benchmarks should be constructed so that the only difference between them is the performance aspect you want to test. The “accessing time” benchmarks are not accurate because the array and list tests simply use i, while the dictionary tests construct symbols within the loop. That is, the performance difference is not only due to the different implementation of at but also because of the ++ and asSymbol operations, which are not relevant to at time.
I understand why you wrote it that way – because most of the time, dictionary keys are symbols rather than numbers (which leads to the overgeneralization that dictionary keys should/must be symbols). But this has accidentally introduced another uncontrolled difference, so we have no way to know how much of the performance difference is because of the methods, and how much because of inconsistent test scenarios.
I’d use numbers in the dictionaries.
Second, avoid unnecessary slow operations. Environment variables are actually entries in an identity dictionary! So your array tests also include dictionary lookups. Use declared or interpreter variables instead. It isn’t necessary to create sub-arrays either.
And a side note – benchmark is in sc3-plugins, so, before doing the comparison, I’ll replace this with bench from the main class library. (And another sidenote: It’s useful to run the test multiple times. I prefer to do at least 5 bench calls and average the results. However, in this case, I found that for “Array add,” I got 5-point averages ranging between 3 and 12 ms, so, not only was the execution time over-reported, the margin of error was almost as large as the measurement itself. I ended up bumping the number of iterations up to 1000.)
Let’s compare your version of the tests vs mine:
| Test |
Yours |
Mine |
| Array add |
22.02 |
1.84 |
| List add |
22.68 |
3.87 |
| Event add |
19.72 |
2.28 |
| IDict add |
19.29 |
2.20 |
| Dict add |
71.91 |
44.1 |
| Array at |
1.85 |
1.32 |
| List at |
6.29 |
1.44 |
| Event at |
18.82 |
1.72 |
| IDict at |
19.91 |
1.70 |
| Dict at |
38.2 |
17.14 |
In the array-add test, when I remove the [("i"++i).asSymbol, i] and switch to interpreter variables, execution time drops by more than 90%. This means that 90% of your measurement is actually the time to look up ~array and to construct the symbols and 2-item arrays, and only 10% the add time + the loop! This explains the surprising result that Event / IdentityDictionary add seemed faster: because these tests are doing less work.
This underscores the importance of making sure the test scenarios are as absolutely identical as possible, except for the one thing you want to compare.
The third column does reflect the results I would expect. List is slightly slower than Array because each add or at performs one additional method dispatch. Event and IdentityDictionary are practically identical in performance, and a bit slower than Array. Dictionary is much slower because Event/IDict do the hash table lookup in the C++ backend, while Dictionary does it in (slower) sclang code.
hjh
PS My modified benchmarks:
// Writing time of Array, Event, Dictionary and IdentityDictionary.
(
z = (0..9999);
f = { |func, num = 5|
var sum = 0;
num.do {
sum = sum + bench(func, false);
};
sum / num * 1000 // ok, let's do ms
};
)
f.({ a = []; z.do { |item| a = a.add(item) } }, 1000);
f.({ l = List[]; z.do { |item| l.add(item) } }, 1000)
f.({ e = (); z.do { |item| e.put(item, item) } }, 1000)
f.({ i = IdentityDictionary(); z.do { |item| i.put(item, item) } }, 1000)
f.({ d = Dictionary(); z.do { |item| d.put(item, item) } }, 1000)
// Accessing time of Array, Event, Dictionary and IdentityDictionary.
f.({ a.size.do { |i| a[i] } }, 1000)
f.({ l.size.do { |i| l[i] } }, 1000)
f.({ e.size.do { |i| e[i] } }, 1000)
f.({ i.size.do { |j| i[j] } }, 1000) // note, the collection is 'i' so we need a different index name
f.({ d.size.do { |i| d[i] } }, 1000)