Maximum Collection size(s)?

Development
1

Hi All,

At the moment I’m working with some large data sets, and if I’m not careful, end up crashing sclang with the following message returned to the post window:

Interpreter has crashed or stopped forcefully. [Exit code: 11]

Some example code:

~testSize = 2.pow(25).asInteger  // works fine
~testSize = 2.pow(27).asInteger  // Array FAILS here (64bit ints?) <--
~testSize = 2.pow(29).asInteger  // FloatArray FAILS here (32bit ints?) <--

~testArray = Array.newClear(~testSize)
~testArray = FloatArray.newClear(~testSize)

My system is:

  • SuperCollider 3.13.0-rc1
  • MacOS 13.0.1.
  • Physical Memory: 128.0 GB

Is there an easy way to find out the maximum sizes available for the various Collection subclasses? (E.g., max size supported for Array, List, FloatArray, etc.)

Also, is this limit fixed with the SC compile? (Presumably, yes?)

Thanks!!

2

Not a clue, but ints are 32bit in sc, and floats 64.

Might you be able to chunk it up? Having lists of lists?

3

FloatArray is a type of RawArray.

RawArrays are more like C arrays – you know in advance the data type of every element, so you can predict the memory size based on the size of the data type times the number of elements.

FloatArray is for single precision floats, so the storage size will be 4 * array_size. (For a raw array of double precision floats, use DoubleArray.) It’s suggestive, though not conclusive of anything, that 4 * (2 ** 29) == (2 ** 31).

Array is an array of “slots” – basically pointers to SC objects. This is how Array can store mixed types, with the side effect that Array requires more storage (because the memory chunk for the array itself is – I think(?) – storing pointers, and the objects require storage on top of that).

Not to my knowledge. I believe (but haven’t checked the source code) that RawArrays just malloc() what they need. But you’re not running out of physical RAM… so I don’t know.

hjh

4

As far as I have tested, the followings are the limited maximum size of each of them:

~anArray = Array.newClear(134217724)
~aFloatArray = FloatArray.newClear(536870896)

Sadly, I could not understand the reason for the limitation, but I hope knowing those limitations could be helpful for some work of mine and yours.

5

Thanks @prko, I can confirm this is also the case on my installation.

As an aside:

// Array
2.pow(27).asInteger - 134217724  // -> 4

// FloatArray
2.pow(29).asInteger - 536870896  // -> 16
6

Therefore, it would be acceptable to assume that the maximum size of

  • an array instance is (2 ** 27) - (2 ** 2)
  • a float array instance is (2 ** 29) - (2 ** 4)

Does each of them require a certain number of bytes to allocate to physical memory?

7

One problem is that in sclang collection sizes are limited to 32 bits - even on 64-bit platforms.

To make things worse: sclang’s memory allocator, while having size_t as the size parameter, internally works with signed 32-bit integers. If you pass a larger size than 2^31 (= 2147483648) it enters “undefined behavior” territory and crashes.

A “slot” in 64-bit sclang takes up 16 bytes (8 bytes for the union + 8 bytes for the type tag and padding). 2^31 / 16 = 134217728. This matches @prko’s empirical limit for Array sizes.

1 Like
8

Of course, if there’s no memory usage then you haven’t stored anything.

For RawArrays, it’s datatype size * number of elements.

For Arrays, a slot looks like this:

typedef struct pyrslot {
    long tag;

    union {
        int64 c; /* char */
        int64 i;
        double f;
        void* ptr;
        struct PyrObject* o;
        PyrSymbol* s;
        struct PyrMethod* om;
        struct PyrBlock* oblk;
        struct PyrClass* oc;
        struct PyrFrame* of;
        struct PyrList* ol;
        struct PyrString* os;
        struct PyrInt8Array* ob;
        struct PyrDoubleArray* od;
        struct PyrSymbolArray* osym;
        struct PyrProcess* op;
        struct PyrThread* ot;
        struct PyrInterpreter* oi;
    } u;
} PyrSlot;

4 bytes for a type tag, then either 64 bits of data or a 64-bit pointer + padding = 16 bytes per slot (and in the case of a pointer, the object needs additional storage, e.g. struct PyrObject* o; has to point somewhere)… posted then saw Spacechild’s correction.

hjh

1 Like
9

Actually, the signed integer overflow already happens in the Array methods. E.g.:

Note how numbytes is a int and can easily overflow.

Sadly, sclang has never been made safe for 64-bit systems…

10

Ehm. Are the following codes C or C++? I need to learn the required part of that language to understand the problem.

11

Both technically C++, but in the older unsafe style, you’d be better off reading about C to understand what’s going on.

1 Like