Arrays as arguments to SynthDef?

This sounds all doable. I think it needs splitting the task: choosing the data structure in which Buffers and cue points are stored and writing the SynthDef for switching (also see James’ general remarks in this parallel thread Parsing array into interpretable string for genetic algorithm - #4 by jamshark70, it perfectly applies here too).

I would approach it like this: audio Buffers can be stored in an Array. Cue points (I assume you have several start and end points per Buffer) can also be stored in Buffers. These cue point buffers (one cue point buffer per audio buffer) would also be stored in an Array. Cue point numbers per Buffer could be stored in a separate buffer or in the cue point Buffers themselves.
By doing so there is no need to use array arguments at all, you can define the first Buffer index and the number of Buffers (analogously for cue point Buffers) as SynthDef args. Choice is done with TIRand:

// Array of Buffers
~buffers = [ ... ] 

~bufOffset = ~buffers[0].bufNum

~numBuffers = ~buffers.size

// Array of Buffers
~cuePoints = [ ... ] 

~cuePointOffset = ~cuePoints[0].bufNum

// numbers of sections to choose from per Buffer, could also be stored in cuePoint Buffers
~cuePointPairNums = Buffer(...)



SynthDef(..., { |bufOffset, numBuffers, cuePointOffset, ( cuePointPairNums ) ...| 
 	...
})

The next task would be writing the SynthDef for switching. If you search the forum for PlayBufCF several threads with related topics are popping up. If you don’t want to use PlayBufCF see James’ SynthDef as a template: How to get rid of popping sounds when jumping around Playbuf - #2 by jamshark70 Basically you’d just have to toggle between two PlayBufs (or BufRds) with crossfade (e.g. with SelectX)