From the point of view of actual calculations: all audio related calculations in scsynth are started at soundcard buffer boundaries.
This means that for large buffers, there is a lot to be done, but can be amortised over the whole period of the buffer, for short buffers, less is to be done, but it has do be completed within a shorter time. My guess is: Because your set is heavy, you can see the effect of the trade off between these two.
The balance may depend on what kind of things are really going on, probably also hardware architecture etc.