The client uses calculates OSC bundle timetags based on the system clock. The audio callback, however, is driven by the clock of the audio hardware. Both clocks drift over time at different rates and possibly in different directions.
The Server tries to convert the incoming OSC time tags to sample time. (On Linux and Windows this is done with a DLL filter. On macOS the OSC time offset is periodically reestimated by a dedicated timer thread. Naturally, both strategies are only approximations.) This guarantees that the Client will not move ahead or fall behind the Server, which is important in a real-time performance context.
The problem is that the logical time of a TempoClock on the Client does not correspond to the sample time on the Server. That’s why you’re seeing this drift in your recording.
AFAICT the only possible way to get sample accurate timing on the language (and still avoid clock drift) is to drive the language scheduler by the audio callback. If anyone is interested in how this could work, see the following discussion: Keeping sclang and scsynth in hard sync
FWIW Supernova has a useSystemClock option. If you set this to false, bundles are scheduled based on the sample time. This means that a duration of 1 second in the language corresponds exactly to 48000 samples on the Server (assuming a samplerate of 48 kHz). The downside is that the notion of “now” will drift apart between Client and Server, possibly causing “late” bundles. It’s probably not an issue if you don’t keep the Server running for an extended period of time. Otherwise you can mitigate it by setting a larger Server latency.