Haven’t look at this in detail, but my understanding is that xset creates two synths and cross-fades between them. Probably what happens is that the entire “chain” inside the Ndef, meaning all sources array, get duplicated in this process, which is why your base signal gets restarted. I guess Ndef would have to be “smarter” to avoid this and figure out which control belongs two which source in the array and only duplicate that one for the xfade.
I think the simple workaround is to separate your filter to a separate Ndef and use external signal routing.