Looks good! The quarks are (at least mostly) only for language-side extensions written in sclang. Server plugins is a bit more complicated - there is the SC3-plugins package, which currently is in a semi-frozen state, unlikely to accept new contributions.
I recall there has been discussions about extending the quarks system to include server plugins, but it’s afaik not there yet. Meanwhile I guess the best option is to make your own binary release (as you’ve already done), and maybe make a pull request to the Awesome SuperCollider list?