The Future of SuperCollider Development Efforts

Development
10

@semiquaver I was maybe hoping we’d get more of a discussion on the general idea of teams before going into the specifics, so I’ll maybe hold off on a detailed explanation for now.

In short, teams are to facilitate shared ownership and direction - having teams doesn’t solve the way those teams make decisions, or how inter-team dependencies work: we still need transparent and safe procedures for that.

The idea would be that teams aren’t fixed - they can be started and stopped:

  • The teams reflect the development efforts of the Project
  • Teams would ideally have a minimum size
  • If enough people want to work on something, a new team forms
  • If a team falls below the minimum size, it is disbanded and the Project stops putting effort into something it can’t sustain

Hopefully that provides enough of an answer for now? I don’t want to bias the discussion towards having teams if it’s not something we’re interested in.

1 Like
11

Many thanks Lucile, your input is greatly appreciated, thank you for sharing your thoughts.

I agree that we need to make sure that there’s a clear way to make decisions, and that the idea of having teams doesn’t address that.

I think we need to admit that we do not have the engineering capacity right now to facilitate what is being asked of us. All of the development efforts are currently being spent on keeping it all building - I’m not sure how many people are aware of this, but that is the reality insofar as I can see (please correct me if I’m wrong though).

So, I think a well defined organisational structure coupled with mentorship will help bring people into the Project, and skill-up those that are already here but don’t feel able to contribute. But maybe others disagree, and that’s ok, all ideas are welcome!

I don’t think our ideas are at odds with each other, but could maybe be combined.

Having teams solves half of the puzzle - how we organise ourselves - but perhaps something like earned authority could be a way to solve the other half: how we make decisions. I have some concerns about such an approach, but I’d really like to hear from others. I’ll take a step back from this thread for a bit to let others share their thoughts.

1 Like
12

I agree with @jrsurge’s ideas, but I also have to underline what Lucile wrote: The direction of development should be guided by seniority and/or experience within the project or development in general, as development resources are limited and should be allocated in a responsible and sustainable way. This does not have to be a conservative direction, but experience is probably the most valuable guide to how to do things, and perhaps when not to do things.

One thing that turned me off as a (former?) contributor was that it was hard to understand whose words made a decision final and whose did not. It turns out that SC does not have such people (although I am still not really sure), and while I can now understand why SC is structured that way, I actually found it not very inviting, because at the moment of a (first) contribution I am more interested in a final yay/nay, a why and maybe how on the contribution, rather than delving into the branches of arguments.
Instead of guiding a newcomer, I felt like I was moderating a discussion with people who have more power than me (because they still decide) but also don’t know their power/seniority on a topic, which created an uncomfortable atmosphere of power imbalance. This setup may work for people who tend to be more open, but people who struggle with social situations and try to avoid them are put in an awkward position here.
Specifying who the benevolent dictator is to guide the discussion would help a lot as a newcomer, at least for me. Managing the power dynamics and legacy code of a 25+ year old ongoing project is just one task too many for a newcomer.

Another question should be how to maintain and increase maintainability, so what should the coding standards be? For example, make tests mandatory or introduce test coverage?

My proposition

would be: Create a Dev Council that welcomes everyone and discusses the direction and gives power to the Benevolent Dictator(s) [please come up with a better name for this!] and the Council acts as the sovereign.
These dictators are responsible for coordinating development within their realms for a limited time, which would most likely include the areas mentioned by jrsurge. [Sorry for the offensive vocabulary.]

The Dictators will be responsible for organizing any discussion and feedback on implementation details, and respond (maybe?) within a week. I know this is a lot to ask, but with a rotation or split duty this might be possible.

The idea of teams seems to be more inclusive, but for me personally this is not the case because it increases my frustration as it becomes so difficult to come to a decision and such a decision making process can take some weeks or months and now I have to organize a discussion again - this is the reason why I stopped contributing.

However the SC decides to move forward, I think it is always good to talk about how to contribute! Torvalds once said “just pick the FOSS projects that suit your work and social style” and I think this is a good credo to live by - it will be impossible to make everyone happy, but the best way is just to discuss it in the community. I am really curious how the development of SC will continue! :slight_smile:

All in all, I would definitely like to become a more active contributor again!

PS: I also find the current state of discussion platforms a bit difficult - e.g. I would like to come to dev meetings, but they are only announced on scsynth.org and I check scsynth.org maybe once a month, so I always miss them and also picked up this topic only by chance.
I check GitHub almost daily and there is no post about dev meetings or this topic (since both platforms require an account to participate, it seems inevitable to make 2 posts in parallel?).
This topic should also be discussed w/ the GitHub community as well IMO.

@jrsurge : I would gladly take some CPP tandem sessions with you :slight_smile: I really enjoy this idea b/c with CPP I feel like some 2 hour google research could have been resolved in 2 minutes by asking an experienced dev “why is it like this?”

1 Like
13

IMO the topic of seniority is quite important, in particular with complex software like SuperCollider. One thing to always keep in mind is that seniority is (mostly) independent from age! I think we should try to keep “senior” SC developers in the boat. There is just so much tacit knowledge and experience that can be lost otherwise.

Personally, my time budget for open source development has become more limited and 99% of it goes into Pure Data and my personal projects. However, if anyone has any questions about scsynth or supernova internals, or C++ in general, you can always ping me on GitHub (Spacechild1).

Since we have talked about development models, I thought I may talk briefly about Pure Data, as its model is completely different from SC.

Generally, Miller Puckette has been the BDFL for 25 years. Every new feature or non-trivial change to the codebase needs to be approved and merged by him. However, since a couple of years we have a dedicated develop branch for trivial bugfixes (Commits · pure-data/pure-data · GitHub). There is currently a group of 3 people (including myself) who have write access. If a PR gets reviewed and one of us regards it as “trivial”, we can merge it right into develop. Also, since we are all seasoned developers who know the codebase inside out and trust each other, we are allowed to push trivial bugfixes directly onto the develop branch – without a PR or formal review process. We then make a PR with a list of all the fixes (e.g. stable development branch by umlaeute · Pull Request #2027 · pure-data/pure-data · GitHub) and Miller typically just merges it into master. It is a bit unusual, but I think it has worked great so far. Have a look at the recent history of the develop branch and imagine we had to open a PR for every single fix…

To be clear: I am not saying that we should copy what Pd is doing – it certainly is not how you would run an open source project with high stakes – I just wanted to give another perspective. In the end, every project needs to find its own way of doing things.

2 Likes
14

Blockquote
PS: I also find the current state of discussion platforms a bit difficult - e.g. I would like to come to dev meetings, but they are only announced on scsynth.org and I check scsynth.org maybe once a month, so I always miss them and also picked up this topic only by chance.

Pro tip: On any thread on scsynth.org (I do this specifically on the dev meeting poll and announcement threads), click on the “bell” icon on the right-hand side of the thread, then pick “Watching” from the list. You will then receive email for every posting on that thread.

1 Like
15

Developer meetings are also announced on the SuperCollider channel on Slack:

1 Like
16

Thanks so much for this initiative, it is really a good moment for that. For now, I just want to say that I’ll try to help and support wherever I can.

I am not on slack, so I’ll try to keep myself up to date here.

3 Likes
17

As I’m currently reading Daniel Pink’s Drive, I stumbled upon the – now already quite old – Agile Manifesto. In line with the thoughts of the book, it’s very much about making working and decision processes less frustrating – which I can relate to many posts in the recent debate. Although the idea comes from the business world, there seem to be bridges between the agile manifesto and open source, e.g., see:

Also, a quick combo web search shows a lot of hints, so the relation between the concepts is something people are thinking about. As I’m not a professional programmer, maybe I’m just repeating discussions some of you are very familiar with, but maybe there’s something in it worth to be considered.

18

First of all I want to express my gratitude to all the developers for the years of work on the project :slightly_smiling_face::pray:

I also like the move towards a structure that will allow more voices to be more easily heard and get involved. The only thing that kind of worried me was this part:

“‘The developers’ as a concept comes up in a few places in this forum, and we think is important for the survival of this software that representation better mirrors the user base. This should reflect not just racial representation and gender expression but all development expertise and contribution levels.”

I believe that the only thing that matters in terms of the relative authority in a development team is competence. I don’t see how racial identity or other expressions are relevant in any way unless there has been bias in the team that excluded people based on such characteristics which are not relevant to the competencies. That is to say that of course we should be careful to correct any bias if present (I have never seen any evidence of such bias in the SC community personally) but never try to enforce representation based on identity characteristics as this is not a correction for bias but an introducing of bias.

In general, I loved the mentorship suggestions and I would support @lnihlen 's suggested model of earned authority in conjunction with clear opportunities for “gathering” such earnings. For instance, I have been coding with SC for 20 years and I can most probably contribute to the sclang portion of the project to some extend. Maybe clearly designated “TODO’s” tagged with a skills level to enter that “earned authority” ladder would help me to get in when I have some time and do some work that will give back to this project which I love :slightly_smiling_face:

4 Likes
19

There’s a preponderance of evidence supporting the notion that “merit-based” systems that ignore the existing biases in our culture are actually more biased against underrepresented folks than are systems that try to address systematic inequity. We need systems that are aware of and try to address racism, classism, sexism, homophobia, transphobia, and the rest. I strongly oppose a “merit-based” proposal. Who decides that “merit”?

What I was proposing was a “contribution-based” system, where folks with a prior history of strong contributions to the software would have more authority over that software. To be clear, in this system, I personally have no authority.

This contribution-based system presumes a couple of things that I’m not sure SuperCollider has:

a) a group of senior developers with widely recognized authority over the project who would:

  • make final calls to resolve disputes
  • set overall technical direction
  • decide when to extend that authority to newer contributors and when to revoke it
  • do the enormous amount of work to identify tasks suitable for beginners, and render them in a format that can be consumed by those folks

b) a large group of motivated volunteers who are willing and able to work their way into the senior group

c) a healthy code of conduct team keeping everyone safe

As a senior dev, it takes more effort to identify and properly document a “good first issue” than it does to just fix it. Technical debt skews this bias even further in the wrong way, because of all the “baggage” the senior devs have to explain to a newcomer before they can write a single line of code. The tech debt o SC is so bad that, for a newcomer to C++ programming, I would guess it will take at least a year of time investment from more senior people on the team before that newcomer begins to produce more work than they consume in support. Attrition on OSS is brutal. How can they be sure each newcomer is worth the time? That’s time they could have spent developing the project, or even lowering the tech debt or scaling the support systems to make it easier on the next newcomer.

As a junior dev, particularly one without much prior experience, there’s a lot to learn and a lot of time investment before even one line of code lands in the repository. How can they be sure that investment will be worth it? My first view of SuperCollider dev was watching two senior devs resign due to burnout and abuse. I think that told me all I wanted to know about SC dev, which is why I started developing “SC-adjacent” projects,

The technical debt harms SC in another way here. I don’t think SC is a great first project for junior devs. The C++ code in particular is a disorganized, under-documented, antiquated mess. Yes, dealing with code bases like this is a part of the job, but not the part of the job I would typically advise junior people spend their time on. I want my impressionable junior devs to spend time on code bases that inspire them to even more discipline and best practices. I want them to internalize the message that with the right structures, clear documentation, and low tech debt we can move mountains. That is not the SC vibe.

If a junior dev had their heart set on this code base I would advise them to bias their work towards respecting the senior devs time. Don’t show up whining on the forum about how they aren’t making it easy for you to contribute. Dig in and get stuck and spend time frustrated with things and reading 15-year old stack overflow posts in desperation. That’s the job and that’s the learning experience.

I think even if we end up with a different governance model progress on (a), (b), or (c) would be great for this project. It’s ironic that there are so few senior devs left that the community doesn’t even understand how much trouble SC is in as a project. Like, it takes a certain amount of software eng experience to see the red flags but they are all over this one.Ya’ll are teetering on the brink and you don’t even know it. Makes me sad.

3 Likes
20

I don’t think SC is a great first project for junior devs.

I don’t think anybody expects an audio programming environment to be a good place to start for junior developers. In practice what I have seen a lot with Pd and SC is that people start to learn C or C++ by writing externals/plugins and once they become comfortable they might start hacking on the core.

The C++ code in particular is a disorganized, under-documented, antiquated mess.

Let’s be clear which part of SC we are talking of. What you are describing matches my experience with the sclang codebase, but I wouldn’t say the same thing about scsynth and supernova. Yes, scsynth is written in an antiquated C++98 “C with classes” style, but IMO the codebase is pretty small, organized and not too hard to understand once you know the overall architecture. Actually, I have the feeling that scsynth is the one part of SC that will survive the longest. The one thing I totally agree with is the almost complete lack of comments; that is really unfortunate. (To everyone reading this: always document your code as you are writing it – your future self and fellow humans will thank you!)

1 Like
21

I favor @jrsurge 's idea of standing up groups as a way of both increasing the number of participants ( ( Lucille’s point (b) ) and also partitioning discussion and effort. Ideally this would free up time for the most senior devs (who should probably form a meta group).

Support and onboarding become more efficient with groups as information can propagate.

Groups would also present a clear ladder of technical difficulty - moderation/docs < classLib < server < lang for example - a helpful path to bring ppl in.

This thread began with enthusiasm from people who aren’t currently contributing, that should be exploited rather than tamped down!

Reading responses form @dscheiba and @lnihlen it seems one question is how to respect and best make use of developer seniority and earned authority in this scheme - Two thoughts:

Perhaps groups could be expected or required to include at least one and/or be sponsored by a senior “trusted” person.

Or groups could include a designated chair who’s authority could be accepted and revoked by some collection of more senior devs outside the group.

1 Like
22

But wouldn’t this be the status quo? :slight_smile:

I think this ladder thinking could suggest a false sense of difficulty, complexity, or perhaps even value.
Everyone should be invited to contribute according to their interest and ability, thinking in a hierarchy suggests a prioritization that IMO interferes with such a credo.
If you have never programmed before, but are interested in DSP programming, who am I to tell you that you should start somewhere else? Curiosity is the best driving force in learning, and we should encourage that as much as possible.

Revisiting the idea of multiple BD within each domain of sc: I think it comes from the experience that it is much easier to have a 1:1 discussion than a 1:N / aka group - especially in technical domains.
Of course, this can lead to abuse of power and privilege, but it also seems to be a very efficient use of resources (something that seems necessary?), but I think a group/council can make sure that this assigned person operates in areas that the group considers desirable behavior.

A core question should be why there seems to be a lack of contribution (at least that is the motivation I read from the post)? Is it because people don’t feel included or because of the organization of the contribution? [assuming it is possible to separate them].

Personally, I think you are doing a most excellent job in the area of inclusion and transparency (this discussion is a prime example), and my suggestion would be that the management and structuring of the contribution could be improved.
At least the vagueness in/of feedback and the bureaucracy due to the 1:N setup is what frustrated me during contribution and is the reason I paused contributing, but after all I would really like to include some new ideas and do some maintenance work within the SC framework.

@josh - you asked the scsynth community for feedback, but what does the “dev council” consider to be a goal or next action within this discussion?

23

I’m hoping - to be honest - that a sort of dev council forms from this conversation, and the current people doing dev figure out how they can contribute to it.
I like the suggestions of forming teams, and I’d be very happy to be part of the scsynth / plugins group. So - how should we go about this? Should we just try this route and start organizing? One path forward may be making the initial groups as a thread here with a welcome message and listing the scope of that group - then go from there.

3 Likes
24

I would like to pick up this thread because the future of SC development is very important to me. I am using SC now for more than 20 years and grew very fond of it. I feel like I have become a reasonably experienced user, not least because of the great support from everybody in this forum.

I am using SC on macOS and Linux. On the former I run it with the native JACK backend, which is why I have to build SC regularily myself. I also tinkered with implementing and testing some features I would like to see in SC, both in the language and the server. So I have some basic notions of the code base.

If there was a possibility for me to directly contribute to the development of SC, I would be happy to get a chance to grow into such a role. The idea of developer groups who coach less experienced members is very appealing to me - I hope these will materialise and function well. As I am going to retire in a few years, I will soon have more time to spend on coding.

I also hope that this thread will pick up speed again and SC development will have a successful furture!

7 Likes
25

yes indeed! @josh would you be willing to lead things off by posting an initial thread for a scsynth/plugins group? I think it would be encouraging to all to see experienced people launching these efforts! Members could be encouraged to subscribe by clicking the bell - and if the group became sufficiently active multiple threads could be started and tagged with the group name perhaps.

One concern that both @jrsurge and @lnihlen stressed is safety - I think it would be good to stand up a CoC/moderation group alongside this first dev group. If there is any chance of giving previous developers the confidence to to re-engage it would be great!

1 Like
26

Anyone interested in joining the first Dev group - to focus on the class library - @jordan has started a thread here: Class Library Developer Group.

As there has been a lot of discussion in this thread about the importance of seniority and earned authority I hope some of our senior devs will head on over there and help get the conversation started!

I do think that we should stand up a group to look at the CoC and moderation - if @jrsurge or anyone with more experience here would like to get that started I’ll defer - otherwise I’ll put together a post a little later…

@Mike_McCormick @eckel @jrsurge @dscheiba

1 Like
27

I jumped on board for helping with development during the pandemic, but unfortunately my personal life exploded and I got real sick about it for a few years. I’m back now, and I’m wondering who to talk to. It looks like things got kind of unwieldy last year.

Look, the do-ocracy is most accessible IMO. We can teach each other and learn from each other by doing. I’ll make a pull request, I’ll get feedback from developers at all levels, and at the end it’s either accepted or rejected. If my efforts stagnate, others can clone my in-progress work and flesh it out, and my name still appears in the contribution history (if people take care to use git correctly).

I care deeply about accessibility in tech, both the commercial industry and the practice. I’m one of the marginalized. I’m disabled and can’t work full-time. I am a very competent dev but I don’t have the educational or class background, and the US South is a socioeconomic war zone. I lost 3 people last year. I have gotten stolen from when I tried to freelance. I could get better work based on the portfolio of contributions to SC I am building, but I wouldn’t want it! It’s all for-profit medical, petro, and war! My entire culture is rotten!

We work in a hostile environment. It’s a ton of work to build bridges to each other and focus on value and creativity, and you’re gonna watch some of what you build get destroyed so cheaply… I’m not into religion, myself, but I get it. There’s no absolute solution. We have to do our best and let forces more powerful than us sort out the rest.

But at the end of the day, for me all this extra rhetoric is just an additional accessibility barrier. Talk is cheap. Let’s go back to Github and tear into it. And if people need money to stabilize their lives so they can contribute, give it to them. If people dislike/have enough money and want to trade on value alone, that’s fine too. My late sister used to say “get in where you fit in”. We have a lot to learn from each other and a lot to build together. Don’t be shy, let’s get back to it. The war pigs don’t have to come after us with guns if they got us with drugs and drama. Avoid fear, shame, and dogma. Let us creatives create.

4 Likes
28

The Class Library Dev group and the Server Dev group each met a couple of times last year and there were some (I thought!) super interesting discussions and initiatives. Later at a general meeting the fear was expressed that setting up multiple groups might fracture the community so it was decided to fold the groups back into a single thread - the plan was that meetings would be held weekly with a rotating focus. This scheme persisted for maybe a month but sadly there have been no meetings at all this year.

Its heartening to see an uptick of good things happening on Github and esp to see new blood starting to stretch out there… But one shortcoming of the Do-ocracy I think is that its hard to coordinate a larger vision or to triage scarce resources. And a good number of worthy efforts seem to be stalled due to a lack of decision/review. Even if meetings might mostly be a time sink, it is nice to see who else is doing stuff and to be able to make some quick and dirty decisions in a face to face setting.

So I wonder whether there would be any purpose in restarting the smaller Dev groups? or whether ppl feel like the scene at Github is sufficient?

2 Likes
29

As stuff scales up, I can see more organizational infrastructure being handy. But really it’s about communication with the users, and since this is a programming environment there’s overlap between users and maintainers but not fully, so we still need to reach the users.

So questions like “Does your code use this library?”, “Do you have any thoughts about these proposed changes?” could appear in polls and discussions in the Discord/Slack/here. But yeah even that is like, a lot of channels to check. Keeping the technical discussion next to the code can help a lot.

I’m currently doing a big push of updates to the HID subsystem of sclang, which I personally use heavily. So it’s somewhat on me communicate my ideas and get feedback. I’ve been putting most of that content on Github and then I can link to it on these other channels.

If anybody joins me, then we become a de facto working group. I think the thought I’m trying to express is that this sort of thing is emergent, not contrived. We negotiate as we work. If you wanna know what’s happening with SC’s development, watch the code and issues. And if we discuss something on another channel we can always drop a link to it on an issue/PR comment.

Off-topic: I’m seeking funding for my work because I’m disabled but don’t have my shit together and am not on government assistance, and my tech career aborted before I could build up savings. Refer me a disability lawyer, and buy me some groceries in the meantime: https://fundrazr.com/42MvCf