Hadron Updates

Development
1

While active with Hadron development, I will publish monthly (ish) updates on the Hadron website, cross-posted to the scsynth forum. The hope is to drive engagement and volunteerism.

We’ve Moved!

After hearing the news of their involvement with ICE, I looked into leaving GitHub. However, this recent rebranding of GitHub to center around their Large Language Model product, Copilot, provided a second big push to go.

I’m moving all Hadron-related projects to a self-hosted GitLab instance I set up at Solitary Bees. Please ping me if you’d like me to send you an account invite. We’re still setting up shop regarding continuous integration and other DevOps infrastructure, but I plan to keep Hadron and my other personal projects there for the foreseeable future.

Fuzzing

Fuzzing is a vital part of modern testing strategies, and I’m pleased to report we’ve added fuzzing support to this new Rust-based iteration of Hadron. Please read the instructions at docs/fuzzing.md in the Hadron repository, try them out, and file any bug reports if you find them.

Parser Updates

A JIT compiler like Hadron needs to parse sclang quickly to allow for rapid feedback to users on input code snippets. Modern language frontends see a lot of invalid and incomplete code due to the rise of fast-feedback development tooling like LSP, popularized by VSCode.

So, not only do parsers need to be fast, but they need to give actionable feedback to users while they are typing. We’d like to “raise the bar” for user feedback with Hadron, providing great error messages with lots of context and helpful hints about what might be going wrong.

I’m writing two parsers, one that parses the input source code in detail and one that only builds an “outline” of the parsed code and is much more robust in the face of errors. If parsing fails using the detailed parser, it can hand off to the outline parser for error recovery and to make better suggestions around the input code. Furthermore, the outline parser will allow for lazy compilation of the class library.

I copied another speed-centric design choice from the Carbon Language Project by representing the parse tree in a linear array, holding the parse nodes in postorder traversal order. This structure keeps the parser inner loops free of memory allocations. This linear structure also makes an iterative parser implementation easier, saving additional computation and memory required by a recursive parser.

Help Wanted

  • Rust: #1 I’ve broken the detailed parser work into a bunch of handle_ functions in lang/src/toolchain/parser/tree. Please let me know if you’d like to tackle a particular part!
  • DevOps: #2 Are you interested in learning how to build a continuous integration pipeline on Google Cloud?
  • Docs: #3 The parser has been moving around quite a bit but should now be settled down enough that we could start to pay down some of the RustDoc debt.
  • Rust: To get early feedback on the parser, particularly the error messages, I’ll be spinning up a Rust-based LSP implementation targeting WASM.
  • TypeScript, VSCode: Given a rust-based WASM LSP crate, add the trimmings for a full-featured sclang VSCode extension.
  • Anyone: Suggestions and feedback. Is there something on your mind about sclang and Hadron? Let me know!
11 Likes
2

Very nice idea with the update posts, keep them coming! :slight_smile:

I was not aware that you switched to Rust - I’m still very new to Rust and trying to get my first project up and running, but I’ll be skimming through the code occasionally, but it seems to have a good sweet spot between maintainability, abstraction and speed. I hope some day I can contribute some things.

I also recently took a look at implementing a sclang lexer in Rust and came across this nice looking library called GitHub - zesterer/chumsky: Write expressive, high-performance parsers with ease. which seems like a very nice approach at lexer/ast construction - but it seems you also got one working now, very nice, congratulations! :slight_smile:

Concerning the CI - why do you want to use the Google Cloud for this? I always appreciated the Gitlab CI (although the Github Actions are more convenient nowadays) and once you have a runner up and running it is also pretty nice - at least better than Jenkins :smiley: If you want to stay closer to GitHub but want to avoid GitHub I think Gitea is also worth taking a look - it has something similar to GitHub actions, see Compared to GitHub Actions | Gitea Documentation - but most likely you still need an environment for your runner in this case.

3

I looked at a number of parsing libraries in Rust, including chumsky, but ultimately decided to hand-code one out of a combination of reasons:

a) Most wouldn’t, or wouldn’t easily, support building a linear/postorder parse tree output, relying instead on the traditional dynamically-allocated tree data structures, and
b) I didn’t like their API for creating detailed error messages, and
c) Many sacrifice speed for usability, whereas I feel like a handwritten parser you can have both

I am using GitLab CI. The GitLab instance at solitarybees.us is a VM running on Google Cloud. The runner is a separate instance VM. I just have some sysadmin work left to do to get everything set up, or a volunteer could step in and save me the trouble.

This is actually my second attempt at leaving GitHub, and the first attempt was using Gitea. I like Gitlab better. But yes, either Gitea or Gitlab will require the same amount of work in rebuilding my CI system in this new context. I can do it, but this second attempt on Hadron I’m trying to be more communicative about what I’m doing and more open in asking for help, in the hopes that I can build a sustainable community around the project this time.

6 Likes
4

This is a beautiful project and very necessary. Thank you for pursuing it. I’d love to help out if I can once things settle down around here.

Sam

3 Likes
5

I’ve renamed this thread to Hadron Updates and will post all of them here, to avoid excessive thread spam on the forum, and to also allow people to subscribe to the thread should they desire to get notifications about updates. Here’s the December update:

Hadron 2023 December Update

This has been a busy time for me professionally, with lots of business travel, so progress is a bit slower than I would like. Want to help accelerate Hadron development? Get involved and volunteer!

Hadron and Language Evolution

I wrote a short article outlining some of my plans to support SuperCollider language evolution. Please take a look. In short, Hadron will establish a clear Code of Conduct and governance model, it has an Apache license allowing for commercial projects, and it promises interoperability between legacy code and new language features. There’s a lot more to add here, but hopefully, this can start a conversation about Hadron’s role in the future evolution of SuperCollider. Feedback welcome!

Parser Development

Hand-coding a parser is a lot of work! I restructured the parser testing to compare it to output from Sparkler, my ANTLR4-based SuperCollider parser library, and this has helped development by increasing my confidence in the parsing code. I hope to wrap up the main part of the parser, barring more exotic constructions like list comprehensions and generator expressions, by the end of the year. Next up, semantic analysis, then VM bytecode generation! Let me know if you’d like to get involved.

Continuous Integration

I got CI set up and working, so Hadron changes are again validated with formatting and unit testing, as well as a WASM32 build to ensure we maintain WASM compatibility in the language core. I roughed out some ideas for continuous fuzzing infrastructure, but this was feeling like a big project and a distraction from Hadron so I put it aside for now. I continue to fuzz Hadron ad-hoc between each parser commit and have found several issues this way. I can’t recommend fuzzing language frontends enough!

Help Wanted

  • Code of Conduct: I’d like Hadron to have a robust CoC to protect and foster a positive community where all are welcomed who feel they can meaningfully contribute. If you have experience moderating CoC for programming languages, I’d love to chat.
  • Rust: #1 I’ve broken the detailed parser work into a bunch of handle_ functions in lang/src/toolchain/parser/tree. Please let me know if you’d like to tackle a particular part! These smaller issues are a great way to get started hacking on Hadron.
  • Docs: #3 The parser has been moving around quite a bit but should now be settled down enough that we could start to pay down some of the RustDoc debt.
  • Fuzzing Infra: Feel like working on a standalone async Rust project not on the critical path but critically important to Hadron? I have just the project for you.
  • Rust: To get early feedback on the parser, particularly the error messages, I’ll be spinning up a Rust-based LSP implementation targeting WASM.
  • TypeScript, VSCode: Given a rust-based WASM LSP crate, add the trimmings for a full-featured sclang VSCode extension.
  • Anyone: Suggestions and feedback. Is there something on your mind about sclang and Hadron? Let me know!
5 Likes
6

GPL also allows commercial projects as long as you give back to the community…

7

I didn’t say that GPL didn’t.

8

So why the step back, or the change?

9

I feel that this language here is a bit unfair. I want to answer this question because there may be other readers who are curious about my decision to license my own work differently, and I’m happy to explain. But calling the change a “step back” feels like if I answer this question without pointing this out, I would be tacitly accepting the perspective that Hadron’s license is somehow less than SCv3’s, which I don’t feel is the case.

Trick question aside, I’m essentially saying in more words what I said above. There’s been a lot of debate (for no essential change) about this on this very forum, I have no interest in continuing it. My view is that SCv3 has the right license for it. With Hadron I wanted users to have more options. I think people should be able to have choices on how they license their creative works with Hadron. The fact that sclang is an interpreted language and so the interpreter must usually be distributed along with software written in sclang is an important part of my licensing decision.

I know there are others who don’t share my views, and out of respect for their views I’m broadcasting the point up front - Hadron is Apache licensed, contributions to Hadron will necessarily also be Apache, so if that’s not ok, please don’t contribute.

2 Likes
10

FWIW, I agree, and want to recognize the skill that’s gone into deflecting energy that could have escalated in a negative direction.

hjh

3 Likes
11

The word is not about you, but about software licenses. You don’t need to turn it into personal offense, because clearly it is not.

Ok, I started to understand your rationale, although I’m not sure I agree (if I guessed correctly). (The argument of the small inventor/creator is very insufficient re a much larger problem of the software system)

(I will not continue this conversation if that’s the way you guys are trying to push it.)

13

Hadron Update 2024 February

New Logo

Hadron has a new logo design, created by Peter Hague. I wanted something evocative
of the SuperCollider logo, but representing a fresh take on things.
I feel Peter did a great job and am very grateful for his work.

Parser (mostly) Parsing

Using the ANTLR-generated Sparkler parser
for comparative testing really helped me to build confidence in the parser, and it’s
code complete, except a few more exotic constructs like
list comprehensions and
multi-variable assignments.
Both are pretty self-contained parsing projects with a ton of similar worked examples
already in the parser, so would make great first projects to tackle if you’re considering
diving in to Hadron contributions.

I’ve fuzzed the parser a bit and fixed a few obvious flaws it found, but further fuzzing is
always welcome. Try it out and please share any findings you generate!

The big missing piece of the parser is diagnostics. One of the best things about hand-writing a
parser is the opportunity to provide handwritten feedback to the user about parsing errors. I’ve
stubbed out a preliminary implementation but there’s a ton left to do here. Error recovery is also
not started, although I do have some design ideas started in this space.

Server

I completed the text model, so now Hadron can efficiently maintain an in-process copy of text under
edit. The goal was to reduce latency on parsing and further analysis, by skipping large copies and
allowing Hadron to understand what text had actually changed. I plan to start fuzzing the server
API next.

hadron.run

I’ve been pushing early prototypes of the Hadron WASM work here. I’d expect this site to
stay pretty broken for a while, but the goal is to facilitate code sharing and SuperCollider
hacking in a manner similar to Compiler Explorer does for C++ and Rust. We’ve a long way to
go, but it’s exciting to see the site up and serving bits, even if they are somewhat broken
and woefully unfinished bits :).

What’s Next

Generally the focus is going to be on end-to-end compilation and interpretation of a few very simple
sclang expressions. This requires building some nontrivial infrastructure, like a prototype garbage
collector and dispatch system. I want to get enough of the interpreter working that I can build out
the integration testing design, and then widen the platform from there.

16 Likes
14

Great news Lucile! I think there was supposed to be a link to the WASM site, but it didn’t make it into the post. Where is that?

Sam

15

Sure thing Sam, it’s https://hadron.run

1 Like
16

It seems that Hadron is now hosted on sourcehut, right?

17

It is, yes, Hadron’s new development home is hadron: SuperCollider Implementation. There’s links to the Discord and email lists I’ve set up there. Please note that all Hadron communication fora are subject to the Hadron Code of Conduct, here: ~lnihlen/hadron (main): CODE_OF_CONDUCT.md - sourcehut git . Folks coming from this community in particular should note that the rules around plagiarism are quite different from what’s tolerated in the SuperCollider community.

I don’t feel safe or included on this forum and so will be deactivating my account. If you’re interested in Hadron, please feel free use the communication platforms I’ve set up to support keeping myself motivated and productive on Hadron.

Cheers!

\L

1 Like
18

Okay. Thanks for the information.

19

@lnihlen, sorry to hear you won’t be around this forum.

Thanks for the updated Hadron links. I really appreciate your work on this project, and your wider contributions to the SC community.

20

indeed! excited to see where Hadron goes and will miss your presence here!

21

Really sad to hear about your decision - hopefully this will not be indefinite.

Regarding the license - I think this is a thorny issue, but I also respect that you want to license your work under your conditions.
The problem I see is that Hadron has the potential on becoming the new reference implementation of sclang (wasm support alone is a killer feature). But this is in conflict with the license of the SC environment/ecosystem.
If there is a decision to make Hadron a first class citizen of the SC eco system in replacing sclang we would have to separate certain code areas b/c of the license (which is work but feasible), but there are also many current and past contributors which only contribute under GPL - at least I would be such a person.
Therefore I have doubts that Hadron will replace sclang anytime, but then it will be difficult communicate/argue with users why we not only have 2 servers, but then also 2 language implementations - which would be really bad if they start to diverge for some reason.

I would be really interested what your opinion on this is.

I don’t know what a good modus operandi would be for discussion now :confused: Feel free to PM me if you want to give some insights from your part or look out for a mail of mine in the internet as I don’t like Discord.