This is probably correct. I don’t know the details of how node.js handles subprocess execution - I wonder, if it is ONLY given an executable name (e.g. sclang), it uses the usual shell functionality to search PATH to find the binary? This would be ideal.
A quick architecture discussion here…
Having sclang host it’s OWN language server is not ideal. However:
-
sclanghas enough introspection to supply MOST of the language server functionality already. This means bootstrapping a LOT of functionality without requiring a full parser that would be able to give us symbols etc. for the entire class library. As tooling for a “proper” language server comes along (e.g. being able to access and store the SuperCollider parse tree with a coherent API), things that are currently done usingsclangintrospection could be migrated. I have a practical concern here: I use SuperCollider to make music, and I wanted to build an better environment for myself to make music first and foremost. This was a way to iteratively provide better workflows and tooling without it being contingent on a “refactor the entire SuperCollider compile pipeline” epic. -
I can’t develop or maintain a full language server for
sclangby myself, and the SuperCollider project as a whole is bottlenecked re. maintenance of C++ code. Writing the majority of language server functionality insclangmeans that a much larger number of the SC community can help maintain, add features, fix bugs, etc.
In the long term, I’d like to improve the overall architecture iteratively:
- If some language server functionality is prohibitively slow, migrate this to C++ primitives or (ideally) address this as a general improvement to sclang performance.
- When the sparkler parser / sclang’s own internal parser are improved, use these to supply code metadata (symbols etc) rather than relying on introspection. Once introspection is no longer needed, the language server could be run outside of the “active” sclang process.
- In the long term, I would imagine a setup where the core language server business logic is still written in sclang, and runs as a separate standalone process. Performance-critical “database” functionality would be migrated to either a C++ library baked into sclang, or external C++ tool with simple API’s for doing things like autocomplete requests.
- I don’t plan to ever remove the “live” connection between the language server and an sclang process, even if these eventually run as separate processes. A really critical part of what the LSP provides is the ability to autocomplete and show metadata about things like environment variables and def-style constructs (
Ndef,Pdef, …). This stuff is really huge for performance-focused / livecoding SuperCollider use, and I think it’s important to leverage this as much as possible.
1 Like
The subprocess API’s in vscode do seem to use the shell PATH. So, maybe the algo here for ALL platforms is something like:
if (tryToExecute("sclang")) {
sclangPath = "sclang"
} else if (exists(defaultPath) {
sclangPath = defaultPath
} else {
sclangPath = askUserForPath(defaultPath);
}
A few minutes ago I built sc from source on Ubuntu ARM.
In my experience, the path of sclang installed after building from source is as follows:
/usr/local/bin/sclang
Now I can use SuperCollider in vscode, but I cannot use SC-IDE due to the following error when starting SC-IDE:
compiling class library...
Found 871 primitives.
Compiling directory '/usr/local/share/SuperCollider/SCClassLibrary'
Compiling directory '/usr/local/share/SuperCollider/Extensions'
Compiling directory '/home/parallels/.local/share/SuperCollider/Extensions'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/Notator'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/Strang'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/sc3-dot'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/Singleton'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/Collapse'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/WindowViewRecall'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/Log'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/UnitTest2'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/Deferred'
Compiling directory '/home/parallels/.local/share/SuperCollider/downloaded-quarks/LanguageServer'
numentries = 900188 / 14256350 = 0.063
5965 method selectors, 2390 classes
method table size 14883552 bytes, big table size 114050800
Number of Symbols 14061
Byte Code Size 450972
compiled 391 files in 1.21 seconds
Info: 4 methods are currently overwritten by extensions. To see which, execute:
MethodOverride.printAll
compile done
localhost : setting clientID to 0.
internal : setting clientID to 0.
Couldn't set realtime scheduling priority 1: Operation not permitted
ERROR: Message 'implementingClass_' not understood.
RECEIVER:
class Document (0xaaaaebbeeb00) {
instance variables [19]
name : Symbol 'Document'
nextclass : instance of Meta_DocumentSymbolProvider (0xaaaaeb52e400, size=19, set=5)
superclass : Symbol 'Object'
subclasses : instance of Array (0xaaaaec5fa080, size=1, set=2)
methods : instance of Array (0xaaaaebbeec80, size=81, set=7)
instVarNames : instance of SymbolArray (0xaaaaebbef500, size=16, set=3)
classVarNames : instance of SymbolArray (0xaaaaebbef780, size=8, set=2)
iprototype : instance of Array (0xaaaaebbef600, size=16, set=4)
cprototype : instance of Array (0xaaaaebbef840, size=8, set=3)
constNames : nil
constValues : nil
instanceFormat : Integer 0
instanceFlags : Integer 0
classIndex : Integer 226
classFlags : Integer 0
maxSubclassIndex : Integer 227
filenameSymbol : Symbol '/usr/local/share/SuperCollider/SCClassLibrary/scide_scqt/ScIDE.sc'
charPos : Integer 9667
classVarIndex : Integer 285
}
ARGS:
class LSPDocument (0xaaaaebbdf2c0) {
instance variables [19]
name : Symbol 'LSPDocument'
nextclass : instance of Meta_LSPDocumentChange (0xaaaaec3422c0, size=19, set=5)
superclass : Symbol 'Document'
subclasses : nil
methods : instance of Array (0xaaaaebbdf440, size=77, set=7)
instVarNames : instance of SymbolArray (0xaaaaebbdfcc0, size=24, set=4)
classVarNames : nil
iprototype : instance of Array (0xaaaaebbdfe40, size=24, set=5)
cprototype : nil
constNames : nil
constValues : nil
instanceFormat : Integer 0
instanceFlags : Integer 0
classIndex : Integer 227
classFlags : Integer 0
maxSubclassIndex : Integer 227
filenameSymbol : Symbol '/home/parallels/.local/share/SuperCollider/downloaded-quarks/LanguageServer/LSPDocument.sc'
charPos : Integer 1151
classVarIndex : Integer 293
}
CALL STACK:
DoesNotUnderstandError:reportError
arg this = <instance of DoesNotUnderstandError>
Nil:handleError
arg this = nil
arg error = <instance of DoesNotUnderstandError>
Thread:handleError
arg this = <instance of Thread>
arg error = <instance of DoesNotUnderstandError>
Object:throw
arg this = <instance of DoesNotUnderstandError>
Object:doesNotUnderstand
arg this = <instance of Meta_Document>
arg selector = 'implementingClass_'
arg args = [*1]
Meta_LSPDocument:initClass
arg this = <instance of Meta_LSPDocument>
Meta_Class:initClassTree
arg this = <instance of Meta_Class>
arg aClass = <instance of Meta_LSPDocument>
var implementsInitClass = nil
ArrayedCollection:do
arg this = [*1]
arg function = <instance of Function>
var i = 0
Meta_Class:initClassTree
arg this = <instance of Meta_Class>
arg aClass = <instance of Meta_Document>
var implementsInitClass = nil
ArrayedCollection:do
arg this = [*227]
arg function = <instance of Function>
var i = 131
Meta_Class:initClassTree
arg this = <instance of Meta_Class>
arg aClass = <instance of Meta_Object>
var implementsInitClass = nil
Process:startup
arg this = <instance of Main>
var time = 1.243486635
Main:startup
arg this = <instance of Main>
var didWarnOverwrite = false
^^ ERROR: Message 'implementingClass_' not understood.
RECEIVER: Document
Couldn't set realtime scheduling priority 1: Operation not permitted
SCDoc: Indexing help-files...
SCDoc: Indexed 1428 documents in 0.52 seconds
LSPDocument needs to be in a scide_vscode folder.
hjh
Yes, obviously one of the essential features of lsp is to be “live”. I didn’t question it.
But wouldn’t it be possible to have two sclang processes? (I didn’t mean to suggest using another language for lsp). That’s what I thought when writing the first message.
@jamshark70
@semiquaver
Thanks!
I saw the file “Document.sc” in the “scide_vscode” folder after installing Quark in vscode by running “SuperCollider: Update LanguageServer.quark”. I mistakenly thought this file was “LSPDocument.sc”.
After removing the path to this quark in the SC-IDE preferences and its folder using “Files”, I reinstalled it by running “SuperCollider: Update LanguageServer.quark”. Now the folder “scide_vscode” does not exist.
This is strange because I did not change the Quark folder after each installation!
The first time it seems to install the master version of LanguageServer.quark; the second time it seems to install the development version of LanguageServer.quark.
I see… I just had a try.
There are two possible fixes (which I just noted in github as well):
- Move the LSPDocument class definition into scide_vscode.
- Or, guard the
implementingClass_call:Document.tryPerform(\implementingClass_, LSPDocument);. (EDIT: This could turn out to be problematic though, after the proposed refactoring.)
Easy fix…
Also prko is correct that, currently, “update language server quark” is fetching the development branch, which doesn’t have scide_vscode/. Manually doing git checkout master resolves this (though, eventually, users shouldn’t have to do this by hand).
hjh
1 Like
Trying to install via virtual machines on the same machine confused me.
I tested on Ubuntu, MacOS and Windows with SC3.14.0_dev. Some problems have been solved, but some problems remain. On Windows, the results I reported with images are the same.
Yes, the fix is relatively simple! Thanks!
I now also think that the vscode-sc extension could replace the SC-IDE if some minor issues are fixed. I appreciate @scztt and I am sorry that I am just using what you have made for us and asking for more features without helping you solve these problems. What you are doing is beyond my capabilities, unfortunately…
1 Like
Yes, definitely there could be two. Right now, since the “active” process owns basically all the data we need, a second process wouldn’t be doing anything other than basically proving a proxy for the active process. Splitting it in two wouldn’t even get you more stability in case of e.g. a crash or compile error (at least not without work), since the Language Server sclang process would have the same disconnection problems as vscode has now. Probably the time to make the split is when we can get the full parse tree and a symbols database as data, rather than just having to introspect it from the running sclang itself.
What it might give you, though, is a response to introspection requests that wouldn’t interfere with musical timing in the primary sclang process. Since the sclang interpreter isn’t preemptive, requests from vscode could conceivably block musical threads in the interpreter, which is undesirable. The blockage is likely to be short and probably would never be noticed – but it could be eliminated if requests go to a second process. OSC messaging could sync Pdef etc types of keys.
hjh
Until you can perform SOME meaningful amount of non-trivial work in the language server sclang instance, two processes is just pushing the problem around. There are some expensive calculations that COULD be pushed to a second process now, but there are two things holding me back:
- The language server still has reliability issues that affect everyday usability. Another process increases the potential for issues here, so I’m hesitant to introduce until reliability is improved enough so that things work seamlessly for a “normal” SC user, in “normal” usage scenarios.
- 95% of the performance cost of the LSP can be fixed by caching to avoid duplicate work. This has to be done regardless of whether we have a second process or not, so this is where I’ll focus energy for now.
- I have no performance metrics whatsoever, so I have no idea whether LSP stuff is even remotely slow enough to affect musical timing. My first priority is to add some performance metrics to the language server, to get a picture of which operations are expensive enough to look at optimizing.
FWIW, the language server has a VERY loosely organized LSPDatabase class. This is an early attempt to factor out requests for data that (a) have the potential to be cached, and (b) could be performed against a serialized data set rather than the active sclang process (e.g. things like searching the list of methods). As it evolves, I would envision a setup where all of the LSP runs out of process, and LSPDatabase is an abstraction for a connection to both an “active” sclang process, and any other sources of data required for the LSP (e.g. a parse tree).
1 Like
FWIW I fully agree that a second process for the language server is, now, the sort of premature optimization that Knuth called “the root of evil.”
In principle, time-critical applications should avoid time-unbounded operations in high priority threads. In sclang, the concern is orders of magnitude less severe than in scsynth – server messaging latency can absorb interpreter jitter up to a couple hundred ms in typical use (though we don’t know how many users are running with significantly lower s.latency) while scsynth has hard targets on the order of 10 ms (or less, for live fx use). So it’s probably not a big deal in sclang, and the vscode-sclang environment is currently experimental, so it’s way too early to consider a big architectural change like a second process.
The one that gives me a little pause is the document formatter: synchronously passing large (unbounded) strings to the sclang-format process, and synchronously reading them back (where sclang-format is probably very fast, but sclang is blocking for the reply, so it needs to be very very very fast). One suggestion you had made to me is to enable “format on type” in vscode – which, from a user interaction perspective, is fine – but does vscode then invoke the formatter for every character typed? I’d hope that vscode would wait for a pause, but I don’t know if it does or not. If so, then I’d take it upon myself to do those benchmarks since that seems potentially heavy.
But that investigation would be some ways down the road (and FWIW I haven’t even been able to build sclang-format yet). To be clear, I’m not arguing that this is a fatal flaw – I’m just considering potential risk areas, to check carefully later.
hjh
Note that this is only applies to sequencing. When playing a live MIDI instrument, you really don’t want sclang to block for dozens or hundreds of milliseconds. Now, in most cases you wouldn’t type code while playing an instrument. However, I can think of cases where you’d want to live code while another musician is playing a MIDI instrument.
So we should indeed be careful about blocking the sclang interpreter for possibly unbounded durations.
I’ve got a local branch of sclang that’s doing non-blocking IO to interact with the formatter - so this is at least in theory solveable (getting this code into SuperCollider develop will be a journey, as it probably should be part of a larger refactoring of pipe code). In practice, it may be better to have vscode invoke the formatter directly - I view. the sclang wrapper around the formatter as more of a fallback for LSP clients that aren’t set up to connect to the formatter directly.
Cool, yes, I agree that 1/ it’s better not to involve sclang in the code formatter and 2/ a non-blocking Pipe would be a very very welcome feature!
hjh
BTW: Here are a couple of color customizations that make VSCode less unpleasant to look at (“workbench: color customizations” settings, and add into settings.json):
"workbench.colorCustomizations": {
"editorBracketMatch.border": "#233843",
"editorBracketMatch.background": "#434e20",
"editor.lineHighlightBackground": "#1B2B34"
}
VSCode’s default preference for outlines highlighting the current line, and matching brackets, I found to be visually noisy and a major turnoff. Took a bit of searching to find how to change that.
"editorBracketMatch.background" is a tough one to tune: too dark and you can’t see it; too light and it obliterates the bracket itself. This color is maybe OK but I’ll probably tweak it later.
hjh
Yes. But, for example, the same words that are highlighted, so you can edit them all at the same time, if applicable. I also find it a bit annoying sometimes.
speaking of colo[u]rs: I’ve commented on a bug, but I’m not certain it is not a meatware issue of yours truly… when I cmd-shift-d a Ugen to get its ref, it always works, sometimes the browser’s background it dark grey, sometimes white. That I can live with although I’d like to understand why.
another much bigger issue. As I click a link in a helpfile, very frequently the page loading is white and empty.
In both cases, is it me the problem? if so any pointer is welcome