(feat) Semantic tokens

[jasonlyu123]

related to #71

This implements the semantic token for typescript. And because of the typescript implementation, this feature doesn't work with javascript.

I tried to add a special highlight to the event dispatcher function but the event semantic token type is not supported by a lot of themes. It ends up not highlighted. I didn't include it in the PR but maybe we could make it optional and hide it behind a setting so that people with the supported theme can enable it.

The original feature request asks for svelte specific highlight but I didn't think of any svelte specific semantic token type. Most of the special syntax doesn't have much to be semantically analyzed. The action, transition, and animation are all supposed to be a function. What do you guys think? Is there any syntax you would like to be covered in semantic tokens?

Screenshot:

Theme: GitHub dark

Without semantic tokens

With semantic tokens

[dummdidumm]

Very nice! Only thing I gotta check some more is how much it affects performance and how different it looks (in terms of default true or false). We might also think about checking the TypeScript settings and not enable semantic highlighting if the user turned it off for TypeScript, even if it's turned on for Svelte.

About your question: I don't have that much of an opinion on that and I like the current coloring. We might start with what we have here and enhance if there's more feedback.

[dummdidumm]

Had a close look, seems either my eyes don't see it or my theme doesn't colorize as much, either way there's not much difference although I see some semantic tokens applied when inspecting with the token inspector 😄
I noticed one thing we can do for speedup, see code comment.

[jasonlyu123]

Another problem I forgot to mention is that the source map issue of each block is highlighted by the semantic token. So it does look a little bit odd.

[dummdidumm]

I'm okay with this since it's opt-out. We should really tackle this off-by-one-error at some point, if that's even possible. Edit: I see you opoened an issue over at Magic String for this, nice! Edit 2: I think the issue you found is actually correct behavior, but there's still something wrong in my opinion (see the issue for more details).

[Monkatraz]

Just to throw my two cents in about this feature: My personal theme makes heavy use of semantic colors, and I really appreciate the effort undertaken to implement them here.

On a more technical note: Many themes for VSCode are quite old - they don't specify semantic token colors even though they could make the syntax highlighting significantly more consistent or meaningful. A good example would be with const - using semantic tokens, you can tell at a glance which variables are const and which are let. This extends to functions being passed as variables, too.

tldr: on a lot of themes you won't see much of a difference, but on others it's huge

EDIT: here, I got an example!

Off:

On:

Just for fun, here is what you can notice in that last picture:

• FaunaDB.Client was recognized as a class, not a type
• this.client gets colored correctly as a property, not a generic variable
• Functions of this are colored as functions, even when they're not being called
• q is being colored as a constant anywhere it's used, which is correct
• ENV, which is a constant, is being colored as a constant even though it lives in an entirely separate file