Make ast::TokenKind more like lexer::TokenKind
#137902
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`BinOpToken` is badly named, because it only covers the assignable
binary ops and excludes comparisons and `&&`/`||`. Its use in
`ast::TokenKind` does allow a small amount of code sharing, but it's a
clumsy factoring.
This commit removes `ast::TokenKind::BinOp{,Eq}`, replacing each one
with 10 individual variants. This makes `ast::TokenKind` more similar to
`rustc_lexer::TokenKind`, which has individual variants for all
operators.
Although the number of lines of code increases, the number of chars
decreases due to the frequent use of shorter names like `token::Plus`
instead of `token::BinOp(BinOpToken::Plus)`.
For consistency with `rustc_lexer::TokenKind::Bang`, and because other `ast::TokenKind` variants generally have syntactic names instead of semantic names (e.g. `Star` and `DotDot` instead of `Mul` and `Range`).
rustbot
added
S-waiting-on-review
|
Some changes occurred in src/tools/rustfmt cc @rust-lang/rustfmt |
compiler-errors
approved these changes
Mar 3, 2025
|
r? compiler-errors @bors r+ rollup |
bors
added
S-waiting-on-bors
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Mar 4, 2025
…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#135767 (Future incompatibility warning `unsupported_fn_ptr_calling_conventions`: Also warn in dependencies) - rust-lang#137852 (Remove layouting dead code for non-array SIMD types.) - rust-lang#137863 (Fix pretty printing of unsafe binders) - rust-lang#137882 (do not build additional stage on compiler paths) - rust-lang#137894 (Revert "store ScalarPair via memset when one side is undef and the other side can be memset") - rust-lang#137902 (Make `ast::TokenKind` more like `lexer::TokenKind`) - rust-lang#137921 (Subtree update of `rust-analyzer`) - rust-lang#137922 (A few cleanups after the removal of `cfg(not(parallel))`) - rust-lang#137939 (fix order on shl impl) - rust-lang#137946 (Fix docker run-local docs) - rust-lang#137955 (Always allow rustdoc-json tests to contain long lines) - rust-lang#137958 (triagebot.toml: Don't label `test/rustdoc-json` as A-rustdoc-search) r? `@ghost` `@rustbot` modify labels: rollup
rust-timer
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Mar 4, 2025
Rollup merge of rust-lang#137902 - nnethercote:ast-lexer-TokenKind, r=compiler-errors Make `ast::TokenKind` more like `lexer::TokenKind` This is step 2 of rust-lang/compiler-team#831. r? `@spastorino`
github-actions bot
pushed a commit
to model-checking/verify-rust-std
that referenced
this pull request
Mar 14, 2025
…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#135767 (Future incompatibility warning `unsupported_fn_ptr_calling_conventions`: Also warn in dependencies) - rust-lang#137852 (Remove layouting dead code for non-array SIMD types.) - rust-lang#137863 (Fix pretty printing of unsafe binders) - rust-lang#137882 (do not build additional stage on compiler paths) - rust-lang#137894 (Revert "store ScalarPair via memset when one side is undef and the other side can be memset") - rust-lang#137902 (Make `ast::TokenKind` more like `lexer::TokenKind`) - rust-lang#137921 (Subtree update of `rust-analyzer`) - rust-lang#137922 (A few cleanups after the removal of `cfg(not(parallel))`) - rust-lang#137939 (fix order on shl impl) - rust-lang#137946 (Fix docker run-local docs) - rust-lang#137955 (Always allow rustdoc-json tests to contain long lines) - rust-lang#137958 (triagebot.toml: Don't label `test/rustdoc-json` as A-rustdoc-search) r? `@ghost` `@rustbot` modify labels: rollup
flip1995
pushed a commit
to flip1995/rust
that referenced
this pull request
Mar 20, 2025
…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#135767 (Future incompatibility warning `unsupported_fn_ptr_calling_conventions`: Also warn in dependencies) - rust-lang#137852 (Remove layouting dead code for non-array SIMD types.) - rust-lang#137863 (Fix pretty printing of unsafe binders) - rust-lang#137882 (do not build additional stage on compiler paths) - rust-lang#137894 (Revert "store ScalarPair via memset when one side is undef and the other side can be memset") - rust-lang#137902 (Make `ast::TokenKind` more like `lexer::TokenKind`) - rust-lang#137921 (Subtree update of `rust-analyzer`) - rust-lang#137922 (A few cleanups after the removal of `cfg(not(parallel))`) - rust-lang#137939 (fix order on shl impl) - rust-lang#137946 (Fix docker run-local docs) - rust-lang#137955 (Always allow rustdoc-json tests to contain long lines) - rust-lang#137958 (triagebot.toml: Don't label `test/rustdoc-json` as A-rustdoc-search) r? `@ghost` `@rustbot` modify labels: rollup
nnethercote
added a commit
to nnethercote/rust
that referenced
this pull request
Apr 17, 2025
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.
PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.
This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
`Delimiter` values to `TokenKind`.
Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
- } else if self.token != token::CloseDelim(Delimiter::Brace) {
+ } else if self.token != token::CloseBrace {
```
nnethercote
added a commit
to nnethercote/rust
that referenced
this pull request
Apr 17, 2025
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.
PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.
This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
`Delimiter` values to `TokenKind`.
Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
- } else if self.token != token::CloseDelim(Delimiter::Brace) {
+ } else if self.token != token::CloseBrace {
```
GuillaumeGomez
pushed a commit
to GuillaumeGomez/rust
that referenced
this pull request
Apr 18, 2025
…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#135767 (Future incompatibility warning `unsupported_fn_ptr_calling_conventions`: Also warn in dependencies) - rust-lang#137852 (Remove layouting dead code for non-array SIMD types.) - rust-lang#137863 (Fix pretty printing of unsafe binders) - rust-lang#137882 (do not build additional stage on compiler paths) - rust-lang#137894 (Revert "store ScalarPair via memset when one side is undef and the other side can be memset") - rust-lang#137902 (Make `ast::TokenKind` more like `lexer::TokenKind`) - rust-lang#137921 (Subtree update of `rust-analyzer`) - rust-lang#137922 (A few cleanups after the removal of `cfg(not(parallel))`) - rust-lang#137939 (fix order on shl impl) - rust-lang#137946 (Fix docker run-local docs) - rust-lang#137955 (Always allow rustdoc-json tests to contain long lines) - rust-lang#137958 (triagebot.toml: Don't label `test/rustdoc-json` as A-rustdoc-search) r? `@ghost` `@rustbot` modify labels: rollup
nnethercote
added a commit
to nnethercote/rust
that referenced
this pull request
Apr 20, 2025
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.
PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.
This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
`Delimiter` values to `TokenKind`.
Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
- } else if self.token != token::CloseDelim(Delimiter::Brace) {
+ } else if self.token != token::CloseBrace {
```
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Apr 22, 2025
… r=petrochenkov
Remove `token::{Open,Close}Delim`
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.
PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.
This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
`Delimiter` values to `TokenKind`.
Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
- } else if self.token != token::CloseDelim(Delimiter::Brace) {
+ } else if self.token != token::CloseBrace {
```
r? `@petrochenkov`
andrew-otiv
pushed a commit
to andrew-otiv/rust
that referenced
this pull request
Apr 22, 2025
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.
PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.
This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
`Delimiter` values to `TokenKind`.
Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
- } else if self.token != token::CloseDelim(Delimiter::Brace) {
+ } else if self.token != token::CloseBrace {
```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is step 2 of rust-lang/compiler-team#831.
r? @spastorino