Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote · 2024-04-18T23:28:21Z

A third attempt at this; the first attempt was #96724 and the second was #114647.

rustbot · 2024-04-18T23:28:25Z

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

ijackson · 2024-05-10T15:28:11Z

❤️ @nnethercote for working on this. Thank you! I'm not sure if there's a way for me to help, as someone who doesn't really know much about the compiler innards, but please LMK if you think of something.

nnethercote · 2024-05-10T20:45:22Z

@ijackson: thanks! I'm curious why you are interested in this change, given that it's a compiler internals rearrangement?

nnethercote · 2024-05-10T20:47:31Z

@ijackson: Oh, I see, you are interested in #67062 being fixed. Unfortunately my current thoughts are that this PR alone won't be enough to fix that issue, though it's a necessary stepping stone.

ijackson · 2024-05-10T22:40:41Z

@ijackson: Oh, I see, you are interested in #67062 being fixed. Unfortunately my current thoughts are that this PR alone won't be enough to fix that issue, though it's a necessary stepping stone.

Right. It seems ... quite nontrivial. So, thanks.

oriongonza · 2024-05-12T21:49:53Z

After this is done TokenKind will become Copy right?

nnethercote · 2024-05-12T22:35:11Z

After this is done TokenKind will become Copy right?

Yes.

Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for the `c2!` macro. - tests/ui/macros/trace_faulty_macros.rs: there is some sub-optimal spacing in the printing of `A { a : a, b : 0, c : _, .. }`, which will be fixed in the next commit. The spacing of `1+1` improves -- it now matches the formatting in the source code. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.

…, r=<try> Print `token::Interpolated` with token stream pretty printing. This is a step towards removing `token::Interpolated` (rust-lang#124141). It unavoidably changes the output of the `stringify!` macro, generally for the better. r? `@petrochenkov`

nnethercote · 2024-05-17T06:47:27Z

#125174 carves off a piece of this PR so it can be merged separately.

Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com./rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`

Rollup merge of rust-lang#125236 - dtolnay:expandtest, r=nnethercote Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com./rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`

Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for all the `c2*` macros. The AST pretty printer now has more thorough testing thanks to rust-lang#125236. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.

petrochenkov · 2024-05-23T20:01:24Z

It's great to see that enum InvisibleOrigin allows to migrate the parser to delimited groups relatively simply, with just the maybe_whole to maybe_reparse_metavar_seq replacement.

Of course it prevents a lot of interesting stuff like reparsing expr as pat and similar, like it would work in a purely token-based model, but all that can be carefully introduced later, when it's possible to do backward compatibly.

petrochenkov · 2024-05-23T20:02:23Z

How hard would it be to get this to a perf run?
(With or without the NtExpr/NtLiteral stuff.)

petrochenkov · 2024-05-23T20:03:04Z

Blocked on #125174.
@rustbot blocked

rustbot · 2025-04-02T05:29:38Z

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred in compiler/rustc_sanitizers

cc @rust-lang/project-exploit-mitigations, @rcvalle

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

nnethercote · 2025-04-02T05:33:00Z

#133436, #137517, #138083, and #138478 have all been merged, removing all Nonterminal variants except for NtBlock. The remaining commits in this PR now remove NtBlock plus some cleaning up.

@bors try @rust-timer queue

bors · 2025-04-02T05:34:12Z

⌛ Trying commit 1830245 with merge df3277b...

…Kind-Interpolated, r=<try> Remove `Nonterminal` and `TokenKind::Interpolated` A third attempt at this; the first attempt was rust-lang#96724 and the second was rust-lang#114647. r? `@ghost`

bors · 2025-04-02T07:40:55Z

☀️ Try build successful - checks-actions
Build commit: df3277b (df3277bc867c119b5867d21eb2431db22e13c439)

rust-timer · 2025-04-02T09:21:02Z

Finished benchmarking commit (df3277b): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	1.1%	[0.3%, 2.0%]	10
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.5%	[-1.8%, -0.1%]	65
Improvements ✅ (secondary)	-0.7%	[-2.5%, -0.1%]	48
All ❌✅ (primary)	-0.3%	[-1.8%, 2.0%]	75

Max RSS (memory usage)

Results (primary 0.8%, secondary 4.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.8%	[1.6%, 2.1%]	2
Regressions ❌ (secondary)	4.0%	[3.0%, 4.9%]	2
Improvements ✅ (primary)	-1.2%	[-1.2%, -1.2%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.8%	[-1.2%, 2.1%]	3

Cycles

Results (primary -0.0%, secondary 7.6%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.3%	[1.3%, 1.3%]	1
Regressions ❌ (secondary)	7.6%	[4.0%, 11.2%]	10
Improvements ✅ (primary)	-1.3%	[-1.3%, -1.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.0%	[-1.3%, 1.3%]	2

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 774.398s -> 774.46s (0.01%)
Artifact size: 365.94 MiB -> 365.54 MiB (-0.11%)

compiler/rustc_ast/src/tokenstream.rs

petrochenkov · 2025-04-02T12:52:54Z

r=me after answering #124141 (comment).
@rustbot author

rustbot · 2025-04-02T12:52:56Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

nnethercote · 2025-04-02T23:29:35Z

I will wait a bit before merging this. I want to see if there is any fallout from #138478 first, because that was a big and complicated change.

nnethercote · 2025-04-13T22:14:03Z

I have fixed a few minor problems with the NtExpr removal, I think this is ok to go now.

@bors r=petrochenkov

nnethercote · 2025-04-14T03:07:05Z

@bors r=petrochenkov

bors · 2025-04-14T03:07:07Z

📌 Commit 1830245 has been approved by petrochenkov

It is now in the queue for this repository.

bors · 2025-04-14T03:56:59Z

⌛ Testing commit 1830245 with merge f836ae4...

bors · 2025-04-14T07:06:46Z

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing f836ae4 to master...

github-actions · 2025-04-14T07:08:28Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 15f58c4 (parent) -> f836ae4 (this PR)

Test differences

Show 14 test diffs

Additionally, 14 doctest diffs were found. These are ignored, as they are noisy.

Job group index

Job duration changes

x86_64-apple-2: 4712.2s -> 6406.2s (36.0%)
dist-x86_64-apple: 7203.2s -> 8743.1s (21.4%)
dist-apple-various: 6106.7s -> 7268.0s (19.0%)
dist-aarch64-apple: 5768.9s -> 4724.0s (-18.1%)
dist-x86_64-netbsd: 5856.7s -> 5003.6s (-14.6%)
x86_64-apple-1: 9425.3s -> 8687.7s (-7.8%)
dist-armv7-linux: 5262.9s -> 5598.6s (6.4%)
x86_64-msvc-ext3: 7580.4s -> 7985.6s (5.3%)
dist-aarch64-linux: 5511.4s -> 5225.7s (-5.2%)
x86_64-msvc-ext2: 5866.5s -> 6110.3s (4.2%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2025-04-14T08:23:04Z

Finished benchmarking commit (f836ae4): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

If the regression was expected or you think it can be justified,
please write a comment with sufficient written justification, and add
@rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
If you think that you know of a way to resolve the regression, try to create
a new PR with a fix for the regression.
If you do not understand the regression or you think that it is just noise,
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	1.0%	[0.3%, 2.1%]	11
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.5%	[-1.4%, -0.2%]	48
Improvements ✅ (secondary)	-1.0%	[-2.4%, -0.3%]	36
All ❌✅ (primary)	-0.2%	[-1.4%, 2.1%]	59

Max RSS (memory usage)

Results (secondary -1.5%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.5%	[0.9%, 2.0%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-7.5%	[-7.5%, -7.5%]	1
All ❌✅ (primary)	-	-	0

Cycles

Results (primary -1.3%, secondary 3.5%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	5.9%	[1.6%, 9.7%]	12
Improvements ✅ (primary)	-1.3%	[-1.3%, -1.3%]	1
Improvements ✅ (secondary)	-3.8%	[-4.5%, -2.2%]	4
All ❌✅ (primary)	-1.3%	[-1.3%, -1.3%]	1

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 780.383s -> 779.942s (-0.06%)
Artifact size: 365.54 MiB -> 365.12 MiB (-0.11%)

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 18, 2024

nnethercote marked this pull request as draft April 18, 2024 23:28

nnethercote mentioned this pull request Apr 18, 2024

Start removing Nonterminal #114647

Closed

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch 2 times, most recently from 42a623a to c133e16 Compare April 23, 2024 03:51

petrochenkov self-assigned this Apr 28, 2024

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch from c133e16 to 7aef5db Compare May 16, 2024 10:50

This comment has been minimized.

Sign in to view

nnethercote mentioned this pull request May 17, 2024

Print token::Interpolated with token stream pretty printing. #125174

Merged

This comment was marked as resolved.

Sign in to view

dtolnay mentioned this pull request May 18, 2024

Add tests for -Zunpretty=expanded ported from stringify's tests #125236

Merged

rustbot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 23, 2024

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 2, 2025

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 2, 2025

petrochenkov reviewed Apr 2, 2025

View reviewed changes

compiler/rustc_ast/src/tokenstream.rs Show resolved Hide resolved

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 14, 2025

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 14, 2025

bors merged commit f836ae4 into rust-lang:master Apr 14, 2025
7 checks passed

rustbot added this to the 1.88.0 milestone Apr 14, 2025

This was referenced Apr 14, 2025

Include whitespace in "remove |" suggestion and make it hidden #137872

Open

Actually ensure that TokenKind's fields are safely copyable #139743

Closed

Mark-Simulacrum added the perf-regression-triaged The performance regression has been triaged. label Apr 14, 2025

nnethercote deleted the rm-Nonterminal-and-TokenKind-Interpolated branch April 15, 2025 06:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote commented Apr 18, 2024 •

edited

Loading

rustbot commented Apr 18, 2024

ijackson commented May 10, 2024

nnethercote commented May 10, 2024

nnethercote commented May 10, 2024

ijackson commented May 10, 2024

oriongonza commented May 12, 2024

nnethercote commented May 12, 2024

This comment has been minimized.

nnethercote commented May 17, 2024 •

edited

Loading

This comment was marked as resolved.

petrochenkov commented May 23, 2024 •

edited

Loading

petrochenkov commented May 23, 2024

petrochenkov commented May 23, 2024

rustbot commented Apr 2, 2025

nnethercote commented Apr 2, 2025

This comment has been minimized.

bors commented Apr 2, 2025

bors commented Apr 2, 2025

This comment has been minimized.

rust-timer commented Apr 2, 2025

petrochenkov commented Apr 2, 2025

rustbot commented Apr 2, 2025

nnethercote commented Apr 2, 2025

nnethercote commented Apr 13, 2025 •

edited

Loading

nnethercote commented Apr 14, 2025

bors commented Apr 14, 2025

bors commented Apr 14, 2025

bors commented Apr 14, 2025

github-actions bot commented Apr 14, 2025

rust-timer commented Apr 14, 2025

Remove Nonterminal and TokenKind::Interpolated #124141

Remove Nonterminal and TokenKind::Interpolated #124141

Conversation

nnethercote commented Apr 18, 2024 • edited Loading

rustbot commented Apr 18, 2024

ijackson commented May 10, 2024

nnethercote commented May 10, 2024

nnethercote commented May 10, 2024

ijackson commented May 10, 2024

oriongonza commented May 12, 2024

nnethercote commented May 12, 2024

This comment has been minimized.

nnethercote commented May 17, 2024 • edited Loading

This comment was marked as resolved.

petrochenkov commented May 23, 2024 • edited Loading

petrochenkov commented May 23, 2024

petrochenkov commented May 23, 2024

rustbot commented Apr 2, 2025

nnethercote commented Apr 2, 2025

This comment has been minimized.

bors commented Apr 2, 2025

bors commented Apr 2, 2025

This comment has been minimized.

rust-timer commented Apr 2, 2025

Overall result: ❌✅ regressions and improvements - please read the text below

petrochenkov commented Apr 2, 2025

rustbot commented Apr 2, 2025

nnethercote commented Apr 2, 2025

nnethercote commented Apr 13, 2025 • edited Loading

nnethercote commented Apr 14, 2025

bors commented Apr 14, 2025

bors commented Apr 14, 2025

bors commented Apr 14, 2025

github-actions bot commented Apr 14, 2025

Test differences

Job duration changes

rust-timer commented Apr 14, 2025

Overall result: ❌✅ regressions and improvements - please read the text below

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote commented Apr 18, 2024 •

edited

Loading

nnethercote commented May 17, 2024 •

edited

Loading

petrochenkov commented May 23, 2024 •

edited

Loading

nnethercote commented Apr 13, 2025 •

edited

Loading