Add citool command for generating a test dashboard #139978

Kobzol · 2025-04-17T15:39:57Z

This PR implements an initial version of a test suite dashboard, which shows which tests were executed on CI and which tests were ignored. This was discussed here. The dashboard is still quite bare-bones, but I think that it could already be useful.

The next step is to create a job index, similarly to the post-merge report, and link from the individual tests to the job that executed them.

You can try it locally like this:

$ cargo run --manifest-path src/ci/citool/Cargo.toml --release \
    -- test-dashboard 38c560ae681d5c0d3fd615eaedc537a282fb1086 --output-dir dashboard

and then open dashboard/index.html in a web browser.

CC @wesleywiser

r? @jieyouxu

jieyouxu · 2025-04-18T08:45:22Z

FTR, the dashboard currently looks like:

jieyouxu

Thanks for working on this! The impl looks good, played around with the generated dashboard locally and the initial version's content looks like what I'd expect.

src/ci/citool/Cargo.toml

src/ci/citool/src/main.rs

jieyouxu · 2025-04-18T09:01:43Z

An example test (edited test path to be from tests/):

tests/codegen/autodiff/scalar.rs (passed: 0, ignored: 23)

That seems a bit surprising -- passed: 0 looks right, I definitely don't satisfy //@ needs-enzyme locally. However the ignored: 23 seems a bit high -- this is a codegen test that doesn't use any revisions 🤔

Or for instance,

tests/assembly/compiletest-self-test/use-minicore-no-run.rs (passed: 24, ignored: 0)

That passed: 24 seems too high?

Or what do these actually indicate?

Kobzol · 2025-04-18T09:13:40Z

The number is a cartesian product of stage x target x CI job combinations where the test was executed or ignored. In most cases, there is 1 stage and 1 target per CI job, so it's more or less the number of CI jobs that ran the test.

jieyouxu · 2025-04-18T09:14:18Z

Hm right, it's the coalesced counts.

jieyouxu · 2025-04-18T09:31:26Z

The number is a cartesian product of stage x target x CI job combinations where the test was executed or ignored. In most cases, there is 1 stage and 1 target per CI job, so it's more or less the number of CI jobs that ran the test.

Should we maybe add this as like a banner thing / or include this remark somehow on the page, because I imagine I won't be the first one to ask this question :D

Kobzol · 2025-04-18T09:37:38Z

Yeah, I would like to extend the amount of information on the page, but the problem is that with 20k tests every small amount of HTML per test is gonna add up quickly 😆 The page already has 2 MiB. I wanted to land this as an initial version and then do incremental improvements in follow-up PRs, to avoid having too much stuff at once in a PR.

jieyouxu · 2025-04-18T09:43:54Z

Oh I just mean an overall note, like

Here's how to interpret the "passed" and "ignored" counts: they are coalsced between stage x target x CI jobs where the test was executed or ignored.

Not like, individual per-test remarks, if that makes sense?

EDIT: with some inspect element gaming, I mean

jieyouxu · 2025-04-18T09:45:58Z

But in any case I don't think it's worth blocking over that, so feel free to r=me with or without the remark.

jieyouxu · 2025-04-18T10:28:23Z

Thanks!
@bors r+ rollup

bors · 2025-04-18T10:28:26Z

📌 Commit cecf167 has been approved by jieyouxu

It is now in the queue for this repository.

Kobzol · 2025-04-18T10:30:29Z

I was just about to add the note =D I also noticed one thing I want to fix.

@bors r-

`assembly/asm` contained a test named `asm/aarch64-el2vmsa.rs`, while it should have been only `arch64-el2vmsa.rs`.

Kobzol · 2025-04-18T10:45:53Z

Added the note and removed duplicated subdirectory name in the name of nested tests.

jieyouxu

Looks good, thanks. r=me after PR CI is 🍏

Kobzol · 2025-04-18T11:58:12Z

@bors r=jieyouxu rollup

bors · 2025-04-18T11:58:14Z

📌 Commit b18e373 has been approved by jieyouxu

It is now in the queue for this repository.

Add citool command for generating a test dashboard This PR implements an initial version of a test suite dashboard, which shows which tests were executed on CI and which tests were ignored. This was discussed [here](https://rust-lang.zulipchat.com/#narrow/channel/238009-t-compiler.2Fmeetings/topic/.5Bsteering.5D.202025-04-11.20Dealing.20with.20disabled.20tests/with/512799036). The dashboard is still quite bare-bones, but I think that it could already be useful. The next step is to create a job index, similarly to the post-merge report, and link from the individual tests to the job that executed them. You can try it locally like this: ```bash $ cargo run --manifest-path src/ci/citool/Cargo.toml --release \ -- test-dashboard 38c560a --output-dir dashboard ``` and then open `dashboard/index.html` in a web browser. CC `@wesleywiser` r? `@jieyouxu`

…iaskrgr Rollup of 8 pull requests Successful merges: - rust-lang#137454 (not lint break with label and unsafe block) - rust-lang#138934 (support config extensions) - rust-lang#139297 (Deduplicate & clean up Nix shell) - rust-lang#139834 (Don't canonicalize crate paths) - rust-lang#139868 (Move `pal::env` into `std::sys::env_consts`) - rust-lang#139978 (Add citool command for generating a test dashboard) - rust-lang#140000 (skip llvm-config in autodiff check builds, when its unavailable) - rust-lang#140007 (Disable has_thread_local on i686-win7-windows-msvc) r? `@ghost` `@rustbot` modify labels: rollup

Add citool command for generating a test dashboard This PR implements an initial version of a test suite dashboard, which shows which tests were executed on CI and which tests were ignored. This was discussed [here](https://rust-lang.zulipchat.com/#narrow/channel/238009-t-compiler.2Fmeetings/topic/.5Bsteering.5D.202025-04-11.20Dealing.20with.20disabled.20tests/with/512799036). The dashboard is still quite bare-bones, but I think that it could already be useful. The next step is to create a job index, similarly to the post-merge report, and link from the individual tests to the job that executed them. You can try it locally like this: ```bash $ cargo run --manifest-path src/ci/citool/Cargo.toml --release \ -- test-dashboard 38c560a --output-dir dashboard ``` and then open `dashboard/index.html` in a web browser. CC ``@wesleywiser`` r? ``@jieyouxu``

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#137454 (not lint break with label and unsafe block) - rust-lang#138934 (support config extensions) - rust-lang#139297 (Deduplicate & clean up Nix shell) - rust-lang#139834 (Don't canonicalize crate paths) - rust-lang#139978 (Add citool command for generating a test dashboard) - rust-lang#140000 (skip llvm-config in autodiff check builds, when its unavailable) - rust-lang#140007 (Disable has_thread_local on i686-win7-windows-msvc) r? `@ghost` `@rustbot` modify labels: rollup

Add citool command for generating a test dashboard This PR implements an initial version of a test suite dashboard, which shows which tests were executed on CI and which tests were ignored. This was discussed [here](https://rust-lang.zulipchat.com/#narrow/channel/238009-t-compiler.2Fmeetings/topic/.5Bsteering.5D.202025-04-11.20Dealing.20with.20disabled.20tests/with/512799036). The dashboard is still quite bare-bones, but I think that it could already be useful. The next step is to create a job index, similarly to the post-merge report, and link from the individual tests to the job that executed them. You can try it locally like this: ```bash $ cargo run --manifest-path src/ci/citool/Cargo.toml --release \ -- test-dashboard 38c560a --output-dir dashboard ``` and then open `dashboard/index.html` in a web browser. CC ```@wesleywiser``` r? ```@jieyouxu```

Rollup of 9 pull requests Successful merges: - rust-lang#137454 (not lint break with label and unsafe block) - rust-lang#139297 (Deduplicate & clean up Nix shell) - rust-lang#139535 (Implement `Default` for raw pointers) - rust-lang#139753 (Make `#[naked]` an unsafe attribute) - rust-lang#139922 (fix incorrect type in cstr `to_string_lossy()` docs) - rust-lang#139978 (Add citool command for generating a test dashboard) - rust-lang#140007 (Disable has_thread_local on i686-win7-windows-msvc) - rust-lang#140016 (std: Use fstatat() on illumos) - rust-lang#140025 (Re-remove `AdtFlags::IS_ANONYMOUS`) r? `@ghost` `@rustbot` modify labels: rollup

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#137454 (not lint break with label and unsafe block) - rust-lang#138934 (support config extensions) - rust-lang#139297 (Deduplicate & clean up Nix shell) - rust-lang#139834 (Don't canonicalize crate paths) - rust-lang#139978 (Add citool command for generating a test dashboard) - rust-lang#140000 (skip llvm-config in autodiff check builds, when its unavailable) - rust-lang#140007 (Disable has_thread_local on i686-win7-windows-msvc) r? `@ghost` `@rustbot` modify labels: rollup

…enton Rollup of 8 pull requests Successful merges: - rust-lang#138934 (support config extensions) - rust-lang#139091 (Rewrite on_unimplemented format string parser.) - rust-lang#139753 (Make `#[naked]` an unsafe attribute) - rust-lang#139762 (Don't assemble non-env/bound candidates if projection is rigid) - rust-lang#139834 (Don't canonicalize crate paths) - rust-lang#139868 (Move `pal::env` to `std::sys::env_consts`) - rust-lang#139978 (Add citool command for generating a test dashboard) - rust-lang#139995 (Clean UI tests 4 of n) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#139978 - Kobzol:ci-test-summary, r=jieyouxu Add citool command for generating a test dashboard This PR implements an initial version of a test suite dashboard, which shows which tests were executed on CI and which tests were ignored. This was discussed [here](https://rust-lang.zulipchat.com/#narrow/channel/238009-t-compiler.2Fmeetings/topic/.5Bsteering.5D.202025-04-11.20Dealing.20with.20disabled.20tests/with/512799036). The dashboard is still quite bare-bones, but I think that it could already be useful. The next step is to create a job index, similarly to the post-merge report, and link from the individual tests to the job that executed them. You can try it locally like this: ```bash $ cargo run --manifest-path src/ci/citool/Cargo.toml --release \ -- test-dashboard 38c560a --output-dir dashboard ``` and then open `dashboard/index.html` in a web browser. CC ````@wesleywiser```` r? ````@jieyouxu````

…enton Rollup of 8 pull requests Successful merges: - rust-lang#138934 (support config extensions) - rust-lang#139091 (Rewrite on_unimplemented format string parser.) - rust-lang#139753 (Make `#[naked]` an unsafe attribute) - rust-lang#139762 (Don't assemble non-env/bound candidates if projection is rigid) - rust-lang#139834 (Don't canonicalize crate paths) - rust-lang#139868 (Move `pal::env` to `std::sys::env_consts`) - rust-lang#139978 (Add citool command for generating a test dashboard) - rust-lang#139995 (Clean UI tests 4 of n) r? `@ghost` `@rustbot` modify labels: rollup

Kobzol added 10 commits April 17, 2025 16:18

Make parent in download_auto_job_metrics optional

d14df26

Extract function for normalizing path delimiters to utils

111c15c

Add command to citool for generating a test dashboard

c8a882b

Add buttons for expanding and collapsing all test suites

a326afd

Add a note about how to find tests that haven't been executed anywhere.

4b31033

Render test revisions separately

1a6e0d5

Create a macro for rendering test results

d2c1763

Print number of root tests and subdirectories

aa9cb70

Turn test_dashboard into a file

08cb187

Add a note about the test dashboard to the post-merge report

cecf167

rustbot assigned jieyouxu Apr 17, 2025

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Apr 17, 2025

Kobzol changed the title ~~Ci test summary~~ Add citool command for generating a test dashboard Apr 17, 2025

jieyouxu approved these changes Apr 18, 2025

View reviewed changes

src/ci/citool/Cargo.toml Show resolved Hide resolved

src/ci/citool/src/main.rs Show resolved Hide resolved

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 18, 2025

bors removed the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Apr 18, 2025

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Apr 18, 2025

Kobzol added 2 commits April 18, 2025 12:32

Add a note that explains the counts

65ce38a

Reduce duplicated test prefixes in nested subdirectories

b18e373

`assembly/asm` contained a test named `asm/aarch64-el2vmsa.rs`, while it should have been only `arch64-el2vmsa.rs`.

jieyouxu approved these changes Apr 18, 2025

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 18, 2025

matthiaskrgr mentioned this pull request Apr 18, 2025

Rollup of 8 pull requests #140012

Closed

matthiaskrgr mentioned this pull request Apr 18, 2025

Rollup of 7 pull requests #140017

Closed

jhpratt mentioned this pull request Apr 19, 2025

Rollup of 9 pull requests #140027

Closed

ChrisDenton mentioned this pull request Apr 19, 2025

Rollup of 8 pull requests #140043

Merged

bors merged commit 56cef5c into rust-lang:master Apr 20, 2025
6 checks passed

rustbot added this to the 1.88.0 milestone Apr 20, 2025

Kobzol deleted the ci-test-summary branch April 20, 2025 05:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add citool command for generating a test dashboard #139978

Add citool command for generating a test dashboard #139978

Kobzol commented Apr 17, 2025 •

edited

Loading

jieyouxu commented Apr 18, 2025

jieyouxu left a comment

jieyouxu commented Apr 18, 2025 •

edited

Loading

Kobzol commented Apr 18, 2025

jieyouxu commented Apr 18, 2025 •

edited

Loading

jieyouxu commented Apr 18, 2025 •

edited

Loading

Kobzol commented Apr 18, 2025

jieyouxu commented Apr 18, 2025 •

edited

Loading

jieyouxu commented Apr 18, 2025

jieyouxu commented Apr 18, 2025

bors commented Apr 18, 2025

Kobzol commented Apr 18, 2025

Kobzol commented Apr 18, 2025

jieyouxu left a comment

Kobzol commented Apr 18, 2025

bors commented Apr 18, 2025

Add citool command for generating a test dashboard #139978

Add citool command for generating a test dashboard #139978

Conversation

Kobzol commented Apr 17, 2025 • edited Loading

jieyouxu commented Apr 18, 2025

jieyouxu left a comment

Choose a reason for hiding this comment

jieyouxu commented Apr 18, 2025 • edited Loading

Kobzol commented Apr 18, 2025

jieyouxu commented Apr 18, 2025 • edited Loading

jieyouxu commented Apr 18, 2025 • edited Loading

Kobzol commented Apr 18, 2025

jieyouxu commented Apr 18, 2025 • edited Loading

jieyouxu commented Apr 18, 2025

jieyouxu commented Apr 18, 2025

bors commented Apr 18, 2025

Kobzol commented Apr 18, 2025

Kobzol commented Apr 18, 2025

jieyouxu left a comment

Choose a reason for hiding this comment

Kobzol commented Apr 18, 2025

bors commented Apr 18, 2025

Kobzol commented Apr 17, 2025 •

edited

Loading

jieyouxu commented Apr 18, 2025 •

edited

Loading

jieyouxu commented Apr 18, 2025 •

edited

Loading

jieyouxu commented Apr 18, 2025 •

edited

Loading

jieyouxu commented Apr 18, 2025 •

edited

Loading