Add support for doctests

[sunshowers]

Currently, nextest doesn't support Rust doctests. This is because doctests are not exposed in stable Rust the way regular test binaries are, and are instead treated as special by cargo test.

One nightly-only fix that might work is to collect doctest executables through -Z unstable-options --persist-doctests. However, this isn't a stable approach so it must be used with care.

Note: You can run cargo test --doc as a separate step from cargo nextest run. This will not incur a performance penalty: cargo nextest run && cargo test --doc will not cause any more builds than a plain cargo test that runs doctests.

[xd009642]

So I use the --persist-doctests attribute and can share some of the issues I've faced with it, (hopefully more people using it will lead to stabilisation and improvements). So if we have a doctest with the should_panic attribute cargo test compiles and runs it and checks if the return code is non-zero, and the test passes only if it's non-zero.

However, the doctest file name is made from a normalisation of the path (all path separators replaced with _, a line number and a number to ensure the files are unique. This is because a doctest on the same line in src/foo/bar.rs and src/foo_bar.rs would otherwise generate the same filename. The line numbers are also done after macro expansion so things like include_str can make it harder to match line numbers with attributes.

I made an issue in rust about bundling attribute information into the file name to make it easier for people writing tools using this feature rust-lang/rust#92597 and if you're interested in seeing my code it's in https://github.com./xd009642/tarpaulin in src/cargo.rs, it's reasonably stable the issues mentioned above are the only parts it can fail. I've had this since the nightly flag was created and the naming convention has only changed once so I've only had one completely unexpected issue that affected all users that cause persist-doctests to be added.

Hope this is helpful, nextest looks pretty nifty 🎉

[WaffleLapkin]

There are also --test-builder/--runtool unstable flags that are used by miri to run doc tests

[nu11ptr]

Can I suggest that until this is added this is called out in the docs in a visible way? I recently was made aware how tricky doctests are to handle, but other users might not be aware. Your crate seems quite nice based on my limited usage, and it would be a shame if they didn't give it a fair chance because they thought it was broken and "skipping tests".

[sunshowers]

@nu11ptr good idea -- I've added a note to the homepage along with expanding the notes on the listing and running pages f4043b1

[tgross35]

Rustdoc may gain the unstable flag --extract-doctests which will provide a way to list doctests with their attributes and contents. Is this something that nextest could make use of? rust-lang/rust#134531

[tgross35]

Separately, would it be possible for nextest to just run cargo test --doc itself after running other tests? Even without the nice UI it would eliminate the footgun, as well being less to remember/type when running locally.

Of course --extract-doctests or similar would make this unnecessary, but it seems like an easy stopgap until that is usable on stable. (For the record, I would still be quite happy if nextest can run doctests directly only on nightly).

[aDotInTheVoid]

Is this something that nextest could make use of? rust-lang/rust#134531

From a rustdoc perspecive, we defiantly want nextest to be able to use this, and if it can't that's a flaw with rustdoc's thing, and we'd love to hear about this. Ideally, I'd love to see nextest use this (as an experimental feature) before it gets stabilized, to give us some confidence it's broadly useful

[epage]

Cargo wants to have --doc flags on every build command (cargo check --doc) to build doc tests. I'd recommend playing with --extract-doctests inside of cargo to add support for --doc so it can generate the doctest binaries for you with build caching and correct linking (which was brittle when people tried to do this for READMEs in the past).

[sunshowers]

This would be really nice to have -- I don't expect to have any time to work on it soon, but some assistance (maybe a prototype within nextest) would be greatly appreciated.