Recently, a cargo-semver-checks user reached out with a conundrum: cargo-semver-checks reported being unable to build their crate, but when they ran cargo check themselves, it always completed successfully. Something cursed was going on! The fix is part of the upcoming cargo-semver-checks v0.45 release—what better day than Halloween to talk about this ghost story!

Buckle up! I gave an impromptu "Cursed Rust" talk at Rust Forge this year. The audience's excitement about that talk inspired the theme for this post!

The "Not My Fault" rule

In order to check SemVer, cargo-semver-checks has to build the package being checked. Under the hood, it invokes cargo commands—and those commands may sometimes fail. For example, maybe the package currently has compile errors and cannot be built. cargo-semver-checks can't fix this, and needs to report this problem to the user.

This part is extremely important: we must enable the user to independently verify they've run into a genuine issue with their package, and not a cargo-semver-checks bug. To enable this, we produce a "Not my fault" script the user can run in order to observe the issue. The script invokes cargo directly, and does not involve cargo-semver-checks at all.

We have test cases that cover this functionality. Prior to this new release, the output looked something like this:

error: running cargo-doc on crate 'my_crate' failed with output:
-----
 Documenting my_crate v0.1.0 ([PATH]/my_crate)
error: This crate has a compiler error.
 --> [PATH]/my_crate/src/lib.rs:6:1
  |
6 | compile_error!("This crate has a compiler error.");
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error: could not document `my_crate`
-----
error: failed to build rustdoc for crate my_crate v0.1.0
note: this is usually due to a compilation error in the crate,
      and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path ./my_crate &&
          cargo check

The "Not my fault" rule plays two roles: it offers users a better user experience, but also shields the cargo-semver-checks issue tracker from spurious bug reports. Issue triage is extremely expensive as it demands the most scarce resource for an open-source project—maintainer time and energy. Accidentally-invalid ("works as intended, problem is elsewhere") issues are particularly frustrating, because after both investing their time, the reporter and the maintainer simultaneously discover they've been looking at the wrong thing all along.

So far so good! Except...

The build failure reproducer that (sometimes) wasn't

Just one problem: a user reported a case where our generated reproduction didn't actually work. SemVer-checking their crate failed with a build error, but running the "Not my fault" script produced:

    Checking example v1.2.3 (/path/to/example)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.92s

Oof! 😬 Here began the debugging journey... The debugging effort is dramatized to fit the post's narrative structure. It did broadly follow these steps, though they happened in batches and not as a dramatic step-by-step process as this post suggests. But this is a Halloween ghost story, so please indulge me 👻 Debugging is generally easier when users can provide a full repro and offer access to source code. Of course, that's not always an option...

cargo-semver-checks and Rust versions? v0.44.0 (latest) and a Rust version within the range we support.

Was the user performing cross-compilation with the --target attribute? No.

Where was cargo-semver-checks running? In CI, on a common version of Linux.

Where was the user running the "Not my fault" script? Locally, on their laptop, which also runs Linux and has the same versions of cargo-semver-checks and Rust.

Hmm... 🤔

Could the user try running cargo-semver-checks locally on the same machine, just in case?

Oh, it passed? No build failure! Okay, now we're getting somewhere.

To be very safe, could the user verify the Rust version when running rustc --version inside their project directory? "The same Rust is installed" and "the same Rust is invoked when you type in cargo" are not in fact the same statement. I've seen more than one user get burned by running rustup override set ... and then forgetting they've done that. Alas, it was the same Rust this time. It's important to verify these basic assumptions when debugging a complex problem. We don't want to spend 10 hours debugging what feels like a problem in cargo-semver-checks only to find out that the issue was e.g. a difference in rustdoc behaviors between different Rust releases. After all, cargo-semver-checks relies on unstable rustdoc JSON features to work.

I didn't know it at the time, but I was getting warmer.

Is the lockfile checked into the repository? Yes.

The compile error looked like it involved a package dependency. Was there anything special about that dependency? Oh, the package's .cargo/config.toml file uses build.rustflags to set a --cfg flag that the dependency requires?

How very interesting...

If you haven't seen a .cargo/config.toml file before, feel free to give the docs a quick skim. Don't worry, we'll cover all relevant details here.

Builds succeeded from some directories, and failed from others

Here's a "fun" quirk we discovered along the way: running cargo check generally succeeded fine for the user, except if they ran it with --manifest-path. I usually recommend using --manifest-path to simplify running repro steps and avoid fumbling with directory traversals—but here it was proving a problem in itself. Hence, "fun"!

After some discussion, it turns out the user was running my suggested repro scripts by opening a fresh terminal pane, which defaulted to running inside their home directory. Why would that be a problem, you ask? Well...

Where can one place a .cargo/config.toml file and expect cargo to find it?

One place is next to your package or workspace's Cargo.toml file. When you run cargo check inside your project directory, cargo finds the local configuration, even if it needs to walk up the directory tree to find it.

Another place is in your home directory (thankfully, the user didn't have any config there). If cargo finds no configuration local to the project or its ancestor directories, it will check your home directory. Technically, it'll check $CARGO_HOME/config.toml which defaults to $HOME/.cargo/config.toml (Unix) and %USERPROFILE%\.cargo\config.toml (Windows).

Except ... I'm sorry for being pedantic, but the above isn't quite true. You are reading a blog post about a SemVer linter, please don't tell me you are surprised to find pedantic rules-lawyering 😁 If cargo finds no configuration local to the project the current working directory, it starts walking up the filesystem from the working directory. Here's a quote from the cargo documentation:

Cargo allows local configuration for a particular package as well as global configuration. It looks for configuration files in the current directory and all parent directories. If, for example, Cargo were invoked in /projects/foo/bar/baz, then the following configuration files would be probed for and unified in this order:

  • /projects/foo/bar/baz/.cargo/config.toml
  • /projects/foo/bar/.cargo/config.toml
  • /projects/foo/.cargo/config.toml
  • /projects/.cargo/config.toml
  • /.cargo/config.toml
  • $CARGO_HOME/config.toml which defaults to:
    • Windows: %USERPROFILE%\.cargo\config.toml
    • Unix: $HOME/.cargo/config.toml

If your working directory happens to be inside your project when you invoke cargo, this distinction doesn't come into play and the behavior is the same. But you can compile code from anywhere by passing --manifest-path /path/to/Cargo.toml to cargo, which is what our repro scripts were relying on.

Connecting the dots: when cargo was invoked from inside the project directory, it picked up the project's .cargo/config.toml file with the required --cfg flag and compiled successfully. When we invoked it from elsewhere by using --manifest-path to point to the project, it looked in the current directory for config, didn't find any, and produced a compile error because the required --cfg flag wasn't set.

If you want to experience this yourself, I set up a cargo-semver-checks test case repository with this edge case and several others.

Eventually, this turned out to be something of a red herring: the user was invoking cargo-semver-checks from inside the project directory, so the .cargo/config.toml could have been picked up anyway.

But there was something interesting about the compilation error when running cargo check from a different directory: it was identical to what cargo-semver-checks itself was getting from inside the project directory. Was the --cfg parameter still not being passed for some reason, despite being configured in .config/cargo.toml?

cargo doc and cargo check are not the same

Earlier, we said that in order to check SemVer, cargo-semver-checks has to build the package being checked.

Strictly speaking, this is not accurate: we need to document the package, not build it. We run cargo doc to generate rustdoc JSON, not cargo check to generate binary files.

More pointless pedantry? Apparently not! The difference matters quite a bit, as it happens.

cargo allows users to have precise control over the configuration used with their code. cargo check and cargo build interact with rustc, so they (among other things) read the RUSTFLAGS environment variable and the build.rustflags config file value.

cargo doc runs rustdoc, not rustc. Bah, pedantry! Under the hood, rustdoc uses the same machinery as rustc!

Except that means cargo doc doesn't use RUSTFLAGS and build.rustflags, but RUSTDOCFLAGS and build.rustdocflags instead!

The user's .cargo/config.toml file set the required --cfg flag in build.rustflags. It said nothing about build.rustdocflags! So cargo check would pass, and cargo doc would fail due to missing the required --cfg flag.

Bingo 🎯

Resulting improvements in cargo-semver-checks

We shipped both correctness and UX improvements as a result of this work.

On the correctness side, cargo-semver-checks now replicates the cargo configuration search process, detecting and passing on any RUSTFLAGS or RUSTDOCFLAGS it encounters instead of merely overwriting them with its own configuration. Doing this correctly is far from simple: those values can be specified in any of 6 different environment variables or 5 different config keys, including in a target-specific manner that requires cargo-semver-checks to know which target triple it's linting. All this is backed up by 28 new test cases checking various cursed crate and configuration combinations.

On the UX side, our Not my fault scripts are now slightly more clever: they suggest running both cargo check and cargo doc, as well as including a --target flag when one is necessary to select the correct target triple.

Maintaining cargo-semver-checks is a repeated series of examples for a variation of Hofstadter's law: The world is more complex than you think, even when you take into account (this variation of) Hofstadter's law. But that's okay! Every new release is a step forward, and a chance to do something better than before. Given time, patience, funding, egregious numbers of test cases, and my stubbornness about solving problems that get in my way, this mountain will also be topped.

Happy Halloween! 🎃

If you liked this essay, consider subscribing to my blog or following me on Mastodon, Bluesky, or Twitter/X. You can also fund my writing and work on cargo-semver-checks via GitHub Sponsors, for which I'd be most grateful ❤

Discuss on r/rust or lobste.rs.