# Ghosts in the Compilation _Published: 2025-10-31_ *Recently, a `cargo-semver-checks` user reached out with a conundrum: `cargo-semver-checks` reported being unable to build their crate, but when they ran `cargo check` themselves, it always completed successfully. Something cursed was going on! The fix is part of the upcoming `cargo-semver-checks v0.45` release—what better day than Halloween to talk about this ghost story!* Buckle up![^sn-1] - [The "Not My Fault" rule](#the-not-my-fault-rule) - [The build failure reproducer that (sometimes) wasn't](#the-build-failure-reproducer-that-sometimes-wasn-t) - [Builds succeeded from some directories, and failed from others](#builds-succeeded-from-some-directories-and-failed-from-others) - [`cargo doc` and `cargo check` are not the same](#cargo-doc-and-cargo-check-are-not-the-same) - [Resulting improvements in `cargo-semver-checks`](#resulting-improvements-in-cargo-semver-checks) ## The "Not My Fault" rule In order to check SemVer, `cargo-semver-checks` has to build the package being checked. Under the hood, it invokes `cargo` commands—and those commands may sometimes fail. For example, maybe the package currently has compile errors and cannot be built. `cargo-semver-checks` can't fix this, and needs to report this problem to the user. This part is extremely important: we must enable the user to independently verify they've run into a genuine issue with their package, and not a `cargo-semver-checks` bug. To enable this, we produce a **"Not my fault"** script the user can run in order to observe the issue. The script invokes `cargo` directly, and does not involve `cargo-semver-checks` at all. We have [test cases that](https://github.com/obi1kenobi/cargo-semver-checks/blob/315abc3a0e1845cbbdc6df0e7d301f05b77d645f/test_outputs/snapshot_tests/cargo_semver_checks__snapshot_tests__workspace_baseline_compile_error-output.snap) [cover this functionality](https://github.com/obi1kenobi/cargo-semver-checks/blob/main/test_outputs/snapshot_tests/cargo_semver_checks__snapshot_tests__workspace_baseline_conditional_compile_error-output.snap). Prior to this new release, the output looked something like this: ``` error: running cargo-doc on crate 'my_crate' failed with output: ----- Documenting my_crate v0.1.0 ([PATH]/my_crate) error: This crate has a compiler error. --> [PATH]/my_crate/src/lib.rs:6:1 | 6 | compile_error!("This crate has a compiler error."); | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ error: could not document `my_crate` ----- error: failed to build rustdoc for crate my_crate v0.1.0 note: this is usually due to a compilation error in the crate, and is unlikely to be a bug in cargo-semver-checks note: the following command can be used to reproduce the error: cargo new --lib example && cd example && echo '[workspace]' >> Cargo.toml && cargo add --path ./my_crate && cargo check ``` The "Not my fault" rule plays two roles: it offers users a better user experience, but also shields the `cargo-semver-checks` issue tracker from spurious bug reports. Issue triage is extremely expensive as it demands the most scarce resource for an open-source project—maintainer time and energy. Accidentally-invalid ("works as intended, problem is elsewhere") issues are particularly frustrating, because after both investing their time, the reporter and the maintainer simultaneously discover they've been looking at the wrong thing all along. So far so good! Except... ## The build failure reproducer that (sometimes) wasn't Just one problem: a user reported a case where our generated reproduction didn't actually work. SemVer-checking their crate failed with a build error, but running the "Not my fault" script produced: ``` Checking example v1.2.3 (/path/to/example) Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.92s ``` Oof! 😬 Here began the debugging journey...[^sn-2][^sn-3] `cargo-semver-checks` and Rust versions? v0.44.0 (latest) and a Rust version within the range we support. Was the user performing cross-compilation with the `--target` attribute? No. Where was `cargo-semver-checks` running? In CI, on a common version of Linux. Where was the user running the "Not my fault" script? Locally, on their laptop, which also runs Linux and has the same versions of `cargo-semver-checks` and Rust. *Hmm...* 🤔 Could the user try running `cargo-semver-checks` locally on the same machine, just in case? Oh, it *passed*? No build failure! Okay, now we're getting somewhere. To be very safe, could the user verify the Rust version when running `rustc --version` *inside* their project directory? "The same Rust is installed" and "the same Rust is invoked when you type in `cargo`" are not in fact the same statement. I've seen more than one user get burned by running `rustup override set ...` and then forgetting they've done that. Alas, it was the same Rust this time.[^sn-4] I didn't know it at the time, but I was getting warmer. Is the lockfile checked into the repository? Yes. The compile error looked like it involved a package dependency. Was there anything special about that dependency? Oh, the package's `.cargo/config.toml` file uses `build.rustflags` to set a `--cfg` flag that the dependency requires? How very interesting... *If you haven't seen a `.cargo/config.toml` file before, feel free to [give the docs a quick skim](https://doc.rust-lang.org/stable/cargo/reference/config.html). Don't worry, we'll cover all relevant details here.* ## Builds succeeded from some directories, and failed from others Here's a "fun" quirk we discovered along the way: running `cargo check` generally succeeded fine for the user, except if they ran it with `--manifest-path`. I usually recommend using `--manifest-path` to simplify running repro steps and avoid fumbling with directory traversals—but here it was proving a problem in itself. Hence, "fun"! After some discussion, it turns out the user was running my suggested repro scripts by opening a fresh terminal pane, which defaulted to running inside their home directory. Why would that be a problem, you ask? Well... Where can one place a `.cargo/config.toml` file and expect `cargo` to find it? One place is next to your package or workspace's `Cargo.toml` file. When you run `cargo check` inside your project directory, `cargo` finds the local configuration, even if it needs to walk up the directory tree to find it. Another place is in your home directory (thankfully, the user didn't have any config there). If `cargo` finds no configuration local to the project or its ancestor directories, it will check your home directory.[^sn-5] Except ... I'm sorry for being pedantic, but the above isn't quite true.[^sn-6] If `cargo` finds no configuration local to ~~the project~~ *the current working directory*, it starts walking up the filesystem *from the working directory*. Here's a quote from the `cargo` documentation: > Cargo allows local configuration for a particular package as well as global configuration. It looks for configuration files in the current directory and all parent directories. If, for example, Cargo were invoked in `/projects/foo/bar/baz`, then the following configuration files would be probed for and unified in this order: > - `/projects/foo/bar/baz/.cargo/config.toml` > - `/projects/foo/bar/.cargo/config.toml` > - `/projects/foo/.cargo/config.toml` > - `/projects/.cargo/config.toml` > - `/.cargo/config.toml` > - `$CARGO_HOME/config.toml` which defaults to: > - Windows: `%USERPROFILE%\.cargo\config.toml` > - Unix: `$HOME/.cargo/config.toml` > > Source: [The Cargo Book, Configuration, Hierarchical Structure](https://doc.rust-lang.org/stable/cargo/reference/config.html#hierarchical-structure) If your working directory happens to be inside your project when you invoke `cargo`, this distinction doesn't come into play and the behavior is the same. But you can compile code from anywhere by passing `--manifest-path /path/to/Cargo.toml` to `cargo`, which is what our repro scripts were relying on. Connecting the dots: when `cargo` was invoked from inside the project directory, it picked up the project's `.cargo/config.toml` file with the required `--cfg` flag and compiled successfully. When we invoked it from elsewhere by using `--manifest-path` to point to the project, it looked in the *current* directory for config, didn't find any, and produced a compile error because the required `--cfg` flag wasn't set. If you want to experience this yourself, I set up a `cargo-semver-checks` test case repository with [this edge case](https://github.com/obi1kenobi/c-s-c-testcase-cargo-config-toml/blob/main/workspaces/crate-level-config/README.md) and [several others](https://github.com/obi1kenobi/c-s-c-testcase-cargo-config-toml/tree/main/workspaces). Eventually, this turned out to be something of [a red herring](https://en.wikipedia.org/wiki/Red_herring): the user was invoking `cargo-semver-checks` from inside the project directory, so the `.cargo/config.toml` could have been picked up anyway. But there was something interesting about the compilation error when running `cargo check` from a different directory: it was *identical* to what `cargo-semver-checks` itself was getting from *inside* the project directory. Was the `--cfg` parameter *still* not being passed for some reason, despite being configured in `.config/cargo.toml`? ## `cargo doc` and `cargo check` are not the same Earlier, we said that in order to check SemVer, `cargo-semver-checks` has to build the package being checked. *Strictly speaking*, this is not accurate: we need to *document* the package, not *build* it. We run `cargo doc` to generate rustdoc JSON, not `cargo check` to generate binary files. More pointless pedantry? Apparently not! The difference matters quite a bit, as it happens. `cargo` allows users to have precise control over the configuration used with their code. `cargo check` and `cargo build` interact with `rustc`, so they ([among other things](https://doc.rust-lang.org/stable/cargo/reference/config.html#buildrustflags)) read the `RUSTFLAGS` environment variable and the `build.rustflags` config file value. `cargo doc` runs `rustdoc`, not `rustc`. Bah, pedantry! Under the hood, `rustdoc` uses the same machinery as `rustc`! Except that means `cargo doc` doesn't use `RUSTFLAGS` and `build.rustflags`, but [`RUSTDOCFLAGS` and `build.rustdocflags`](https://doc.rust-lang.org/stable/cargo/reference/config.html#buildrustdocflags) instead! The user's `.cargo/config.toml` file set the required `--cfg` flag in `build.rustflags`. It said nothing about `build.rustdocflags`! So `cargo check` would pass, and `cargo doc` would fail due to missing the required `--cfg` flag. Bingo 🎯 ## Resulting improvements in `cargo-semver-checks` We shipped both correctness and UX improvements as a result of this work. On the correctness side, `cargo-semver-checks` now replicates the `cargo` configuration search process, detecting and passing on any `RUSTFLAGS` or `RUSTDOCFLAGS` it encounters instead of merely overwriting them with its own configuration. Doing this correctly is far from simple: those values can be specified in any of [6 different environment variables or 5 different config keys](https://doc.rust-lang.org/cargo/reference/config.html#buildrustflags), including in a target-specific manner that requires `cargo-semver-checks` to know which target triple it's linting. All this is backed up by [28 new test cases](https://github.com/obi1kenobi/cargo-semver-checks/pull/1471/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR1684-R1824) checking [various cursed crate and configuration combinations](https://github.com/obi1kenobi/c-s-c-testcase-cargo-config-toml/tree/main/workspaces). On the UX side, our **Not my fault** scripts are now slightly more clever: they suggest running *both* `cargo check` *and* `cargo doc`, as well as [including a `--target` flag when one is necessary](https://github.com/obi1kenobi/cargo-semver-checks/pull/1480/files) to select the correct target triple. Maintaining `cargo-semver-checks` is a repeated series of examples for a variation of [Hofstadter's law](https://en.wikipedia.org/wiki/Hofstadter%27s_law): The world is more complex than you think, even when you take into account (this variation of) Hofstadter's law. But that's okay! Every new release is a step forward, and a chance to do something better than before. Given time, patience, [funding](https://github.com/sponsors/obi1kenobi/), [egregious numbers of test cases](https://github.com/obi1kenobi/cargo-semver-checks/pull/1471/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR1684-R1824), and my stubbornness about solving problems that get in my way, this mountain will also be topped. Happy Halloween! 🎃 *If you liked this essay, consider [subscribing to my blog](/subscribe/) or following me on [Mastodon](https://hachyderm.io/@predrag), [Bluesky](https://bsky.app/profile/predr.ag), or [Twitter/X](https://twitter.com/PredragGruevski). You can also fund my writing and work on `cargo-semver-checks` via [GitHub Sponsors](https://github.com/sponsors/obi1kenobi), for which I'd be most grateful ❤* *Discuss on [r/rust](https://www.reddit.com/r/rust/comments/1okunjh/ghosts_in_the_compilation/) or [lobste.rs](https://lobste.rs/s/pyzk5t/ghosts_compilation).* [^sn-1]: I gave an impromptu ["Cursed Rust" talk at Rust Forge](https://youtu.be/6Scgq9fBZQM?t=18197) this year. The audience's excitement about that talk inspired the theme for this post! [^sn-2]: The debugging effort is dramatized to fit the post's narrative structure. It did broadly follow these steps, though they happened in batches and not as a dramatic step-by-step process as this post suggests. But this is a Halloween ghost story, so please indulge me 👻 [^sn-3]: Debugging is generally easier when users can provide a full repro and offer access to source code. Of course, that's not always an option... [^sn-4]: It's important to verify these basic assumptions when debugging a complex problem. We don't want to spend 10 hours debugging what feels like a problem in `cargo-semver-checks` only to find out that the issue was e.g. a difference in rustdoc behaviors between different Rust releases. After all, `cargo-semver-checks` relies on *unstable* rustdoc JSON features to work. [^sn-5]: Technically, it'll check `$CARGO_HOME/config.toml` which defaults to `$HOME/.cargo/config.toml` (Unix) and `%USERPROFILE%\.cargo\config.toml` (Windows). [^sn-6]: You are reading a blog post about a SemVer linter, please don't tell me you are surprised to find pedantic rules-lawyering 😁 Copyright (C) Predrag Gruevski 2025. [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en)