This post is coauthored by Tomasz Nowak and Predrag Gruevski. It describes work the two of us did together with Bartosz Smolarczyk, Michał Staniewski, and Mieszko Grodzicki.

Anecdotally, cargo-semver-checks is a helpful tool for preventing the semver violations that every so often cause ecosystem-wide pain. This is why it earned a spot in the CI pipelines of key Rust crates like tokio, and also why the cargo team hopes to integrate it into cargo itself.

While anedotal evidence is nice, we wanted to get concrete data across a large sample of real-world Rust code. Inspired by Crater, A tool that builds a large number of public Rust crates and runs their test suites to check for Rust compiler regressions. we used cargo-semver-checks to lint the top 1000 most-downloaded library crates on crates.io. Our test setup has no connection to the Crater project. However, between us this work was affectionately known as semver-crater as a succinct and clear description of the work. We hope the team working on Crater doesn't mind this way of paying homage to our source of inspiration.

The outcome was a goldmine of valuable data.

TL;DR + table of contents

Long story short: semver accidents are common. They happen even in the most carefully-developed projects run by the most experienced maintainers. The maintainers are not to blame, and improved tooling is our best way forward. cargo-semver-checks is part of that improved tooling story, since it found every semver violation we report here.

Across more than 14000 releases We only considered non-yanked releases published in 2017 or later. We only scanned minor and patch releases, since major version releases do not have any semver obligations. We also skipped 919 releases which we were not able to build with a modern version of Rust. That left us with 14389 total scanned releases. of the top 1000 most downloaded crates, on average:

The most common sources of semver violations were:

Table of contents

We'll dig deeper into our findings in a bit. First, we have to discuss a key point: none of this is maintainers' fault.

This is a failure of tooling, not humans

Thanks to semver, cargo update can easily upgrade dependency versions to bring in performance upgrades, security fixes, and new functionality. These benefits are significant and should not be understated.

Unfortunately, the benefits come at a cost to maintainers. While many semver rules seem "obvious," there's also a long tail of complex rules with tricky edge cases. For example, editing the details of private types can sometimes result in a major breaking change in a public API elsewhere in the library — in more than one way. Spooky action at a distance!

Demanding perfection from maintainers would be naïve, unreasonable, and unfair. Whenever hardworking, conscientious, well-intentioned people make a mistake, the failure is not with the people but in the system.

Blaming human error would also be out of line with Rust's existing practices. After all, Rust adopted borrow-checking to address accidental and costly mistakes originating from another system of complex rules. The parallels to semver and cargo-semver-checks are clear: in both cases, we rely on automated systems to check the rules that are not amenable to manual checking by humans.

Analyses like this one are key to learning how we can do better. Our findings help us understand the needs of the ecosystem, contextualize our impact thus far, and determine how to best help Rustaceans going forward.

Detailed results & how we validated them

Automated linters can sometimes have false-positives, so we spent substantial effort on validating our results.

We discovered a total of 3062 verified semver violations across all scanned crate releases. Each of those was first reported by cargo-semver-checks and then validated by a combination of automated and manual means.

Detailed results (click to expand)

Here is a table showing all the different kinds of verified semver violations we discovered. We show which cargo-semver-checks lint caught each semver violation, and how many different releases and crates had that kind of violation.

lint nameindividual itemsdifferent releasesaffected crates
inherent method missing7914127
enum variant added38213860
constructible struct adds field34312334
auto trait impl removed3185745
struct missing2916640
function missing2675033
inherent method const removed13953
derive trait impl removed1151111
enum variant missing1122718
struct pub field missing793216
enum missing782620
trait missing452419
method parameter count changed221412
enum marked non-exhaustive1644
struct repr(C) removed1233
constructible struct adds private field976
inherent method unsafe added933
function parameter count changed844
function unsafe added822
unit struct changed kind532
enum tuple variant field missing422
tuple struct to plain struct422
enum tuple variant field added333
enum repr int removed111
enum struct variant field added111

As part of our validation process, we discarded approximately 10000 other instances where cargo-semver-checks reported an issue that was determined to be either erroneous (confirmed false-positive) or inconclusive (e.g., causing rustc to crash when attempting to use the affected release in a new crate).

Here are the major components of our validation process.

Automated validation via "witnesses"

For each reported semver violation, we created a witness – a code snippet that compiles on the older library version, but fails to compile on the newer version due to the semver-violating change. This is how we prove that code external to the library, such as code in a downstream use case, can be impacted by that semver issue.

For example, imagine a library with the following code:

pub enum Example {
    First,
    Second,

    // Imagine the following variant is added
    // in a minor version. This violates semver,
    // since `Example` is an exhaustive enum.
    Third,
}

The witness for this code would look like this:

use dependency::Example;

fn witness(value: Example) {
    match value {
        Example::First => {}
        Example::Second => {}
    }
}

This snippet compiles successfully with the original version, but is affected by the breaking change in the new version:

error[E0004]: non-exhaustive patterns: `Example::Third` not covered
  --> src/lib.rs:4:11
   |
4  |     match value {
   |           ^^^^^ pattern `Example::Third` not covered
   |

Handling the #[doc(hidden)] attribute

In the code above, what if the Example enum was marked #[doc(hidden)]? Items marked with this attribute don't appear in the documentation of the crate's public API, but are still accessible outside the crate. This can be useful, for example, in crates that expose macros: the macros' internal implementation details are usually not themselves a stable public API, even though they must be public for the macros to work. #[doc(hidden)] items therefore have reduced semver obligations: if our Example enum above was #[doc(hidden)], adding a new variant would not have violated semver. Interestingly, #[doc(hidden)] items still have some semver obligations.

While we've done some work on correctly handling #[doc(hidden)], today's version of cargo-semver-checks still has a false-positive here. Thanks to this survey, we saw that #[doc(hidden)] is by far the most common source of false-positives in cargo-semver-checks. We are prioritizing shipping a fix here. A witness wouldn't detect this as false-positive, either — it would also claim a violation.

We discarded over 6000 such false-positives! We used a combination of automated and manual triage, ensuring that flagged items are neither directly hidden nor indirectly hidden via #[doc(hidden)] on a containing module.

Our automated triage process relied on rustdoc's JSON output format. It detected hidden items by finding items that are emitted only when rustdoc is passed the nightly-only --document-hidden-items flag.

We followed this up by manually inspecting the source code of any items that were not eliminated as hidden via automated means. This step protected our results against possible false-negatives caused by bugs in our automated script, in rustdoc or its JSON backend, or in the nightly-only rustdoc flag we used.

Non-exhaustiveness prior to #[non_exhaustive]

These days, it's easy to forget that #[non_exhaustive] is a fairly recent addition — it was only stabilized in Rust 1.40, released in late December 2019. Our analysis covers releases made from 2017 onward, covering 3 years in which #[non_exhaustive] did not exist in the Rust language. In 2023, we expect that non-exhaustive types are marked #[non_exhaustive], and additions to exhaustive types are a clear-cut major breaking change. It seems unfair to apply the same standard to code released in 2017–2019.

Semver is about communicating expectations with users. Prior to the introduction of the #[non_exhaustive] attribute, maintainers noted non-exhaustiveness in doc comments or via enum variants with names like __Nonexhaustive. As these were the community-accepted ways of indicating non-exhaustiveness at the time, that is the standard to which we held crates in our analysis. We manually triaged exhaustiveness violations with those kinds of documented non-exhaustiveness. This is an example of how the rules of semver change over time as a function of community expectations. In 2018, Rustaceans might have expected an enum to have its non-exhaustiveness communicated via a doc comment or a __Nonexhaustive variant. In 2023, we expect that non-exhaustive enums have the #[non_exhaustive] attribute — if the attribute isn't set, we probably wouldn't look for exhaustiveness information in the enum's doc comment. Then consider the act of adding a variant to an enum only specified as non-exhaustive in a doc comment: that's a major breaking change in 2023, but not in 2018.

Consulting maintainers

Having verified our results via both automated and manual means, we decided to add one last check: we privately reached out to several maintainers of affected crates and discussed our findings with them.

In all cases, those maintainers confirmed our findings as correct.

In most cases, the maintainers stated the semver violations were novel, and not previously discovered nor reported anywhere to their knowledge.

In a tiny number of cases, maintainers reported making a semver-breaking change on purpose. In one example, a part of a library was unintentionally made public in one release and that change was rolled back in the subsequent release, which is technically a removal of public API.

Such situations are why cargo-semver-checks aims to aid and inform maintainers, not take away their power to decide what's best for their crate. We consider semver-checking akin to the cargo publish check about uncommitted changes: inform the user about the findings, but allow them to explicitly opt into proceeding if they are confident that's the right thing to do.

This is only a fraction of all semver violations

While this work found many real-world semver violations, our current setup could only hope to detect a fraction of all such issues.

There are good reasons to believe there are many more semver issues still to be discovered:

Just scratching the surface of our work

This case study summarizes several engineer-years' worth of work done by five people. It shows that cargo-semver-checks can discover semver violations in real-world Rust code, and is therefore effective in helping today's maintainers avoid semver violations in their new releases.

But this is just a slice of what we built and discovered. We didn't get to talk about many other interesting topics, like:

If you are curious to learn more, we have a few resources for you to check out!

The work described in this post was part of the bachelors' thesis project for Tomasz Nowak, Bartosz Smolarczyk, Michał Staniewski, and Mieszko Grodzicki. Their thesis is available here, and contains many more details that we couldn't fit here.

More information on cargo-semver-checks is available on its GitHub page. It's safe to assume that the vast majority of bug reports opened by Tomasz, Bartosz, Michał, Mieszko, or Predrag in the last year were discovered as a result of the semver survey described in this post.

Various nuances of semver in Rust have already been covered on this blog, and more posts on the subject are sure to follow. You can subscribe to this blog via RSS or via email.

If you maintain Rust crates, are you using cargo-semver-checks already? Why, or why not?

Discuss on r/rust or lobste.rs.

Thanks to Tim McNamara, Luca Palmieri, Steve Klabnik, oli-obk, weihanglo, and Ed Page for their feedback on drafts of this post. All mistakes belong to the post authors alone.