Magnus' blog

Book review: Rust for Rustaceans

This is a combination of my thoughts and learnings from Rust for Rustaceans by Jon Gjengset (2021 edition). I have had it lying in my bag for a while now, slowly reading through chapters and taking notes about subjects that I have not previously experienced. This post is mainly my notes on things to remember, which can serve as a good starting point for deciding whether the book has something to offer for you as well.

My experience with Rust has mainly been through minor projects, which have mostly been web projects for university courses and unlaunched side projects. I have also used the language heavily for different coding challenges as a way to get more familiar. I think I am a good representation of an intermediate Rust programmer, which is the book’s target audience, since I have run into all the beginner traps. I completed a Master’s in Information Technology relatively recently (4 years ago), so most of the computer science subjects are still relatively fresh as of this writing. With this, my overall aim for the book is to learn something new about Rust and use it to develop more advanced Rust projects in the future.

Review

The book covers more advanced subjects that are really useful to know before diving into them. The initial chapters mostly sum up subjects an intermediate Rust developer should be somewhat familiar with, where the later chapters does a good deep dive in to more advanced details. Especially the chapters about async/await and unsafe had a lot of interesting details that I did not know. It is well written and worth a read.

The book is very well written, but that is also a minor downside. It describes everything well, but primarily using words. Sometimes code examples are missing, and other times there are too many code examples. For the harder-to-understand sections, some illustrations would have been useful — for example, in understanding concurrency or unsafe.

Learnings

These learnings are mostly notes about the book, giving an overview of interesting subjects. They mostly serve to help me remember the book, but can also be useful to check if the concepts are new to you too.

Memory

A wide pointer (or fat pointer) is a pointer where the size is encoded as part of the pointer instead of being known at compile time.

The second chapter has an interesting point about alignment and the layout of the Rust types in memory. In Rust, the layout of the memory is not guaranteed like in C. The Rust compiler can move fields around such that they are stored more efficiently. You control this by adding a repr to the type. Additionally, it describes how alignment can cause types to take up additional space for padding such that the fields are byte-aligned and easier for the CPU to access.

This reminded me of an old example showing that C actually contains the possibility of polymorphism. You just have to make the structs have the same fields at the start, and those will have the same memory layout and can be easily reused. This does not work in Rust, since without explicitly choosing the layout, the compiler will optimize it, meaning two structs with the same fields might look different in their memory layout.

Dynamic dispatching and trait objects

Dynamic dispatching happens when you use the dyn keyword instead of impl. I had an intuitive understanding of impl as it works similarly to template in C++, but I have never quite understood dyn. This part explains how the dynamic dispatching happens: instead of creating a method, it passes a trait object consisting of a reference and a vtable such that the correct method can be chosen dynamically at runtime. This is somewhat similar to how most other languages do it, so it demystified the dyn keyword for me.

Generic Traits

Generic traits work mostly as I would expect. The interesting points are the ways you can “share” the details. For example, you can make a blanket implementation that works for all the generics with impl<T> MyTrait for T where T:, which is kind of the same as having a shared base object, but without the inheritance.

Interfaces

Chapter 3, Designing Interfaces, is interesting, but mostly something I already understood well. It argues that a good interface should follow four principles:

  • Unsurprising: Names should be consistent with the ecosystem. Commonly used traits should be implemented: Debug, PartialEq, PartialOrd…
  • Flexible: Only take ownership of the type when it is required. Use the least constrained type. For example, instead of String use &str or even better impl AsRef<str>.
  • Obvious: Document every public interface. Use the type system to guide in only providing valid input and calling correct methods.
  • Constrained: Constrain what is available to the user, as any change to an exposed type could be a breaking change. Re-export types in the base to allow for moving them internally.

Errors

Errors should implement std::error::Error and Display, which provide extra helpful structures. I did not know this and could have used it a couple of times.

Opaque Errors can be implemented with Box<dyn Error + Send + Sync + 'static>. They can be used in Result when the Error is not something the user can handle. In reality they should probably panic, but returning this allows them to clean up before they panic. I do not like this approach. I previously ran into it and was really annoyed that I was not able to have better error handling, but this is apparently something that is discussed in the community as a good practice.

Testing

The Test Harness being a Rust binary just compiled to run the tests is an interesting detail. It can also be overridden with a custom one.

Doctests mean any examples included in Rust docs are tested by compiling them, ensuring they work. This is an interesting detail that ensures that users can copy them directly.

The Clippy linter is important. I was already using it.

cargo-fuzz and proptest can be used for fuzzing and property-based testing. These seem like interesting crates to try in a project.

Miri can be run next to tests to check for failures to uphold safety requirements.

Loom can be used to verify different scenarios in a thread context.

Performance testing should watch out for performance variance, compiler optimizations, and I/O overhead. These are quite common worries, but especially with Rust as it performs a lot of optimizations.

Macros

Macros are something I have seen, but never had a use case for. This is probably because I have not yet ventured into more advanced programs. They are useful for code minimization.

Declarative macros are advanced regexes that allow matching and injecting code. They have hygiene because their variable names do not conflict with the other variables.

Procedural macros are the more advanced macros, which take a token stream and return a token stream. Common use cases are function-like macros, attribute macros (like #[test]) and derive macros (like #[derive(Debug)]). With them, the syn library can be used for easier definitions. They do not have hygiene, which the implementer has to take care of.

For most day-to-day use cases, the declarative macros are more than enough.

Async await

A generator is used by the compiler to store the state of an async function. This is kind of the same idea as most other async-await implementations. The main difference is that Rust needs to ensure the memory safety of the related information. This is done with the Pin trait, which specifies that memory is locked to a certain place and will not move. This is primarily used for the async functions and generators themselves, as they cannot be moved, to ensure they are still safe in relation to the rest of their context. There are macros like Box::pin(x) that pin to the heap. Alternatively, a new pin!() macro can pin to either. Most structs do not need to be pinned. For these, the Unpin trait is implemented automatically. This specifies that the pinned data can be moved around. The compiler does this by looking at all fields and checking whether they are Unpin. If they are, the overall struct is too.

All async needs a runtime, and Rust does not provide one by default, although most applications use the tokio crate.

An async method is a Future, which has to be poll()ed in order to continue. At some point a result is expected to be returned. An executor is responsible for doing this continuous checkup. If a future owns another future, the top level will be called, and then it has to call the levels further down until it reaches the leaf futures. The leaf futures are the places where the actual waiting happens. To avoid continuously calling these futures — for example, a future waiting for network traffic might wait a while — a Waker is used to wake up the awaited future when the traffic arrives. This is implemented in the Future’s context such that it is possible to use custom ones.

When working with async, all awaits will themselves be part of the same generator. In order to separate the work, the spawn!() macro has to be used to specify that work can continue independently of the current task. This is, for example, useful in a classic HTTP server setup that continuously accepts new clients.

Unsafe

Unsafe allows extra behaviour such as pointer dereferencing, and is used as a marker to tell:

  • An unsafe block {} is used to tell that the author of that code has determined the code to be safe.
  • unsafe fn is used to tell that a caller of the method needs to ensure it is safe.
  • unsafe trait is used to tell that an implementation of the trait needs to ensure safety, but it can still be safe to call it.

These methods are not inherently dangerous, but the developer has to ensure certain conditions when they are called. It is important not just to ensure this condition holds at the time of calling the method, but also to ensure that it remains accurate when the program changes.

Naming of unsafe methods should follow the standards. A common pattern is to use _unchecked when a safety check is omitted.

The Send and Sync traits are examples of unsafe traits. These will usually be applied automatically by the compiler. If you want to apply them manually, it is because the compiler has not, because there is something it cannot guarantee. The implementation is therefore unsafe, as the developer needs to argue as to why it is safe.

When writing unsafe code, it is important to assert that certain behaviours do not happen. In cases where the code has to be performant, this can be done only in tests with cfg!(test) or debug_assert!. This provides extra safety in the implementation.

Concurrency

The difference between async and concurrency is that async could in theory all run on a single thread, kind of like Node.js does it. The concurrency chapter talks about how to work with Rust in multiple threads. The book describes basic concurrency at a good level, with the correctness and performance talk at the start, then a section about concurrency models — Shared Memory, Worker Pools and Actors — and a section about lower-level concurrency with atomic types and the like. Additionally, it gives recommendations about starting simple and only optimizing when something is proven slow, using stress tests to trigger failures, and using the Loom tool to test concurrency.

The section in relation to concurrency that I found the most interesting is about the memory ordering of atomic types. Specifically, about how the different orderings have different guarantees about how executions happen in threads, which are not always linear. The different memory orderings are interesting to know about, but I think in most cases I would stick to Ordering::SeqCst, which provides the most guarantees, but is also the slowest.

Foreign Function Interfaces

Foreign Function Interfaces allow Rust to work with other languages. They can use different setups, but most commonly the C interface is used, as most other languages also work with it. This is done by setting the symbols, like in a C program, in a way that other languages can find and call them. This is needed because the Rust compiler does not guarantee symbol names and allows for multiple functions with the same name.

bindgen and Build Scripts exist that make it possible to generate and inject other code into the build. This could, for example, be used for assembly optimizations if needed.

When designing an FFI, it is a good idea to try to hide away the memory unsafety from the other caller. This can be done in different ways. If it is not possible to use Rust’s type system to encapsulate the interface, it should be made unsafe.

No std + ecosystem

Chapter 12 is dedicated to describing how to work with Rust without a standard library. Interestingly, the language is built up around this by having a layer of a core lib with pure Rust, alloc for anything requiring dynamic allocation, and then std for the remaining parts and a re-export of the previous. This keeps the implementation details hidden from most std users, but allows embedded programming to use a subset. The book describes patterns for how to do custom dynamic memory allocation, panic handling, program initialization, out-of-memory handling, writing safe Rust for hardware and cross-compilation. This is not something I personally expect to use, but it’s an interesting chapter.

Chapter 13 is dedicated to the Rust ecosystem and further learning. The main interesting points are the patterns: index pointer and drop guard. An index pointer is simply a structure that stores both a Vec and indexes or usize numbers pointing to where certain elements are stored. This helps prevent pointers or hard-to-get-right lifetimes. The drop guard is a pattern for ensuring something happens in case a panic happens. It is simply implementing Drop for a custom type that cleans up some other data.

Tips and tricks

These are the tips and tricks I noted throughout the book:

  • Use repr(transparent) when creating a wrapper type without any additional fields such that it shares the same in-memory representation as the underlying struct. For example, struct A and struct NewA(A) would share the same representation and be easily transformed.
  • Use repr(C) when interacting with C programs to ensure the memory layout does not have to be transformed in the call.
  • Check that a type is normal (has all the default implemented traits) with a test that fails if they are not implemented: 
    fn is_normal<T: Sized + Send + Sync + Unpin>() {}
#[test]
fn normal_types() {
  is_normal::<MyType>();
}
  • “Sealed” traits can be done with: 
    pub trait CanUseCannotImplement: sealed::Sealed { }
mod sealed {
  pub trait Sealed {}
  impl<T> Sealed for T where T: TraitBounds {}
}
impl<T> CanUseCannotImplement for T where T: TraitBounds {}
  • The Semver Trick: When a new release happens, an older release can be published that uses the types from the new release such that it interplays with an updated version of the dependency. This allows dependencies requiring newer or older versions to be used with the other version.
  • Hide deprecated public items that should not be used with #[doc(hidden)]. Beware that they should still be documented; they are just not exposed in the docs, but still available in the source.
  • When compile time grows too big, subcrates can be used to split the compilation into multiple crates, which can really speed up development.
  • Features can be used to conditionally include dependencies. Some features are set by default, so you would have to override them if not wanted.
  • A patch can be used in the crate to temporarily specify a different version, for example a git branch or folder, to use instead of the dependency. This allows for checking that a fix to a dependency actually fixes the problem, or allows continued development while the branch is being released in the dependency.
  • opt-level can be used to optimize a program. 0 for “not at all” and 3 for “as much as you can”. Also “s” to optimize for binary size.
    • These can also be overridden for specific dependencies with [profile.dev.package.<package>].
  • The default panic is an unwind that unwinds the stack, although this can be changed to an abort, which is for example useful in embedded programming.
  • #[cfg(condition)] is used for conditional compilation.
    • cfg(windows) or cfg(unix) for the operating system.
    • Packages can also be enabled conditionally with [target.'cfg(windows)'.dependencies].
  • The Minimum Supported Rust Version can be found by reverting versions until the library no longer compiles. This should be done for all libraries intended for public use, as it allows for the widest possible use.
  • Enable the Clippy lints: missing_docs and missing_debug_implementations.
  • Use #[no_mangle] to make the symbol global, so it can be used in FFIs

View next or previous post: