% Data Representation in Rust Low-level programming cares a lot about data layout. It's a big deal. It also pervasively influences the rest of the language, so we're going to start by digging into how data is represented in Rust. # The rust repr Rust gives you the following ways to lay out composite data: * structs (named product types) * tuples (anonymous product types) * arrays (homogeneous product types) * enums (named sum types -- tagged unions) For all these, individual fields are aligned to their preferred alignment. For primitives this is equal to their size. For instance, a u32 will be aligned to a multiple of 32 bits, and a u16 will be aligned to a multiple of 16 bits. Composite structures will have their size rounded up to be a multiple of the highest alignment required by their fields, and an alignment requirement equal to the highest alignment required by their fields. So for instance, ```rust struct A { a: u8, c: u64, b: u32, } ``` will have a size that is a multiple of 64-bits, and 64-bit alignment. There is *no indirection* for these types; all data is stored contiguously as you would expect in C. However with the exception of arrays, the layout of data is not by default specified in Rust. Given the two following struct definitions: ```rust struct A { a: i32, b: u64, } struct B { x: i32, b: u64, } ``` Rust *does* guarantee that two instances of A have their data laid out in exactly the same way. However Rust *does not* guarantee that an instance of A has the same field ordering or padding as an instance of B (in practice there's no *particular* reason why they wouldn't, other than that its not currently guaranteed). With A and B as written, this is basically nonsensical, but several other features of Rust make it desirable for the language to play with data layout in complex ways. For instance, consider this struct: ```rust struct Foo { count: u16, data1: T, data2: U, } ``` Now consider the monomorphizations of `Foo` and `Foo`. If Rust lays out the fields in the order specified, we expect it to *pad* the values in the struct to satisfy their *alignment* requirements. So if Rust didn't reorder fields, we would expect Rust to produce the following: ```rust struct Foo { count: u16, data1: u16, data2: u32, } struct Foo { count: u16, _pad1: u16, data1: u32, data2: u16, _pad2: u16, } ``` The former case quite simply wastes space. An optimal use of space therefore requires different monomorphizations to *have different field orderings*. **Note: this is a hypothetical optimization that is not yet implemented in Rust 1.0.0** Enums make this consideration even more complicated. Naively, an enum such as: ```rust enum Foo { A(u32), B(u64), C(u8), } ``` would be laid out as: ```rust struct FooRepr { data: u64, // this is *really* either a u64, u32, or u8 based on `tag` tag: u8, // 0 = A, 1 = B, 2 = C } ``` And indeed this is approximately how it would be laid out in general (modulo the size and position of `tag`). However there are several cases where such a representation is ineffiecient. The classic case of this is Rust's "null pointer optimization". Given a pointer that is known to not be null (e.g. `&u32`), an enum can *store* a discriminant bit *inside* the pointer by using null as a special value. The net result is that `sizeof(Option<&T>) == sizeof<&T>` There are many types in Rust that are, or contain, "not null" pointers such as `Box`, `Vec`, `String`, `&T`, and `&mut T`. Similarly, one can imagine nested enums pooling their tags into a single descriminant, as they are by definition known to have a limited range of valid values. In principle enums can use fairly elaborate algorithms to cache bits throughout nested types with special constrained representations. As such it is *especially* desirable that we leave enum layout unspecified today. # Dynamically Sized Types (DSTs) Rust also supports types without a statically known size. On the surface, this is a bit nonsensical: Rust must know the size of something in order to work with it. DSTs are generally produced as views, or through type-erasure of types that *do* have a known size. Due to their lack of a statically known size, these types can only exist *behind* some kind of pointer. They consequently produce a *fat* pointer consisting of the pointer and the information that *completes* them. For instance, the slice type, `[T]`, is some statically unknown number of elements stored contiguously. `&[T]` consequently consists of a `(&T, usize)` pair that specifies where the slice starts, and how many elements it contains. Similarly Trait Objects support interface-oriented type erasure through a `(data_ptr, vtable_ptr)` pair. Structs can actually store a single DST directly as their last field, but this makes them a DST as well: ```rust // Can't be stored on the stack directly struct Foo { info: u32, data: [u8], } ``` # Zero Sized Types (ZSTs) Rust actually allows types to be specified that occupy *no* space: ```rust struct Foo; // No fields = no size enum Bar; // No variants = no size // All fields have no size = no size struct Baz { foo: Foo, bar: Bar, qux: (), // empty tuple has no size } ``` On their own, ZSTs are, for obvious reasons, pretty useless. However as with many curious layout choices in Rust, their potential is realized in a generic context. Rust largely understands that any operation that produces or stores a ZST can be reduced to a no-op. For instance, a `HashSet` can be effeciently implemented as a thin wrapper around `HashMap` because all the operations `HashMap` normally does to store and retrieve keys will be completely stripped in monomorphization. Similarly `Result<(), ()>` and `Option<()>` are effectively just fancy `bool`s. Safe code need not worry about ZSTs, but *unsafe* code must be careful about the consequence of types with no size. In particular, pointer offsets are no-ops, and standard allocators (including jemalloc, the one used by Rust) generally consider passing in `0` as Undefined Behaviour. # Drop Flags For unfortunate legacy implementation reasons, Rust as of 1.0.0 will do a nasty trick to any type that implements the `Drop` trait (has a destructor): it will insert a secret field in the type. That is, ```rust struct Foo { a: u32, b: u32, } impl Drop for Foo { fn drop(&mut self) { } } ``` will cause Foo to secretly become: ```rust struct Foo { a: u32, b: u32, _drop_flag: u8, } ``` For details as to *why* this is done, and how to make it not happen, check out [SOME OTHER SECTION]. # Alternative representations Rust allows you to specify alternative data layout strategies from the default Rust one. ## repr(C) This is the most important `repr`. It has fairly simple intent: do what C does. The order, size, and alignment of fields is exactly what you would expect from C or C++. Any type you expect to pass through an FFI boundary should have `repr(C)`, as C is the lingua-franca of the programming world. However this is also necessary to soundly do more elaborate tricks with data layout such as reintepretting values as a different type. However, the interaction with Rust's more exotic data layout features must be kept in mind. Due to its dual purpose as a "for FFI" and "for layout control", repr(C) can be applied to types that will be nonsensical or problematic if passed through the FFI boundary. * ZSTs are still zero-sized, even though this is not a standard behaviour in C, and is explicitly contrary to the behaviour of an empty type in C++, which still consumes a byte of space. * DSTs are not a concept in C * **The drop flag will still be added** * This is equivalent to repr(u32) for enums (see below) ## repr(packed) `repr(packed)` forces rust to strip any padding it would normally apply. This may improve the memory footprint of a type, but will have negative side-effects from "field access is heavily penalized" to "completely breaks everything" based on target platform. ## repr(u8), repr(u16), repr(u32), repr(u64) These specify the size to make a c-like enum (one which has no values in its variants).