full URL was: https://stackoverflow.com/questions/3380795/what-does-amazon-use-for-its-build-and-release-system/12597919#12597919

Sorry for some lack of formatting, I copy/pasted the diff from the answer history and missed cleaning up some things.

My "new thing" I mention has actually happened, and can be viewed here: https://www.qbtbuildtool.com

"Data driven" is considered harmful in the world of solved security issues.

@terabyte So would you say a VersionSet is a moving tag? such as a branch tag 'master' in git can track a branch, whereas the SHA itself distinctly refers to a specific version?

So, a VersionSet is a name, and refers to the most recent versions of all packages (successfully?) built against that VersionSet? Does brazil have nomenclature for the immutable thing that represents the newest versions built against that VersionSet?

Version Sets have "event" ids, which are named versions for that set and a pointer back to the parent (much like a git commit). When you build, by default the latest event is chosen to build against (if you're building locally, the "latest" is periodically updated) or you can choose to build from a different event (for example, if you need to patch a release).

The version set itself will have a name (e.g. "MyTeamsGreatService" as well as a link back to where the VS was created from "live@1593214096"). As already hinted at, events in the version set are delimited by an @.

When you create your version set, you supply one more more "root" packages that act as anchors for all of the other packages. As dependencies get discared, they are removed from the version set event.

You can also merge version sets. live typically has a lot of shared packages used by lots of teams (JVM, JDK, shared config, third party libraries, etc.). Instead of picking and choosing specific versions, you can get the latest that are built into live by importing which will pull all of the new package versions in and rebuild anything that depended on them and create a new version set event for your version set (this is such a common operation that it's possible to schedule it on a regular cadence to avoid stale dependencies). You can import from version sets other than live though, which is useful when sharing internal frameworks across close teams that are not intended to be published outside of that group of teams. That makes libraries shared within a multi-team group easy to build once and push out to all consumers.

Hinted at above, but not detailed is the separation of interface version and build version. When you take a dependency on another package in Brazil, you specify its name and interface version. You never take a dependency on a specific build version (in fact, you can't). The package config provides the graph of packages and interface versions necessary to build, but the version set is what resolves those interface versions to specific build versions.

If you create a package, say the IDL for your service API or config for clients to connect, you specify the interface version (say, "1.0"). Every time you build, the generated package is given a concrete build version ("1.0.1593214096"). That build version is what's stored in the version set for each package node. Developers never manage the build version either as an author or consumer.

There's an implicit contract, then, that within an interface version all changes are backwards compatible. Internal tooling (service definitions, config, etc.) has all been built with this in mind. APIs, object fields, enumeration values, etc. can all be added without concern about affecting consumers (at build or runtime), but if APIs need to be modified in some incompatible way, the interface version is updated (maybe "1.1") and no existing consumers are affected, only consumers that update their dependency to the new interface version. Regardless, when you build a package all consumers are rebuilt to ensure build-time consistency.

I left Amazon about a year before the OP was written, but the conclusion is spot on. Once you understand the build and deployment tools you first wonder how you ever did anything before and then start to fear how you'll do anything once you leave. One major issue, in my experience, is the Brazil packaging concept doesn't have critical mass or would require boiling the ocean to get there. At Amazon, there's one package manager - Brazil. You want a Gem, NPM package, *.so, or JAR dependency? You add it to Brazil. Sometimes you go through great pain to add it to Brazil (especially when importing a new build system). But once the package is there, everything just works. If you look at something like BSD ports, you get the comprehensive set of packages from varying package managers merged into one, but you don't get version sets. MacPorts can get pretty close since you can point to a specific SHA of package definitions along with fuzzy dependency definitions, but there's one primary set. Other package managers with fuzzy versions can't produce repeatable builds. In the Java space most dependency managers use strict versioning, which means its extraordinarily difficult to regularly pick up small minor version or patch version changes.

Going into a bit more detail, Brazil gets dependency types right as well as relatively straightforward version conflict resolution. As OP said, Brazil is a thin set of bootstrap tools which forward control to a "build system" as soon as possible. A build system is just another package in Brazil with an executable in a consistent location, so it's versioned along with the rest of your version set. Brazil itself isn't versioned, but it is really so small that in practice it doesn't matter. Build tool dependencies are scoped to work with each other but are not added to the library path during compile by default. Compile-time dependencies are exactly that. Runtime dependencies are packages that will be deployed, but not added to the library path during compile. Both build and runtime dependencies ultimately get deployed. Test dependencies are just that, a separate test set of dependencies which also include compile dependencies. Dependencies are transitive.

Because brazil-build is primarily a package manager and defers to other build systems for generating the bits, package configuration is super simple. Primarily dependencies and maybe some additional metadata describing the produced artifacts that will be added to any resolved library path. Unlike something like Maven where dependency management and build logic are intermingled, language native build tools are then used. As OP said, make or cmake for C/C++, rake for Ruby, ant for Java. By the time these tools are used and dependency paths are already added, so LD_LIBRARY_PATH or class paths don't need to be mucked with by developers. It also means that if you don't like your build tool, you can switch it out and none of your other dependencies need to change.

When version sets are imported into Apollo, all of the test and build dependencies are stripped away and the remaining graph is passed in.

Awesome, thanks for this detail!

@hohle Just to clarify:

MyTeamsGreatService" as well as a link back to where the VS was created from "live@1593214096

live in this case is the name of another version set that this one was say... branched off of? And say, MyTeamsGreatService will have a whole set of links MyTeamsGreatService@16983628 -> MyTeamsGreatService@16983627 all the way back to some other VersionSet, e.g live@1593214096

Is that an accurate understanding?

That sounds about right. A lot like a git "commit" object, without being content addressable. live was a somewhat special cased version set, since it was "public" within the company. IIRC there weren't any "root" package in live, since you didn't really deploy from it.

OP wrote:

Builds are serialized on version set

This isn't completely true. Parallel builds could happen within a version set as long as their packages didn't overlap. If a version set was used for a single service it was likely that nearly all builds would be serialized since they all ultimately impacted a root package. In live, however, there were many smaller paths which didn't impact one another.

For example, I maintained the internal tmux(1) package. It had a small number of consumers for which hit was a runtime dependency (primarily a packages which was a collection of nice developer tools, and a few other packages to make sure you had previous versions when the tux protocol changed and you upgraded while a session was still running). If I rebuilt tmux, only that small set of 3 or 4 packages was affected so any other builds against live that didn't affect those packages could run in parallel. A widely shared package, like Guava which had a lot of consumers (including transitive consumers) would require rebuilding a lot, which would then block anyone else who wanted to build a package in that set.

Later on "dry run" builds were introduced so that if you did decide to build something that would affect a lot of other things, you could perform a non-blocking build where all of the artifacts would just be sent to /dev/null but at least you would know that everything compiled and unit tests passed before potentially blocking others for some significant period.

Another great feature was that Brazil was tightly integrated with the VCS being used (Perforce, Subversion, or Git). All artifacts would included links back to the commit (in perforce there were a few large depots broken up into individual packages, in git every package was its own repo). When viewing a deployed package there was always a link back to the raw artifacts and the commit the changes were built from.

OP also mentioned that deployments were namespaced (not dissimilar to containers, but prior to CNAMEs). Workspaces were namespaced as well. A workspace consisted of a version set to track and any packages that were checked out. When working locally, any checked out packages would override the versions in the version set and everything else would be resolved from the previously built version. You could have as many of these namespaces as you wanted, they were "private" to your computer. It wasn't uncommon for people to have a large workspace for all of their team's projects, smaller workspaces for individual projects, experimental workspaces for trying things out without breaking a longer lived workspace. Since these were all namespaced, and everything is modeled in the VS, you never had to worry about not having the right RPM installed or working on two projects with conflicting dependencies. Each lived in its own namespace free to evolve independently.

Apollo would also allow you to "override" packages individually so you could deploy an application to your workstation and override an application, library, some config, etc. without needing to touch the rest of the packages in the version set. This was convenient for development, but was not "repeatable" in a strict sense, but pretty nice. When I worked on a monolithic C++ service, for example, I could check out the repo for my team's SO, build it, override it and restart the service to see my changes locally.

It's been many years, so I'm sure I don't have all of the details perfect.

@hohle wrote:

This isn't completely true. Parallel builds could happen within a version set as long as their packages didn't overlap. If a version set was used for a single service it was likely that nearly all builds would be serialized since they all ultimately impacted a root package. In live, however, there were many smaller paths which didn't impact one another.

You are right, of course, I forgot this detail. Thanks very much for expanding this answer and filling in a lot of blanks! Thanks also for confirming so much of what I said and making it feel just a little less like I am some lone crazy person screaming into the void =)

When working locally, any checked out packages would override the versions in the version set and everything else would be resolved from the previously built version. You could have as many of these namespaces as you wanted, they were "private" to your computer. It wasn't uncommon for people to have a large workspace for all of their team's projects, smaller workspaces for individual projects, experimental workspaces for trying things out without breaking a longer lived workspace. Since these were all namespaced, and everything is modeled in the VS, you never had to worry about not having the right RPM installed or working on two projects with conflicting dependencies. Each lived in its own namespace free to evolve independently.

THIS is one of the most important aspects for local development and you did an excellent job of describing it. The build tool I mentioned above, QBT, does this very similarly, and it is the most important feature IMO, from a lone developer's perspective.

CONTEXT: I was on the Amazon build team from 2006-2009, wrote this answer in 2012, QBT was extracted from Palantir and open sourced by @amling in October of 2015, and I created this gist in 2017.

QBT is basically unused by anyone for anything except for core developers @amling and myself, for personal projects, but it is extremely high quality and stable and could be used in an enterprise environment by anyone with sufficient resources and motivation. Using it from the ground up would be fairly easy, but moving a significant codebase to it would be extremely challenging (just like moving a codebase to Brazil and Apollo, as @hohle put it, "would require boiling the ocean to get there").

@terabyte is that why blaze was permuted as bazel by google to sound very similar to brazil 😄 ?

Hi Carl! Nice write-up on Brazil! I hope you are doing well.

@terabyte is that why blaze was permuted as bazel by google to sound very similar to brazil ?

Probably a coincidence...but who knows? =)

Hi Carl! Nice write-up on Brazil! I hope you are doing well.

Hey man, thanks! I am doing well, and I hope you are too! Drop me an email some time if you want to chat or catch up!

This should all be re-written in a much more ranty way imo… 😂

This all sounds very similar to https://nixos.org/ and https://guix.gnu.org/ (when used as package managers, not as distros).

@uri-canva, yeah, that is pretty much Nix's concept of how to manage dependencies of packages.

There is a missing bit in Nix - the deployment system since Nix is "pure". However, after experimenting with Nix I made it work as a deployment system using "side-effects" that I created on top of fixed-output derivation.

But yeah, that amazing to see the idea is getting converged over years into a solid proven-by-many-use-cases and implementations approach.

I work at Amazon and this explanation was far more clear and informative than anything I've ever read in our internal docs. For that, I thank you.

As someone who has worked in the industry before joining Amazon, Brazil always such an amazement to me. To my coworkers who have only worked at Amazon I can never get through how great it is compared to what's out there. Definitely going to be one of the top things I miss when I leave.

Once you understand the build and deployment tools you first wonder how you ever did anything before and then start to fear how you'll do anything once you leave.

You could almost find-replace “Brazil” for “Blaze/Bazel”, “Apollo” with “x20", “smithy” with “chubby/grpc”, and “Amazon” with “Google”, and this post would be exactly correct. The biggest difference between Amazon and Google that took me getting used to (started from Google) was Brazil's source-first philosophy, directly opposite Brazil's package-first philosophy. I don't know enough about BuildXL, but Microsoft has clearly "solved" this as well.

I get that this post was originally written the better part of a decade ago To @terabyte's statement that these are "inevitable" conclusions of a large engineering organization, that is 100% correct then as it is now. And today, a decade later, we're finally getting Bazel, Nix, Docker, and even the language-specific tools (NPM, pip, Gem, as long as you stay within their languages) to provide Reproducibility and Consistency, while coordinating with git to provide Change Management. Yay!