Now a new version of the Notary signing system seeks to change that. A multi-vendor working group was established in December 2019 to improve the image signing experience and solve several of the problems with the original implementation. Notary v2 launched in alpha form in October 2021. Here’s how it makes signing more compatible with modern container usage patterns.
What is Notary?
Notary, also known as Docker Content Trust, provides the mechanisms that sign and verify your container images. The current iteration works by adding your public key to your registry, signing your image with the key’s private counterpart, and then pushing the signed image up to the registry. Other users can verify the image by asking the registry to match its public key against the data they’ve pulled. All this functionality is built into the existing Docker CLI under the docker trust command group.
The Problems With v1
The original version of Notary was developed before the proliferation of Docker registries observed today. It’s designed for Docker Hub first and foremost whereas today you may be using registries from many different providers. GitHub, GitLab, and popular cloud deployment platforms have all started to offer integrated registries.
Notary currently works in tandem with the registry. If you want to use it with a private registry, you must also deploy your own Notary server. This makes it challenging to use image signing in environments which don’t rely on Docker Hub.
v1 doesn’t work between registries either. The signing data is lost when you pull a public image and then push it to a private registry without an accompanying Notary server. You can’t verify whether the private version stays the same as the original while it’s at rest in your registry. Similarly Notary’s current architecture doesn’t offer support for private networks and air-gapped environments that need to be physically isolated from the outside world.
The New v2 Architecture
The next generation of Notary takes the design back to the drawing board to create a simpler experience that’s more universally applicable. One of the project’s objectives is to eventually reach a state where image signature checks are turned on by default, a move which would help protect many more users from possible image tampering.
Signing data will now be pushed and pulled with image data, removing the separate step. Everything needed to verify an image will move alongside it, maintaining its availability when pushed to another registry or used in an air-gapped environment.
Notary v2 isn’t limited to signing container images either. It works with any artifact stored in an OCI-compatible registry. Now you can sign the assets that accompany your images, such as dependency lists in Software Bills of Materials (SBOMs) and the results from image scanning engines. This enforces trust across your entire deployment pipeline by highlighting unauthorized attempts to modify audits and supporting documentation.
Multiple Signers and Signature Promotion
One further area where Notary v1 falls short is when it comes to approving an image for use in your own environment. It supports only one signature per image; if a Docker Hub image is signed by its vendor, you can’t add your own signature to mark the image as suitable for your organization.
Notary v2 adds support for this workflow too. As a downstream image user, you can add new signatures to an image (or any other artifact) which others further down the chain will be able to verify. As an example, it means you’ll be able to verify the following assertions about an image identifying as ubuntu:latest:
The image was published to Docker Hub by Canonical and has not been tampered with since. The image has been signed for use by your organization. The image hasn’t changed since it was cached to your CI server’s private Docker registry.
Notary v2 is capable of maintaining trust through the entire ecosystem, instead of being mostly limited to immediate image pulls from Docker Hub. It encourages you to challenge the traditional assumption that images are “safe” because they’re pulled directly from Docker Hub. Using multiple signatures lets you validate that statement, then record it as your own seal of approval.
Using Notary v2 Today
Notary v2 isn’t ready for general use yet. Nonetheless the first alpha is available to download. The signing and verification component is called notation. It’s currently feature incomplete and offered as a standalone binary that operates independently of the Docker CLI.
Download Notation from its GitHub releases page, extract the executable, and place it somewhere on your path. Begin by generating a test signing certificate for your own use:
Now you can sign images. Notary currently only works with images in a registry. You can use Docker to quickly start a compatible registry on localhost:5000:
Build and push your image to your registry, then use Notation to sign it:
You’ve now added your signature to the image. Try to verify it using the verify command:
This will generate an error because the cert generate-test command doesn’t automatically register the generated certificate’s public key. As Notation won’t know the key used to sign the image, validation will fail. You can rectify this by adding your certificate’s public key to Notation, then trying to verify your image again:
This time Notation should emit the image’s SHA256 signing hash, indicating verification was successful. Now you should be able to pull and verify the image on another machine with Notation installed. Remember to add your certificate’s public key to your second Notation installation.
The –plain-http flag in the commands above enables Notation to use HTTP to connect to the registry. This is necessary for these examples where a Docker registry has been created locally for testing purposes. You should omit this flag when connecting to a real TLS-secured registry.
What’s Next for Notary and Notation?
Notary v2 is still under development and new capabilities will show up in future Notation builds. Certificate revocation, environment-specific verification policies, and support for registries without ORAS support are all on the roadmap.
There’s currently no stated timeframe for a stable release. Once it arrives, Notary v2 will finally add accessible, resilient, and scalable signing to the container image ecosystem. It should make signature verification usable in many more scenarios, reducing the risk of over-the-wire pull tampering and unauthorized image use.
Once it’s rolled out more broadly, your deployment systems will be able to check if an image is an “official” version from its vendor, whether it’s approved for use in your organization, and if it’s got an appropriate security scan result signed with the same key. This will add a welcome layer of extra protection and transparency for security-minded organizations running containers in high risk environments.
