Technology Sharing

Use docker buildx to build cross-platform images

2024-07-12

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Buildx is a build tool officially provided by Docker. It can help users build Docker images quickly and efficiently, and supports the construction of multiple platforms. With buildx, users can build images of multiple architectures, such as x86 and arm architectures, in a single command without having to manually operate multiple build commands. In addition, buildx also supports multi-stage building and caching of Dockerfile, which can greatly improve the efficiency and speed of image building.

Buildx is a CLI plugin for managing Docker builds. It uses BuildKit to extend Docker build functionality.

BuildKit is a high-performance build engine officially provided by Docker, which can be used to replace Docker's original build engine. Compared with the original engine, BuildKit has faster build speed, higher parallelism, less resource usage and better security.

To install and use buildx, you need Docker Engine version 19.03 or higher.

Cross-platform image building strategy

Builder supports building cross-platform images using different strategies.

Using QEMU emulation support in the kernel

If you are using Docker Desktop, QEMU is already supported. QEMU is the simplest strategy for building cross-platform images. It does not require any changes to the original Dockerfile. BuildKit will implement cross-platform program execution through the Linux kernel function binfmt_misc.

working principle:

QEMU is a processor emulator that can simulate different CPU architectures. We can think of it as another form of virtual machine. In buildx, QEMU is used to execute binaries of non-native architectures during the build process. For example, when building an ARM image on an x86 host, QEMU can simulate the ARM environment and run ARM binaries.

binfmt_misc is a module for the Linux kernel that allows users to register executable file formats and corresponding interpreters. When the kernel encounters an executable file of an unknown format, it uses binfmt_misc to find the interpreter associated with that file format (in this case, QEMU) and runs the file.This feature is disabled on the rk3568 platform.

The combination of QEMU and binfmt_misc makes cross-platform building possible via buildx. This allows us to build Docker images for other architectures on a host of one architecture without having to own the actual target hardware.

Although Docker Desktop comes pre-configured with binfmt_misc support for other platforms, for other versions of Docker you may need to start a privileged container using the tonistiigi/binfmt image to enable support.

docker run --privileged --rm tonistiigi/binfmt --install all

Running this command will install interpreter programs for different architectures, namely the qemu emulator in the /usr/bin directory:

The binfmt_misc module is loaded on our build server:

The role of qemu in docker buildx

Docker Buildx is an experimental feature of Docker, which extends Docker's build capabilities, including the use of multiple nodes, qemu, etc. QEMU is an open source virtual machine software that can simulate different CPUs and other hardware, allowing us to build an image of another operating system on one operating system.

Using the qemu function of Docker Buildx can help us build Docker images for a variety of different architectures (such as ARM, MIPS, etc.).

The following is a simple example showing how to use Docker Buildx and QEMU to build a Docker image for the ARM architecture:

  1. # 创建一个新的 buildkit 实例
  2. docker buildx create --name mybuilder --use
  3.  
  4. # 启动 buildkit 实例
  5. docker buildx start mybuilder
  6.  
  7. # 启用 QEMU 驱动支持
  8. docker buildx inspect --bootstrap
  9.  
  10. # 构建一个面向 ARM 架构的 Docker 镜像
  11. docker buildx build --platform linux/arm/v7 -t myimage:latest .

In this example,--platformThe parameter specifies that the target platform we want to build is ARM v7. In this way, Docker will use QEMU to simulate an ARM environment and build a Docker image for the ARM architecture on an x86 architecture machine.

binfmt_misc file system

binfmt-misc is a function provided by the Linux kernel that is similar to the file association function on Windows. However, it is more powerful than file association in that it can not only judge based on the file suffix, but also use different programs to open it based on the file content (Magic Bytes). A typical usage scenario is: using qemu to run binary files on other architecture platforms.

Enable binfmt-misc

To temporarily enable it, use the following command:

$ sudo mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

This method will become invalid after reboot. If you want it to be effective for a long time, you can add a line to the /etc/fstab file:

none  /proc/sys/fs/binfmt_misc binfmt_misc defaults 0 0

You can use the following command to check whether the startup is successful:

  1. $ mount | grep binfmt_misc
  2. binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
  3. $ ls -l /proc/sys/fs/binfmt_misc
  4. 总用量 0
  5. --w------- 1 root root 0 2月   5 22:55 register
  6. -rw-r--r-- 1 root root 0 2月   5 22:55 status

First prepare a program for arm64 architecture, and after execution, an error is found:

bash: ./go-test：无法执行二进制文件: 可执行文件格式错误

Now, we execute the command apt install qemu-user-binfmt, and then run the above arm64 program, and we can find that it can run normally. After installing qemu-user-binfmt, several files will be created in the /proc/sys/fs/binfmt_misc directory, one of which is qemu-aarch64. Let's take a look at the content of this file:

  1. root@ubuntu:/proc/sys/fs/binfmt_misc# cat qemu-aarch64 
  2. enabled
  3. interpreter /usr/bin/qemu-aarch64-static
  4. flags: OC
  5. offset 0
  6. magic 7f454c460201010000000000000000000200b700
  7. mask ffffffffffffff00fffffffffffffffffeffffff
  8. root@ubuntu:/proc/sys/fs/binfmt_misc#

This file describes the rules file:

The first line "enabled" indicates that the rule is enabled;

The second line interpreter /usr/bin/qemu-aarch64-static means using /usr/bin/qemu-aarch64-static to execute the binary file;

The third line: OC represents the flag of operation, the specific meaning is as follows:

P: stands for persist-argv, which means that when the simulator is called, the original arguments (argv) will be preserved. This is useful for cases where silent programs need to know their own names (i.e. argv[0]) when they are run.

O: stands for offset, which means that before starting the simulator, an offset needs to be read from the binary file, and this offset will be used as a parameter of the simulator

C: stands for credentials, which means the emulator will run with the same user ID and group ID used for the original program. This helps ensure that the emulator runs with the same permissions as the original program.

The fourth line: offset 0 means reading the file from the 0th value;

Line 5: magic 7f454c4602010100000000000000000000200b700 indicates the modulus byte to be matched;

The magic field in the ELF file header of the arm64 architecture is as follows, which means that the binfmt_misc file system can determine which architecture simulator to use for this file based on the magic field in the ELF file:

Below are two different architectures

mips architecture: 7f454c46010201000000000000000000000020008
arm64 architecture: 7f454c4602010100000000000000000000200b700

Line 6: mask ffffffffffffff00fffffffffffffffffffffff represents the byte mask, which is used to ignore some unimportant bytes in the file.

Running arm64 architecture Docker images in x86_64 systems

Now we usedockerThe command runs an arm64 image:

  1. $ docker run -it arm64v8/ubuntu bash
  2. Unable to find image 'arm64v8/ubuntu:latest' locally
  3. latest: Pulling from arm64v8/ubuntu
  4. 005e2837585d: Pull complete 
  5. Digest: sha256:ba545858745d6307f0d1064d0d25365466f78d02f866cf4efb9e1326a4c196ca
  6. Status: Downloaded newer image for arm64v8/ubuntu:latest
  7. standard_init_linux.go:207: exec user process caused "no such file or directory"

After some exploration, I found that I only needed to execute the command:apt install qemu-user-static, then start dockercontainerIt's normal.

Using multi-stage cross-builds in Dockerfile

The complexity of cross-compilation does not lie in Docker, but in the program itself. For example, Go programs are very easy to cross-compile. You only need to execute the GOOS and GOARCH environment variables when using go build to build the program.

Create a builder

To use buildx to build a cross-platform image, we need to first create a builder, which can be translated as a builder.

Use the docker buildx ls command to view the builder list:

  1. root@ubuntu:/proc# docker buildx ls
  2. NAME/NODE         DRIVER/ENDPOINT             STATUS  PLATFORMS
  3. mybuild *         docker-container                    
  4.   mybuild0        unix:///var/run/docker.sock running linux/arm64*, linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386
  5. vigilant_hugle    docker-container                    
  6.   vigilant_hugle0 unix:///var/run/docker.sock stopped 
  7. default           docker                              
  8.   default         default                     running linux/amd64, linux/386

The * indicates the builder currently in use. When we run the docker build command, we are using this builder to build the image. The DRIVER/ENDPOINT in the second column indicates the driver used. Buildx supports the following drivers:

  • docker: Use the BuildKit library bundled into the Docker daemon, which is the default BuildKit after installing Docker.
  • docker-container: Create a new dedicated BuildKit container using Docker.
  • kubernetes: Creates a BuildKit Pod in a kubernetes cluster.
  • remote: Connect directly to a manually managed BuildKit daemon.

Because the default builder using the docker driver does not support building cross-platform images with a single command (the --platform parameter of the default builder only accepts a single value), we need to create a new builder using the docker-container driver.

The command syntax is as follows:

$ docker buildx create --name=<builder-name> --driver=<driver> --driver-opt=<driver-options>

The meaning of the parameters are as follows;

--name: Build name, required.

--driver: Builder driver, default is docker-container.

--driver-opt: Driver options, such as the option --driver-opt=image=moby/buildkit:v0.11.3 can install the specified version of BuildKit. The default value is moby/buildkit.

We can create a new builder using the following command:

  1. $ docker buildx create --name mybuilder
  2. mybuilder

Check the builder list again:

  1. $ docker buildx ls
  2. NAME/NODE       DRIVER/ENDPOINT             STATUS   BUILDKIT PLATFORMS
  3. mybuilder *     docker-container
  4.   mybuilder0    unix:///var/run/docker.sock inactive
  5. default         docker
  6.   default       default                     running  20.10.21 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
  7. desktop-linux   docker
  8.   desktop-linux desktop-linux               running  20.10.21 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

You can find that the selected builder has been switched to Mybuilder. If it is not selected, you need to manually switch the builder using the docker buildx use mybuilder command.

Start builder

The current status of our newly created mybuilder is inactive and needs to be started before it can be used.

  1. $ docker buildx inspect --bootstrap mybuilder
  2. [+] Building 16.8s (1/1) FINISHED
  3.  => [internal] booting buildkit                                                                                                                                  16.8s
  4.  => => pulling image moby/buildkit:buildx-stable-1                                                                                                               16.1s
  5.  => => creating container buildx_buildkit_mybuilder0                                                                                                              0.7s
  6. Name:   mybuilder
  7. Driver: docker-container
  8.  
  9. Nodes:
  10. Name:      mybuilder0
  11. Endpoint:  unix:///var/run/docker.sock
  12. Status:    running
  13. Buildkit:  v0.9.3
  14. Platforms: linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

The inspect subcommand is used to check the build status. Use the --bootstrap parameter to start the mybuilder builder. Check the builder list again and you will see that the mybuilder status has changed to running.

  1. $ docker buildx ls
  2. NAME/NODE       DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
  3. mybuilder *     docker-container
  4.   mybuilder0    unix:///var/run/docker.sock running v0.9.3   linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
  5. default         docker
  6.   default       default                     running 20.10.21 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
  7. desktop-linux   docker
  8.   desktop-linux desktop-linux               running 20.10.21 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

The values ​​shown in the PLATFORMS column

linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6 are all the platforms supported by the current builder.

Now use the docker ps command to see that the BuildKit container corresponding to the mybuilder builder has been started.

  1. $ docker ps
  2. CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS         PORTS                                NAMES
  3. b8887f253d41   moby/buildkit:buildx-stable-1   "buildkitd"              4 minutes ago   Up 4 minutes                                        buildx_buildkit_mybuilder0

This container is used to help us build cross-platform images. Do not delete it manually.

Use builder to build cross-platform images

$ docker buildx build --platform linux/arm64,linux/amd64 -t jianghushinian/hello-go .

The syntax of docker buildx build is the same as docker build. The --platform parameter indicates the target platform for building the image, and -t indicates the tag of the image. . indicates that the context is the current directory。 

The only difference is the support for the --platform parameter. The --platform parameter of docker build only supports passing one platform information, such as --platform linux/arm64, which means that an image of a single platform can be built at a time.

Using docker buildx build to build images supports passing multiple platform information at the same time, separated by English commas, so that multiple cross-platform images can be built with only one command.

After executing the above command, we will get a warning:

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

This warning reminds us that we have not specified an output for the docker-container driver. The build results will only be kept in the build cache. Use --push to push the image to the Docker Hub remote repository, and use --load to save the image locally.

This is because the mybuilder we created starts a container to run BuildKit. It cannot directly output the built cross-platform image to the local machine or push it to a remote location. The user must manually specify the output location.

We can try to specify --load to save the image on the local host.

  1. $ docker buildx build --platform linux/arm64,linux/amd64 -t jianghushinian/hello-go . --load
  2. [+] Building 0.0s (0/0)
  3. ERROR: docker exporter does not currently support exporting manifest lists

As a result, an error log will be obtained. It seems that it does not support exporting cross-platform images to the local computer. This is actually because multiple --platforms are passed. If --platform only passes one platform, you can use --load to export the built image to the local computer.

Then we can only push the cross-platform image to the remote warehouse through the --push parameter. But before that, you need to make sure you have logged in using docker login.

$ docker buildx build --platform linux/arm64,linux/amd64 -t jianghushinian/hello-go . --push

Now log in to Docker Hub and you can see the cross-platform image pushed up.

We can also use imagestools to check the manifest information of cross-platform images. This command can only be used to obtain image information in the warehouse, and local images cannot be viewed.

  1. $ docker buildx imagetools inspect jianghushinian/hello-go
  2. Name:      docker.io/jianghushinian/hello-go:latest
  3. MediaType: application/vnd.docker.distribution.manifest.list.v2+json
  4. Digest:    sha256:51199dadfc55b23d6ab5cfd2d67e38edd513a707273b1b8b554985ff562104db
  5.  
  6. Manifests:
  7.   Name:      docker.io/jianghushinian/hello-go:latest@sha256:8032a6f23f3bd3050852e77b6e4a4d0a705dfd710fb63bc4c3dc9d5e01c8e9a6
  8.   MediaType: application/vnd.docker.distribution.manifest.v2+json
  9.   Platform:  linux/arm64
  10.  
  11.   Name:      docker.io/jianghushinian/hello-go:latest@sha256:fd46fd7e93c7deef5ad8496c2cf08c266bac42ac77f1e444e83d4f79d58441ba
  12.   MediaType: application/vnd.docker.distribution.manifest.v2+json
  13.   Platform:  linux/amd64

 

As you can see, this cross-platform image contains images of two target platforms, namely linux/arm64 and linux/amd64.

Personal profile

I have devoted myself to the research of technology for more than 30 years. I am proficient in various languages ​​such as Java, Linux, JavaScript, PHP, CSS, etc. I have made many contributions in the field of open source. I have established a developer documentation site to share some problems in technology development for everyone to read.

my contact information

Mail