hi, i'm tmac

SSH with Docker and BuildKit (with SPM)

Posted Saturday, April 3, 2021.

In a previous post, Docker with private SPM dependencies we discussed how to inject an SSH key into the Dockerfile for private dependency resolution, and also how to make this work in GitHub actions.

With the new Docker BuildKit (requires Engine 18.09 or higher), there is a new option to forward SSH agent connections. BuildKit is on by default in Engine 20.10 and in the action docker/build-push-action@v2. Because it is now the default option, it doesn't require the build to use DOCKER_BUILDKIT=1 build . when building, or have # syntax = docker/dockerfile:1.2 at the top of your Dockerfile to get access to these new features.

While we would always recommend using multi-stage builds (future post coming soon) so that the stage with the SSH key is not available to the public and only for the build, it seems that during the build the old version would leak the private key to the logs in the step echo "$GITHUB_SSH" ... to stdout. For local builds this isn't much of a risk. However on GitHub actions, anyone with access to the actions tab - and therefore the history of all the action's logs - would be able to see all or part of the key.

Using the new --ssh option in the build will stop this from being a possibility. Here is how to set this up for both local builds and on GitHub actions.

Dockerfile Changes

Original

From Docker with private SPM dependencies, our Dockerfile looked as such:

FROM swift:5.3-focal

ARG GITHUB_SSH

# Authorize SSH Host
RUN mkdir -p /root/.ssh && \
    chmod 0700 /root/.ssh && \
    ssh-keyscan github.com > /root/.ssh/known_hosts

# Add the keys and set permissions
RUN echo "$GITHUB_SSH" > /root/.ssh/id_rsa && \
    chmod 600 /root/.ssh/id_rsa

# Copy entire repo into container
COPY . .

# Compile with optimizations
RUN swift build \
    --enable-test-discovery \
    -c release

# Remove SSH keys
RUN rm -rf /root/.ssh/

And this could be built using docker build --build-arg GITHUB_SSH="$(cat ~/.ssh/github_rsa)" -t hiimtmac/cool-app .

Updated

Using the new --ssh option, we can adjust our Dockerfile to look like this:

FROM swift:5.3-focal

# Authorize SSH Host
RUN mkdir -p -m 700 /root/.ssh && \
    touch -m 600 /root/.ssh/known_hosts && \
    ssh-keyscan github.com > /root/.ssh/known_hosts

# Copy entire repo into container
COPY . .

# Compile with optimizations
RUN --mount=type=ssh,id=github swift build \
    --enable-test-discovery \
    -c release

And this could be built using docker build --ssh github=$HOME/.ssh/github_rsa -t hiimtmac/cool-app .. Any RUN command that needs SSH access (in this case the RUN swift build ... command) needs to have --mount=type=ssh,id=... in the beginning.

You can use the default SSH_AUTH_SOCK by pre-pending your RUN commands with just --mount=type=ssh and then building with docker build --ssh default -t hiimtmac/cool-app ., but I like to use a separate key because this is the key that GitHub actions will have access to, and then at no point is my personal private key ever involved in the build. Additionally, if you needed separate keys for github/bitbucket etc you would need to id/name them to differentiate and use them for different RUN commands.

Results

When building the old way with the original Dockerfile, you can see part of the private key leaking to stdout:

docker build --build-arg GITHUB_SSH="$(cat ~/.ssh/github_rsa)" -t hiimtmac/test .

 => [2/6] RUN mkdir -p /root/.ssh &&     chmod 0700 /root/.ssh &&     ssh-keyscan github.com > /root/.ssh/known_hosts           1.6s
 => [3/6] RUN echo "-----BEGIN OPENSSH PRIVATE KEY-----Hg  0.4slease                                                            2.0s
aaaabbbbccccddddeeeeffffgggghhhhiiiijjjjkkkkllllmmmmnnnnooooppppqqqqrr                                                          0.2s
1111222233334444555566667777888899990000aaaabbbbccccdddd   0.4slease                                                            2.1s
 => [4/6] COPY . .                                                                                                              0.2s

However building the new way with the new BuildKit and the updated Dockerfile, you can see that the private key is not in the stdout logs at all:

docker build --ssh github=$HOME/.ssh/github_rsa -t hiimtmac/test .

 => [2/4] RUN mkdir -p -m 700 /root/.ssh &&     touch -m 600 /root/.ssh/known_hosts &&     ssh-keyscan github.com > /root/.ssh/known_hosts           1.0s
 => [3/4] COPY . .                                                                                                                                   0.1s

Actions Changes

Original

From GitHub actions with private SPM dependencies, our actions workflow looked as such:

name: docker-deploy

on:
  push:
    tags:
      - "*"

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - name: Set Tag
        run: echo "TAG=${GITHUB_REF##*/}" >> $GITHUB_ENV
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          push: true
          tags: |
            hiimtmac/fake-repo:${{ env.TAG }}
            hiimtmac/fake-repo:develop
          build-args: |
            GITHUB_SSH=${{ secrets.CICD_SSH }}
      - name: Image digest
        run: echo ${{ steps.docker_build.outputs.digest }}

Updates

Using the new --ssh option, we can adjust our actions workflow to look like this:

name: docker-deploy

on:
  push:
    tags:
      - "*"

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - name: Set Tag
        run: echo "TAG=${GITHUB_REF##*/}" >> $GITHUB_ENV
      - name: Setup SSH Keys and known_hosts
        run: |
          mkdir -p ~/.ssh
          ssh-keyscan github.com >> ~/.ssh/known_hosts
          ssh-agent -a $SSH_AUTH_SOCK > /dev/null
          ssh-add - <<< "${{ secrets.SSH }}"
        env:
          SSH_AUTH_SOCK: /tmp/ssh_agent.sock
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push
        env:
          SSH_AUTH_SOCK: /tmp/ssh_agent.sock
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          push: true
          tags: |
            hiimtmac/fake-repo:${{ env.TAG }}
            hiimtmac/fake-repo:develop
          ssh: |
            github=${{ env.SSH_AUTH_SOCK }}
      - name: Image digest
        run: echo ${{ steps.docker_build.outputs.digest }}

Result

Similar to when building locally, the SSH key (or part of it) is no longer present in the logs!

Secrets Changes

The original format of the private key has to be preserved in the GitHub secret for this to work. In GitHub actions with private SPM dependencies we changed:

-----BEGIN OPENSSH PRIVATE KEY-----
aaabbbcccdddeeefffggghhhiiijjjkkklllmmm
nnnooopppqqqrrrssstttuuuvvvwwwxxxyyyzzz
abcdefghijklmnopqrstuvwxyz==
-----END OPENSSH PRIVATE KEY-----

to:

-----BEGIN OPENSSH PRIVATE KEY-----\naaabbbcccdddeeefffggghhhiiijjjkkklllmmm\nnnnooopppqqqrrrssstttuuuvvvwwwxxxyyyzzz\nabcdefghijklmnopqrstuvwxyz==\n-----END OPENSSH PRIVATE KEY-----

before uploading to GitHub as a secret. In this new workflow, we would want the original representation of the private key in the GitHub secret, not the one with all the newlines removed. Adjust this in GitHub before running the action.

Bonus

Here are a couple of additional steps you can take:

SPM Cache

Specifically for an SPM project you can cache dependencies so that between builds it will only re-fetch dependencies if your Package.resolved has changed.

...

# resolve dependencies - cached layer
COPY ./Package.* ./
RUN --mount=type=ssh,id=github swift package resolve

# Copy entire repo into container
COPY . .

# Compile with optimizations
RUN swift build \
    --enable-test-discovery \
    -c release

...

Note that the --mount=type=ssh,id=github part has been moved from the swift build command to the swift package resolve command as that is the one that needs it to resolve private dependencies.

GitHub Actions Tags

If you want to tag multiple versions based on the SEMVAR git tag, you can do so below like this:

# ...
- name: Set Tags
  run: |
    TAG=${GITHUB_REF##*/}
    COMP=(${TAG//./ })
    PATCH=$TAG
    MINOR=${COMP[0]}.${COMP[1]}
    MAJOR=${COMP[0]}
    echo "PATCH=$PATCH" >> $GITHUB_ENV
    echo "MINOR=$MINOR" >> $GITHUB_ENV
    echo "MAJOR=$MAJOR" >> $GITHUB_ENV
# ...
- name: Build and push
  env:
    SSH_AUTH_SOCK: /tmp/ssh_agent.sock
  id: docker_build
  uses: docker/build-push-action@v2
  with:
    push: true
    tags: |
      hiimtmac/fake-repo:${{ env.PATCH }}
      hiimtmac/fake-repo:${{ env.MINOR }}
      hiimtmac/fake-repo:${{ env.MAJOR }}
      hiimtmac/fake-repo:latest
      hiimtmac/fake-repo:develop
    ssh: |
      github=${{ env.SSH_AUTH_SOCK }}

# ...

For tag 2.4.5 this would create the following Docker images:

  • hiimtmac/fake-repo:2.4.5
  • hiimtmac/fake-repo:2.4
  • hiimtmac/fake-repo:2
  • hiimtmac/fake-repo:latest
  • hiimtmac/fake-repo:develop

In Closing

Some further reading about ssh and secrets in the new BuildKit can be found here, I used these resources to put together this solution:

As always, if you know of a better way to do something, please let me know!

Tagged With: