Gitea act_runner is HIGHLY insecure due to binding of docker.sock into all containers (= root on host) #167

New Issue

vitalif · 2023-04-30T07:50:25Z

vitalif commented

2023-04-30 07:50:25 +00:00

Hi!

act_runner is TERRIBLY INSECURE, it allows all job containers to:

access the docker daemon on host via the socket.
bind host directories into the container.

This allows any job to escape from the container and easily get root privileges on the runner host. Jobs are basically untrusted code so they have to be isolated correctly.

Some ways of how you can escape the container using docker.sock include:

starting a privileged container
using host namespaces
using a bind mount to host directory to overwrite system files

It may be rather OK for act itself because it's only a local testing tool, but for act_runner it's a real blocker.

You should forbid bind mounts in job descriptions (named volumes are probably ok) and remove docker socket access. The latter will probably break docker-related steps, so for them you'll have to use DinD or newer tools like Kaniko or Buildah/Podman.

I like Gitea and I use it for a long time so probably I'll try to patch and test it on my server and submit a PR, but anyway, in the current state Gitea Actions SHOULD NOT be used in production.

Hi! act_runner is **TERRIBLY INSECURE**, it allows all job containers to: 1) access the docker daemon on host via the socket. 2) bind host directories into the container. This allows any job to escape from the container and easily get root privileges on the runner host. Jobs are basically untrusted code so they have to be isolated correctly. Some ways of how you can escape the container using docker.sock include: - starting a privileged container - using host namespaces - using a bind mount to host directory to overwrite system files It may be rather OK for `act` itself because it's only a local testing tool, but for act_runner it's a real blocker. You should forbid bind mounts in job descriptions (named volumes are probably ok) and remove docker socket access. The latter will probably break docker-related steps, so for them you'll have to use DinD or newer tools like Kaniko or Buildah/Podman. I like Gitea and I use it for a long time so probably I'll try to patch and test it on my server and submit a PR, but anyway, in the current state Gitea Actions SHOULD NOT be used in production.

👍 3

vitalif commented

2023-04-30 08:03:21 +00:00

I duplicated it here https://github.com/go-gitea/gitea/issues/24438 due to importance of the issue

lunny commented

2023-04-30 10:36:56 +00:00

We have noticed that, since it's act internal feature, we need to change or act fork or upstream code.

lunny commented

2023-04-30 10:43:00 +00:00

This should be fixed by gitea/act#52

👍 2

vitalif commented

2023-04-30 22:32:53 +00:00

gitea/act#52 is not enough, you should also block user-specified bind mounts, at least unless privileged option is set to true in the runner configuration... and as I already said it'll probably require another base image, node:16-bullseye won't be enough because it doesn't contain docker or podman daemon...

ChristopherHX commented

2023-04-30 23:07:12 +00:00

This is also relevant #19 (comment).

Unless act starts rejecting

container: -self-hosted

You don't need any bind mount to get access to take the runner configuration over with a machine of an attacker.

There are multiple ways for bind mounts in act

container:
  image: ubuntu:latest
  options: -v /runner/.runner:/.runner # This is an example to bind mount the runner config to steal the runner config.

container:
  image: ubuntu:latest
  volumes:
  - /runner/.runner:/.runner

Can you still gain root access if you configure your docker daemon to run without root previleges e.g. rootless? At least you can steal the runner config to gain secrets of the attacked repo.

This is also relevant https://gitea.com/gitea/act_runner/issues/19#issuecomment-739221. Unless act starts rejecting ```yml container: -self-hosted ``` You don't need any bind mount to get access to take the runner configuration over with a machine of an attacker. There are multiple ways for bind mounts in act ```yml container: image: ubuntu:latest options: -v /runner/.runner:/.runner # This is an example to bind mount the runner config to steal the runner config. ``` ```yml container: image: ubuntu:latest volumes: - /runner/.runner:/.runner ``` Can you still gain root access if you configure your docker daemon to run without root previleges e.g. rootless? At least you can steal the runner config to gain secrets of the attacked repo.

vicamo commented

2023-05-01 10:17:35 +00:00

This is also relevant #19 (comment).

Unless act starts rejecting
container: -self-hosted

Somehow I think upstream act is abusing this attribute for a different meaning. The original meaning of self-hosted here is a label shared by all self-hosted action runners, and GitHub uses such label(s) to pick the right runner to execute the job. In act, it becomes "to run this job in host environment". In my opinion, Gitea CI runners should never support such thing due to security concerns, and so do customizable docker options and volume mounts.

> This is also relevant https://gitea.com/gitea/act_runner/issues/19#issuecomment-739221. > > Unless act starts rejecting > ```yml > container: -self-hosted > ``` Somehow I think upstream act is abusing this attribute for a different meaning. The original meaning of self-hosted here is a label shared by all self-hosted action runners, and GitHub uses such label(s) to pick the right runner to execute the job. In act, it becomes "to run this job in host environment". In my opinion, Gitea CI runners should never support such thing due to security concerns, and so do customizable docker options and volume mounts.

vitalif commented

2023-05-01 10:26:32 +00:00

-self-hosted is also a cool note, thanks

vitalif commented

2023-05-01 11:36:56 +00:00

By the way, another option that I'm currently investigating is using Kata containers :) it should even allow to use privileged containers without isolation problems. Bind mounts still have to be blocked though...

UPD Seems it's not easy to deploy :)

By the way, another option that I'm currently investigating is using Kata containers :) it should even allow to use privileged containers without isolation problems. Bind mounts still have to be blocked though... UPD Seems it's not easy to deploy :)

vitalif commented

2023-05-01 22:19:08 +00:00

Also I have an idea for running jobs that actually require privileged containers.
act_runner could leverage DOCKER_HOST=ssh://... and use VMs :)
And to make this support generic it can be implemented via simple bash scripts!
I.e. you specify two bash scripts:

to start a VM and return its address
to stop and purge a VM

Then act_runner just sets DOCKER_HOST=ssh://address-of-vm/ and it should probably be enough for act to function.
This way act_runner can easily support both local VMs and cloud VMs, and both privileged and regular jobs. A privileged job will be just ran in its own VM which will be purged after running it and that's all. Normal jobs will be able to share the same VM...

Also I have an idea for running jobs that actually require privileged containers. act_runner could leverage DOCKER_HOST=ssh://... and use VMs :) And to make this support generic it can be implemented via simple bash scripts! I.e. you specify two bash scripts: 1) to start a VM and return its address 2) to stop and purge a VM Then act_runner just sets DOCKER_HOST=ssh://address-of-vm/ and it should probably be enough for `act` to function. This way act_runner can easily support both local VMs and cloud VMs, and both privileged and regular jobs. A privileged job will be just ran in its own VM which will be purged after running it and that's all. Normal jobs will be able to share the same VM...

vitalif commented

2023-05-01 22:21:17 +00:00

(It seems Github Actions supports privileged jobs, for example FUSE works for me here: https://github.com/yandex-cloud/geesefs/actions/)

vicamo commented

2023-05-02 07:45:14 +00:00

Also I have an idea for running jobs that actually require privileged containers.
act_runner could leverage DOCKER_HOST=ssh://... and use VMs :)

The Travis CI has a keyword sudo, which would launch a fully virtualized VM that enables more features.

For those who have to run jobs within a privileged container/VM, maybe the right way is to add an ephemeral, VM based runner, and specify some keyword sudo/privileged in the runs-on attribute. This, in my opinion, can be done with the host-environment container type currently implemented by nektos/act.

> Also I have an idea for running jobs that actually require privileged containers. > act_runner could leverage DOCKER_HOST=ssh://... and use VMs :) The Travis CI has a keyword **sudo**, which would launch a fully virtualized VM that enables more features. For those who have to run jobs within a privileged container/VM, maybe the right way is to add an ephemeral, VM based runner, and specify some keyword sudo/privileged in the **runs-on** attribute. This, in my opinion, can be done with the host-environment container type currently implemented by nektos/act.

👍 1

vitalif commented

2023-05-02 16:53:33 +00:00

Yeah, I'm talking about the same thing.
Maybe "privileged" instead of "sudo" would be more intuitive for container users, but the idea is the same of course.
It will differ from Github - in Github Actions everything seems to be "sudo" by default - but I think it's an acceptable tradeoff :)
host-env, at the same time, isn't exactly what we need because it just executes commands locally and we need ssh.
Because if a local user has root access to the runner then he can steal the runner token.

Yeah, I'm talking about the same thing. Maybe "privileged" instead of "sudo" would be more intuitive for container users, but the idea is the same of course. It will differ from Github - in Github Actions everything seems to be "sudo" by default - but I think it's an acceptable tradeoff :) host-env, at the same time, isn't exactly what we need because it just executes commands locally and we need ssh. Because if a local user has root access to the runner then he can steal the runner token.

vicamo commented

2023-05-04 21:48:19 +00:00

Yeah, I'm talking about the same thing.
Maybe "privileged" instead of "sudo" would be more intuitive for container users, but the idea is the same of course.
It will differ from Github - in Github Actions everything seems to be "sudo" by default - but I think it's an acceptable tradeoff :)

Agree.

host-env, at the same time, isn't exactly what we need because it just executes commands locally and we need ssh.
Because if a local user has root access to the runner then he can steal the runner token.

Also agree. But just like GitLab has a "shell" executor type that allows users to create executors that are not currently supported in official releases (e.g. LXD), such executor type can be convenient, nice to have.

> Yeah, I'm talking about the same thing. > Maybe "privileged" instead of "sudo" would be more intuitive for container users, but the idea is the same of course. > It will differ from Github - in Github Actions everything seems to be "sudo" by default - but I think it's an acceptable tradeoff :) Agree. > host-env, at the same time, isn't exactly what we need because it just executes commands locally and we need ssh. > Because if a local user has root access to the runner then he can steal the runner token. Also agree. But just like GitLab has a "shell" executor type that allows users to create executors that are not currently supported in official releases (e.g. LXD), such executor type can be convenient, nice to have.

👍 1

MrHaroldA commented

2023-05-08 08:57:28 +00:00

Vegard IT has a 'dind' (Docker in Docker) image available for the Gitea act runner. Will a dind-image like this mitigate, or even resolve this security issue?

Vegard IT has a '[dind' (Docker in Docker) image](https://hub.docker.com/r/vegardit/gitea-act-runner/tags) available for the Gitea act runner. Will a dind-image like this mitigate, or even resolve this security issue?

vicamo commented

2023-05-09 09:13:39 +00:00

Vegard IT has a 'dind' (Docker in Docker) image available for the Gitea act runner. Will a dind-image like this mitigate, or even resolve this security issue?

With no modification to current gitea/act, nor DOCKER_HOST env injected, it will be ignored and prefer host docker daemon.

> Vegard IT has a '[dind' (Docker in Docker) image](https://hub.docker.com/r/vegardit/gitea-act-runner/tags) available for the Gitea act runner. Will a dind-image like this mitigate, or even resolve this security issue? With no modification to current gitea/act, nor DOCKER_HOST env injected, it will be ignored and prefer host docker daemon.

ChristopherHX commented

2023-05-09 09:44:03 +00:00

I looked into their Dockerfile of the dind image above and it is still vulnerable to the things reported here. However not as unsecure as not using dind.

Why?

The docker daemon of the dind image can still bind mount e.g. .runner and can steal your reusable runner configuration (you can use it on a system the attacker controls to spy data of other job requests). Due to a shared filesystem between the dind docker daemon and the runner.
This security update for act was not merged into this act fork ca9b783491, to prevent users from spawning non docker jobs.

With no modification to current gitea/act, nor DOCKER_HOST env injected, it will be ignored and prefer host docker daemon.

In this image is /var/run/docker.sock the dind docker instance, so it cannot be the host docker daemon if you start the image directly like it is designed.

After the security update lands in act_runner, you can create a two container act_runner image (set DOCKER_HOST) + a dind (rootless) image setup without a shared filesystem to mitigate all security issues.

All bind mounts are local (if not using tls to connect, otherwise /certs is shared but that's not really an issue) to the dind container and cannot access any files not explicitly copied from act_runner via docker cp.

A docker-compose.yml file would look similar to

version: "3"

networks:
  runner:
    external: false
volumes:
  docker-certs:
    driver: local
services:
  runner:
    build: .
    environment:
      - DOCKER_TLS_CERTDIR=/certs
      - DOCKER_CERT_PATH=/certs/client
      - DOCKER_TLS_VERIFY=1
      - DOCKER_HOST=tcp://docker:2376
    restart: always
    networks:
      - runner
    volumes:
      - docker-certs:/certs
    depends_on:
      - docker
  docker:
    image: docker:dind-rootless
    restart: always
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=/certs
    networks:
      - runner
    volumes:
      - docker-certs:/certs

It took me a while to get tls working with the official docker client, ~~I would need to check if act_runner can connect too~~ this env variables make the embedded docker client connect just fine via tcp using tls.
The runner service can be probably be replaced by the official gitea act_runner docker image and adding the env variables to configure the runner.

EDIT 10 May 2023 / 12:20 You find a more complete example with the official gitea + act_runner image in my other comment #178 (comment)

I looked into their Dockerfile of the dind image above and it is still vulnerable to the things reported here. However not as unsecure as not using dind. Why? - The docker daemon of the dind image can still bind mount e.g. `.runner` and can steal your reusable runner configuration (you can use it on a system the attacker controls to spy data of other job requests). Due to a shared filesystem between the dind docker daemon and the runner. - This security update for act was not merged into this act fork https://github.com/nektos/act/commit/ca9b783491b4ea1c5d67dbffdb95f34fd4a22573, to prevent users from spawning non docker jobs. > With no modification to current gitea/act, nor DOCKER_HOST env injected, it will be ignored and prefer host docker daemon. In this image is `/var/run/docker.sock` the dind docker instance, so it cannot be the host docker daemon if you start the image directly like it is designed. After the security update lands in act_runner, you can create a two container act_runner image (set `DOCKER_HOST`) + a dind (rootless) image setup without a shared filesystem to mitigate all security issues. All bind mounts are local (if not using tls to connect, otherwise /certs is shared but that's not really an issue) to the dind container and cannot access any files not explicitly copied from act_runner via docker cp. A docker-compose.yml file would look similar to ```yaml version: "3" networks: runner: external: false volumes: docker-certs: driver: local services: runner: build: . environment: - DOCKER_TLS_CERTDIR=/certs - DOCKER_CERT_PATH=/certs/client - DOCKER_TLS_VERIFY=1 - DOCKER_HOST=tcp://docker:2376 restart: always networks: - runner volumes: - docker-certs:/certs depends_on: - docker docker: image: docker:dind-rootless restart: always privileged: true environment: - DOCKER_TLS_CERTDIR=/certs networks: - runner volumes: - docker-certs:/certs ``` It took me a while to get tls working with the official docker client, ~~I would need to check if act_runner can connect too~~ this env variables make the embedded docker client connect just fine via tcp using tls. The runner service can be probably be replaced by the official gitea act_runner docker image and adding the env variables to configure the runner. **EDIT** 10 May 2023 / 12:20 You find a more complete example with the official gitea + act_runner image in my other comment https://gitea.com/gitea/act_runner/issues/178#issuecomment-740066

👍 2

vitalif commented

2023-05-10 10:03:24 +00:00

Thanks, that's interesting too
I still think that act_runner also needs VM support - I want privileged jobs to use kernel NBD in tests + not affect the host's kernel if for example something hangs inside io_uring. So I'm trying to implement it with bash scripts and raw qemu :)

Thanks, that's interesting too I still think that act_runner also needs VM support - I want privileged jobs to use kernel NBD in tests + not affect the host's kernel if for example something hangs inside io_uring. So I'm trying to implement it with bash scripts and raw qemu :)

ChristopherHX commented

2023-05-16 08:26:39 +00:00

I found another exploit to read any env variables of act_runner including GITEA_RUNNER_REGISTRATION_TOKEN if used to autoconfigure.

runs-on: ubuntu-latest
container:
  image: node:16-bullseye
  options: --network=bridge -e GITEA_RUNNER_REGISTRATION_TOKEN
steps:
- run: env

act_runner should delete from the act_runner env GITEA_RUNNER_REGISTRATION_TOKEN before invoking act.

I found another exploit to read any env variables of act_runner including `GITEA_RUNNER_REGISTRATION_TOKEN` if used to autoconfigure. ```yaml runs-on: ubuntu-latest container: image: node:16-bullseye options: --network=bridge -e GITEA_RUNNER_REGISTRATION_TOKEN steps: - run: env ``` act_runner should delete from the act_runner env GITEA_RUNNER_REGISTRATION_TOKEN before invoking act.

👍 3

wener commented

2023-05-23 17:18:03 +00:00

Here is another demo for using dind in k8s https://github.com/wenerme/kube-stub-cluster/blob/main/dev-system/services/docker/dind.yaml , the gitea runner use this dind docker

wolfogre commented

2023-06-21 08:36:24 +00:00

Since act runner supports dind now: gitea/act_runner:nightly-dind-rootless or gitea/act_runner:latest-dind-rootless, see https://hub.docker.com/r/gitea/act_runner/tags

I think it's time to close this issue.

Since act runner supports dind now: `gitea/act_runner:nightly-dind-rootless` or `gitea/act_runner:latest-dind-rootless`, see https://hub.docker.com/r/gitea/act_runner/tags I think it's time to close this issue.

wolfogre closed this issue

2023-06-21 08:36:24 +00:00

johnwmail commented

2023-10-03 08:30:58 +00:00

Since act runner supports dind now: gitea/act_runner:nightly-dind-rootless or gitea/act_runner:latest-dind-rootless, see https://hub.docker.com/r/gitea/act_runner/tags

I think it's time to close this issue.

Any links/example for how to use those dind-rootless image? thanks.

> Since act runner supports dind now: `gitea/act_runner:nightly-dind-rootless` or `gitea/act_runner:latest-dind-rootless`, see https://hub.docker.com/r/gitea/act_runner/tags > > I think it's time to close this issue. Any links/example for how to use those dind-rootless image? thanks.

Sign in to join this conversation.

No Milestone

No Assignees

8 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gitea/act_runner#167