act_runner fails to run without global internet connectivity #64

New Issue

paultag · 2023-03-21T13:55:35Z

paultag commented

2023-03-21 13:55:35 +00:00

Thanks for writing act_runner, very excited to play with this feature!

My friends and I have a LAN that we share over IPSec between our houses. That VLAN (at least on my network) has no global internet connectivity, just connectivity to my friends' houses.

When running an 'act_runner' against my hosted git server, I get the following (bizzare?) error:

FATA[2023-03-21T09:51:28-04:00] dial udp 8.8.8.8:80: connect: network is unreachable

My /etc/resolv.conf doesn't use 8.8.8.8 -- my local RR is at 10.42.3.2. Additionally, it's port 80 (not 53?) but it's still UDP. I greped the code, and tried to use logs to pinpoint where this is happening, but I believe it's happening on a background goroutine (maybe even from a dependency) that's fairly hard to track down.

Excited to try out gitea actions!

Thanks for writing act_runner, very excited to play with this feature! My friends and I have a LAN that we share over IPSec between our houses. That VLAN (at least on my network) has no global internet connectivity, just connectivity to my friends' houses. When running an 'act_runner' against my hosted git server, I get the following (bizzare?) error: ``` FATA[2023-03-21T09:51:28-04:00] dial udp 8.8.8.8:80: connect: network is unreachable ``` My `/etc/resolv.conf` doesn't use `8.8.8.8` -- my local RR is at `10.42.3.2`. Additionally, it's port 80 (not 53?) but it's still UDP. I greped the code, and tried to use logs to pinpoint where this is happening, but I believe it's happening on a background goroutine (maybe even from a dependency) that's fairly hard to track down. Excited to try out gitea actions!

👍 1

lunny commented

2023-03-21 15:13:21 +00:00

Do you run act_runner in docker mode or host mode? If it's docker mode, the dns should come from the container.

paultag commented

2023-03-21 15:23:39 +00:00

@lunny I'm not sure, but I tried again with Docker disabled on the host, and tried it again with a fresh build (without my old log messages)

# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

# ./act_runner daemon
INFO[0000] Starting runner daemon                       
INFO[2023-03-21T11:17:11-04:00] gc: 2023-03-21 11:17:11.262526893 -0400 EDT m=+0.023494060  module=cache_request
FATA[2023-03-21T11:17:11-04:00] dial udp 8.8.8.8:80: connect: network is unreachable

And, with Docker running:

# ./act_runner daemon
INFO[0000] Starting runner daemon                       
INFO[2023-03-21T11:18:41-04:00] gc: 2023-03-21 11:18:41.023704484 -0400 EDT m=+0.019884907  module=cache_request
FATA[2023-03-21T11:18:41-04:00] dial udp 8.8.8.8:80: connect: network is unreachable 
# docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

# cat /etc/resolv.conf | grep namese
nameserver 10.42.3.2

My hunch is it's somewhere in the binary, here's some output that looks sus:

strings act_runner | grep 8\.8\.8\.8
[...]
) VALUES () Values (, bound = , limit = --- mutex:.dockercfg.gitignore.localhost/bin/sleep/dev/stdin/etc/hosts/info/refs/networks//services//setgroups/system/df/toolcache012345678912207031252006-01-0261035156258.8.8.8:80:authority<no value>ARCHIVELOGASENSITIVEASSIGNMENTASYMMETRICATTRIBUTESAUTOEXTENDAccessTimeAdditionalBIT_LENGTHBad varintCACHE-PATHCHARACTERSCHECKPOINTCONNECTIONCONSTRAINTCONVERSIONCOVAR_SAMPChangeTimeChorasmianClassCHAOSClassCSNETCodeCommitConnectionContent-IdCurve25519DAY_MINUTEDAY_SECONDDEALLOCATEDEFERRABLEDELIMITERSDENSE_RANKDESCRIPTORDICTIONARYDISCONNECTDSA-SHA256DeprecatedDevanagariDockerfileDoubleKindECDSA-SHA1END_STREAMENV[...]

(Specifically, there's a 8.8.8.8:80 in there, which is not something I've seen before -- I don't even think 8.8.8.8 works on port 80? Citiation needed?)

@lunny I'm not sure, but I tried again with Docker disabled on the host, and tried it again with a fresh build (without my old log messages) ``` # docker ps Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? ``` ``` # ./act_runner daemon INFO[0000] Starting runner daemon INFO[2023-03-21T11:17:11-04:00] gc: 2023-03-21 11:17:11.262526893 -0400 EDT m=+0.023494060 module=cache_request FATA[2023-03-21T11:17:11-04:00] dial udp 8.8.8.8:80: connect: network is unreachable ``` And, with Docker running: ``` # ./act_runner daemon INFO[0000] Starting runner daemon INFO[2023-03-21T11:18:41-04:00] gc: 2023-03-21 11:18:41.023704484 -0400 EDT m=+0.019884907 module=cache_request FATA[2023-03-21T11:18:41-04:00] dial udp 8.8.8.8:80: connect: network is unreachable # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ``` ``` # cat /etc/resolv.conf | grep namese nameserver 10.42.3.2 ``` My hunch is it's somewhere in the binary, here's some output that looks sus: ``` strings act_runner | grep 8\.8\.8\.8 [...] ) VALUES () Values (, bound = , limit = --- mutex:.dockercfg.gitignore.localhost/bin/sleep/dev/stdin/etc/hosts/info/refs/networks//services//setgroups/system/df/toolcache012345678912207031252006-01-0261035156258.8.8.8:80:authority<no value>ARCHIVELOGASENSITIVEASSIGNMENTASYMMETRICATTRIBUTESAUTOEXTENDAccessTimeAdditionalBIT_LENGTHBad varintCACHE-PATHCHARACTERSCHECKPOINTCONNECTIONCONSTRAINTCONVERSIONCOVAR_SAMPChangeTimeChorasmianClassCHAOSClassCSNETCodeCommitConnectionContent-IdCurve25519DAY_MINUTEDAY_SECONDDEALLOCATEDEFERRABLEDELIMITERSDENSE_RANKDESCRIPTORDICTIONARYDISCONNECTDSA-SHA256DeprecatedDevanagariDockerfileDoubleKindECDSA-SHA1END_STREAMENV[...] ``` (Specifically, there's a `8.8.8.8:80` in there, which is not something I've seen before -- I don't even think 8.8.8.8 works on port 80? Citiation needed?)

paultag commented

2023-03-21 15:42:34 +00:00

https://gitea.com/gitea/act/src/branch/main/pkg/common/outbound_ip.go#L12

Looks to be the source of pain here.

https://gitea.com/gitea/act/src/branch/main/pkg/common/outbound_ip.go#L12 Looks to be the source of pain here.

wolfogre commented

2023-03-22 06:40:49 +00:00

Good catch! I know exactly why, I will fix it later.

👍 1

wolfogre added the

kind

bug

label 2023-03-22 06:41:10 +00:00

paultag commented

2023-03-22 07:08:15 +00:00

amazing! thank you!

some things id love to plant in the back of your head when you're looking at it:

This was hard to debug since it would fatal in a printf from the callsite in act_runner. maybe returning an error would help callers better realize this can fail and crash the program, and also allow the runner to fall back on a provided IP/fqdn.
hostname --ip-address is implemented in a fairly complex - abet predictable - way. https://sources.debian.org/src/hostname/3.23+nmu1/hostname.c/#L359 - maybe something there can help with a v2 of that function
Some runners may be on ipv6 only networks. I have one such vlan too -- or at minimum use ipv6 to communicate with gitea not ipv4.
some runners may have an interface that's not the default route which is used to communicate with the gitea server

providing some sort of escape hatch would be very nice! thanks for your work!

FWIW: my two cents, a minimal change here maybe worth considering - rather then UDP dialing a global address, tcp dial the configured server on the configured port, and using the local IP from that

amazing! thank you! some things id love to plant in the back of your head when you're looking at it: 1) This was hard to debug since it would fatal in a printf from the callsite in act_runner. maybe returning an error would help callers better realize this can fail and crash the program, and also allow the runner to fall back on a provided IP/fqdn. 2) `hostname --ip-address` is implemented in a fairly complex - abet predictable - way. https://sources.debian.org/src/hostname/3.23+nmu1/hostname.c/#L359 - maybe something there can help with a v2 of that function 3) Some runners may be on ipv6 only networks. I have one such vlan too -- or at minimum use ipv6 to communicate with gitea not ipv4. 4) some runners may have an interface that's not the default route which is used to communicate with the gitea server providing some sort of escape hatch would be very nice! thanks for your work! FWIW: my two cents, a minimal change here maybe worth considering - rather then UDP dialing a global address, tcp dial the configured server on the configured port, and using the local IP from that

wolfogre referenced a pull request that will close this issue

2023-03-24 08:55:38 +00:00

Get outbound IP in multiple ways or disable cache server if failed to init #74

wolfogre closed this issue

2023-03-24 09:55:13 +00:00

wolfogre referenced this issue from a commit

2023-03-24 09:55:13 +00:00

Get outbound IP in multiple ways or disable cache server if failed to init (#74)

wolfogre commented

2023-03-24 09:56:12 +00:00

Reopen because #74 is still not ideal. It makes more sense to use the gateway IP address of container network as outbound IP of cache server. However, this requires act to cooperate, some think like:

act creates the network for new container, and returns the network to runner.
runner extracts the gateway IP in the network.
runner uses the gateway IP as outbound IP, and pass it to act as cache server endpoint.
act It continues to create the container with the created network.

Reopen because #74 is still not ideal. It makes more sense to use the gateway IP address of container network as outbound IP of cache server. However, this requires act to cooperate, some think like: - act creates the network for new container, and returns the network to runner. - runner extracts the gateway IP in the network. - runner uses the gateway IP as outbound IP, and pass it to act as cache server endpoint. - act It continues to create the container with the created network.

wolfogre reopened this issue

2023-03-24 09:56:12 +00:00

Sign in to join this conversation.

No Milestone

No Assignees

3 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gitea/act_runner#64