gitea/act_runner
57
117
Fork 91
Code Issues 169 Pull Requests 10 Actions Releases 21 Activity

act_runner fails to run without global internet connectivity #64

New Issue
Open
opened 2023-03-21 13:55:35 +00:00 by paultag · 6 comments
paultag commented 2023-03-21 13:55:35 +00:00
Copy Link

Thanks for writing act_runner, very excited to play with this feature!

My friends and I have a LAN that we share over IPSec between our houses. That VLAN (at least on my network) has no global internet connectivity, just connectivity to my friends' houses.

When running an 'act_runner' against my hosted git server, I get the following (bizzare?) error:

FATA[2023-03-21T09:51:28-04:00] dial udp 8.8.8.8:80: connect: network is unreachable

My /etc/resolv.conf doesn't use 8.8.8.8 -- my local RR is at 10.42.3.2. Additionally, it's port 80 (not 53?) but it's still UDP. I greped the code, and tried to use logs to pinpoint where this is happening, but I believe it's happening on a background goroutine (maybe even from a dependency) that's fairly hard to track down.

Excited to try out gitea actions!

Thanks for writing act_runner, very excited to play with this feature! My friends and I have a LAN that we share over IPSec between our houses. That VLAN (at least on my network) has no global internet connectivity, just connectivity to my friends' houses. When running an 'act_runner' against my hosted git server, I get the following (bizzare?) error: ``` FATA[2023-03-21T09:51:28-04:00] dial udp 8.8.8.8:80: connect: network is unreachable ``` My `/etc/resolv.conf` doesn't use `8.8.8.8` -- my local RR is at `10.42.3.2`. Additionally, it's port 80 (not 53?) but it's still UDP. I greped the code, and tried to use logs to pinpoint where this is happening, but I believe it's happening on a background goroutine (maybe even from a dependency) that's fairly hard to track down. Excited to try out gitea actions!
👍 1
lunny commented 2023-03-21 15:13:21 +00:00
Owner
Copy Link

Do you run act_runner in docker mode or host mode? If it's docker mode, the dns should come from the container.

Do you run act_runner in docker mode or host mode? If it's docker mode, the dns should come from the container.
paultag commented 2023-03-21 15:23:39 +00:00
Author
Copy Link

@lunny I'm not sure, but I tried again with Docker disabled on the host, and tried it again with a fresh build (without my old log messages)

# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

# ./act_runner daemon
INFO[0000] Starting runner daemon                       
INFO[2023-03-21T11:17:11-04:00] gc: 2023-03-21 11:17:11.262526893 -0400 EDT m=+0.023494060  module=cache_request
FATA[2023-03-21T11:17:11-04:00] dial udp 8.8.8.8:80: connect: network is unreachable

And, with Docker running:

# ./act_runner daemon
INFO[0000] Starting runner daemon                       
INFO[2023-03-21T11:18:41-04:00] gc: 2023-03-21 11:18:41.023704484 -0400 EDT m=+0.019884907  module=cache_request
FATA[2023-03-21T11:18:41-04:00] dial udp 8.8.8.8:80: connect: network is unreachable 
# docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

# cat /etc/resolv.conf | grep namese
nameserver 10.42.3.2

My hunch is it's somewhere in the binary, here's some output that looks sus:

strings act_runner | grep 8\.8\.8\.8
[...]
) VALUES () Values (, bound = , limit = --- mutex:.dockercfg.gitignore.localhost/bin/sleep/dev/stdin/etc/hosts/info/refs/networks//services//setgroups/system/df/toolcache012345678912207031252006-01-0261035156258.8.8.8:80:authority<no value>ARCHIVELOGASENSITIVEASSIGNMENTASYMMETRICATTRIBUTESAUTOEXTENDAccessTimeAdditionalBIT_LENGTHBad varintCACHE-PATHCHARACTERSCHECKPOINTCONNECTIONCONSTRAINTCONVERSIONCOVAR_SAMPChangeTimeChorasmianClassCHAOSClassCSNETCodeCommitConnectionContent-IdCurve25519DAY_MINUTEDAY_SECONDDEALLOCATEDEFERRABLEDELIMITERSDENSE_RANKDESCRIPTORDICTIONARYDISCONNECTDSA-SHA256DeprecatedDevanagariDockerfileDoubleKindECDSA-SHA1END_STREAMENV[...]

(Specifically, there's a 8.8.8.8:80 in there, which is not something I've seen before -- I don't even think 8.8.8.8 works on port 80? Citiation needed?)

@lunny I'm not sure, but I tried again with Docker disabled on the host, and tried it again with a fresh build (without my old log messages) ``` # docker ps Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? ``` ``` # ./act_runner daemon INFO[0000] Starting runner daemon INFO[2023-03-21T11:17:11-04:00] gc: 2023-03-21 11:17:11.262526893 -0400 EDT m=+0.023494060 module=cache_request FATA[2023-03-21T11:17:11-04:00] dial udp 8.8.8.8:80: connect: network is unreachable ``` And, with Docker running: ``` # ./act_runner daemon INFO[0000] Starting runner daemon INFO[2023-03-21T11:18:41-04:00] gc: 2023-03-21 11:18:41.023704484 -0400 EDT m=+0.019884907 module=cache_request FATA[2023-03-21T11:18:41-04:00] dial udp 8.8.8.8:80: connect: network is unreachable # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ``` ``` # cat /etc/resolv.conf | grep namese nameserver 10.42.3.2 ``` My hunch is it's somewhere in the binary, here's some output that looks sus: ``` strings act_runner | grep 8\.8\.8\.8 [...] ) VALUES () Values (, bound = , limit = --- mutex:.dockercfg.gitignore.localhost/bin/sleep/dev/stdin/etc/hosts/info/refs/networks//services//setgroups/system/df/toolcache012345678912207031252006-01-0261035156258.8.8.8:80:authority<no value>ARCHIVELOGASENSITIVEASSIGNMENTASYMMETRICATTRIBUTESAUTOEXTENDAccessTimeAdditionalBIT_LENGTHBad varintCACHE-PATHCHARACTERSCHECKPOINTCONNECTIONCONSTRAINTCONVERSIONCOVAR_SAMPChangeTimeChorasmianClassCHAOSClassCSNETCodeCommitConnectionContent-IdCurve25519DAY_MINUTEDAY_SECONDDEALLOCATEDEFERRABLEDELIMITERSDENSE_RANKDESCRIPTORDICTIONARYDISCONNECTDSA-SHA256DeprecatedDevanagariDockerfileDoubleKindECDSA-SHA1END_STREAMENV[...] ``` (Specifically, there's a `8.8.8.8:80` in there, which is not something I've seen before -- I don't even think 8.8.8.8 works on port 80? Citiation needed?)
paultag commented 2023-03-21 15:42:34 +00:00
Author
Copy Link

https://gitea.com/gitea/act/src/branch/main/pkg/common/outbound_ip.go#L12

Looks to be the source of pain here.

https://gitea.com/gitea/act/src/branch/main/pkg/common/outbound_ip.go#L12 Looks to be the source of pain here.
wolfogre commented 2023-03-22 06:40:49 +00:00
Owner
Copy Link

Good catch! I know exactly why, I will fix it later.

Good catch! I know exactly why, I will fix it later.
👍 1
wolfogre added the
kind
bug
 label 2023-03-22 06:41:10 +00:00
paultag commented 2023-03-22 07:08:15 +00:00
Author
Copy Link

amazing! thank you!

some things id love to plant in the back of your head when you're looking at it:

  1. This was hard to debug since it would fatal in a printf from the callsite in act_runner. maybe returning an error would help callers better realize this can fail and crash the program, and also allow the runner to fall back on a provided IP/fqdn.

  2. hostname --ip-address is implemented in a fairly complex - abet predictable - way. https://sources.debian.org/src/hostname/3.23+nmu1/hostname.c/#L359 - maybe something there can help with a v2 of that function

  3. Some runners may be on ipv6 only networks. I have one such vlan too -- or at minimum use ipv6 to communicate with gitea not ipv4.

  4. some runners may have an interface that's not the default route which is used to communicate with the gitea server

providing some sort of escape hatch would be very nice! thanks for your work!

FWIW: my two cents, a minimal change here maybe worth considering - rather then UDP dialing a global address, tcp dial the configured server on the configured port, and using the local IP from that

amazing! thank you! some things id love to plant in the back of your head when you're looking at it: 1) This was hard to debug since it would fatal in a printf from the callsite in act_runner. maybe returning an error would help callers better realize this can fail and crash the program, and also allow the runner to fall back on a provided IP/fqdn. 2) `hostname --ip-address` is implemented in a fairly complex - abet predictable - way. https://sources.debian.org/src/hostname/3.23+nmu1/hostname.c/#L359 - maybe something there can help with a v2 of that function 3) Some runners may be on ipv6 only networks. I have one such vlan too -- or at minimum use ipv6 to communicate with gitea not ipv4. 4) some runners may have an interface that's not the default route which is used to communicate with the gitea server providing some sort of escape hatch would be very nice! thanks for your work! FWIW: my two cents, a minimal change here maybe worth considering - rather then UDP dialing a global address, tcp dial the configured server on the configured port, and using the local IP from that
wolfogre commented 2023-03-24 09:56:12 +00:00
Owner
Copy Link

Reopen because #74 is still not ideal. It makes more sense to use the gateway IP address of container network as outbound IP of cache server. However, this requires act to cooperate, some think like:

  • act creates the network for new container, and returns the network to runner.
  • runner extracts the gateway IP in the network.
  • runner uses the gateway IP as outbound IP, and pass it to act as cache server endpoint.
  • act It continues to create the container with the created network.
Reopen because #74 is still not ideal. It makes more sense to use the gateway IP address of container network as outbound IP of cache server. However, this requires act to cooperate, some think like: - act creates the network for new container, and returns the network to runner. - runner extracts the gateway IP in the network. - runner uses the gateway IP as outbound IP, and pass it to act as cache server endpoint. - act It continues to create the container with the created network.
wolfogre reopened this issue 2023-03-24 09:56:12 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
kind
bug
Something is not working

  
kind
build

  
kind/compatible
  
kind
dependencies

  
kind
docs

  
kind
enhancement
New feature

  
kind
feature

  
kind
help wanted
Need some help

  
kind
proposal

  
kind
refactor

  
related
act
Related to act.

  
related
environment
Related to the environment.

  
related
exec

  
related
gitea
Related to Gitea.

  
related
workflow
Related to the workflow file or actions scripts.

  
reviewed
confirmed
confirmed issue

  
reviewed
duplicate
This issue or pull request already exists

  
reviewed
invalid
Something is wrong

  
reviewed
needs feedback

  
reviewed
wontfix
This won't be fixed

  
reviewed
workaround

No Label
kind
bug
kind
build
kind/compatible
kind
dependencies
kind
docs
kind
enhancement
kind
feature
kind
help wanted
kind
proposal
kind
refactor
related
act
related
environment
related
exec
related
gitea
related
workflow
reviewed
confirmed
reviewed
duplicate
reviewed
invalid
reviewed
needs feedback
reviewed
wontfix
reviewed
workaround
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gitea/act_runner#64
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?