gitea/helm-chart
45
54
Fork 128
Code Issues 31 Pull Requests 7 Actions Releases 87 Activity

Crazy number of Connection closed by logs #258

New Issue
Open
opened 2021-12-04 09:51:23 +00:00 by ghazan3 · 10 comments
ghazan3 commented 2021-12-04 09:51:23 +00:00
Copy Link

Our log server record huge number of "Connection closed by 10.X.Y.Z port ABC" events. They are almost 3 events per second.

My first thought was bruteforce attempt but it is all from a local IP and happening every day all the time.

Is there a way to disable it? or figure out what is making these connections?

	Dec 4, 2021 @ 20:33:21.587	gitea	Connection closed by 10.104.0.7 port 36026
	Dec 4, 2021 @ 20:33:21.381	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:21.381	gitea	Connection closed by 10.104.0.12 port 35286
	Dec 4, 2021 @ 20:33:21.275	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:21.275	gitea	Connection closed by 10.104.0.3 port 63140
	Dec 4, 2021 @ 20:33:18.586	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:18.586	gitea	Connection closed by 10.104.0.7 port 36020
	Dec 4, 2021 @ 20:33:18.384	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:18.384	gitea	Connection closed by 10.104.0.12 port 35280
    Dec 4, 2021 @ 20:33:18.274	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:18.274	gitea	Connection closed by 10.104.0.3 port 63134
	Dec 4, 2021 @ 20:33:15.586	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:15.586	gitea	Connection closed by 10.104.0.7 port 36014
	Dec 4, 2021 @ 20:33:15.384	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:15.384	gitea	Connection closed by 10.104.0.12 port 35274
	Dec 4, 2021 @ 20:33:15.270	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:15.270	gitea	Connection closed by 10.104.0.3 port 63128
	Dec 4, 2021 @ 20:33:12.581	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:12.581	gitea	Connection closed by 10.104.0.7 port 36008

Relevant part of values.yaml

image:
  repository: gitea/gitea
  tag: 1.15.4
  
log:
      LEVEL: error
    log.name:
      COLORIZE: false
    metrics:
      enabled: false
      serviceMonitor:
        enabled: false

We had a dedicated loadbalancer on port 22 and map to ClusterIP of gitea-ssh service. We also use Cloudflare Spectrum service to tunnel SSH connections. Cloudflare does not show there are any connection to this service from public. So I am guessing these are internal connection or false logs?

Our log server record huge number of "Connection closed by 10.X.Y.Z port ABC" events. They are almost 3 events per second. My first thought was bruteforce attempt but it is all from a local IP and happening every day all the time. Is there a way to disable it? or figure out what is making these connections? ``` Dec 4, 2021 @ 20:33:21.587 gitea Connection closed by 10.104.0.7 port 36026 Dec 4, 2021 @ 20:33:21.381 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:21.381 gitea Connection closed by 10.104.0.12 port 35286 Dec 4, 2021 @ 20:33:21.275 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:21.275 gitea Connection closed by 10.104.0.3 port 63140 Dec 4, 2021 @ 20:33:18.586 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:18.586 gitea Connection closed by 10.104.0.7 port 36020 Dec 4, 2021 @ 20:33:18.384 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:18.384 gitea Connection closed by 10.104.0.12 port 35280 Dec 4, 2021 @ 20:33:18.274 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:18.274 gitea Connection closed by 10.104.0.3 port 63134 Dec 4, 2021 @ 20:33:15.586 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:15.586 gitea Connection closed by 10.104.0.7 port 36014 Dec 4, 2021 @ 20:33:15.384 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:15.384 gitea Connection closed by 10.104.0.12 port 35274 Dec 4, 2021 @ 20:33:15.270 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:15.270 gitea Connection closed by 10.104.0.3 port 63128 Dec 4, 2021 @ 20:33:12.581 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:12.581 gitea Connection closed by 10.104.0.7 port 36008 ``` Relevant part of values.yaml ``` image: repository: gitea/gitea tag: 1.15.4 log: LEVEL: error log.name: COLORIZE: false metrics: enabled: false serviceMonitor: enabled: false ``` We had a dedicated loadbalancer on port 22 and map to ClusterIP of gitea-ssh service. We also use Cloudflare Spectrum service to tunnel SSH connections. Cloudflare does not show there are any connection to this service from public. So I am guessing these are internal connection or false logs?
👍 2
Sindvero commented 2022-01-07 21:14:47 +00:00
Copy Link

I also have the same issue using the 1.15.8 release. It doesn't cause any issue that I can see nor connection, so I'm not sure where the logs are coming from.

I also have the same issue using the 1.15.8 release. It doesn't cause any issue that I can see nor connection, so I'm not sure where the logs are coming from.
pi3ch commented 2022-02-13 06:54:53 +00:00
Contributor
Copy Link

Any luck on this? Same issue with 1.16.1

Any luck on this? Same issue with 1.16.1
pi3ch commented 2022-02-18 00:54:56 +00:00
Contributor
Copy Link

I've disabled connection from public. Still getting the logs. It seems to be an inner cluster connection/prob against SSH service.

I've disabled connection from public. Still getting the logs. It seems to be an inner cluster connection/prob against SSH service.
pat-s commented 2022-03-05 16:04:43 +00:00
Member
Copy Link

Looks similar as #224

Looks similar as #224
👍 1
pi3ch commented 2022-03-27 13:39:51 +00:00
Contributor
Copy Link

These logs are related to openssh loglevel (not gitea). See a workaround at #224 (comment)

These logs are related to openssh loglevel (not gitea). See a workaround at https://gitea.com/gitea/helm-chart/issues/224#issuecomment-694372
pi3ch commented 2022-03-27 13:58:33 +00:00
Contributor
Copy Link

Connection attempts are from multiple nodes: cilium, csi or kube-proxy that are not SSH compatible.

Connection attempts are from multiple nodes: `cilium`, `csi` or `kube-proxy` that are not SSH compatible.
justusbunsi commented 2023-09-22 15:37:01 +00:00
Member
Copy Link

@pat-s Isn't this already resolved due to the configurable SSH log level or do I miss something obvious about this issue? #358

@pat-s Isn't this already resolved due to the configurable SSH log level or do I miss something obvious about this issue? #358
pi3ch commented 2023-09-22 23:26:05 +00:00
Contributor
Copy Link

@justusbunsi I can confirm by setting SSH log level (https://gitea.com/pi3ch/helm-chart/src/branch/main/values.yaml#L352) to FATAL the logs are not shown but I don't think this has addressed the underlying problem.

The problem only start to appear when there is LoadBalancer exposing SSH to public.

@justusbunsi I can confirm by setting SSH log level (https://gitea.com/pi3ch/helm-chart/src/branch/main/values.yaml#L352) to FATAL the logs are not shown but I don't think this has addressed the underlying problem. The problem only start to appear when there is LoadBalancer exposing SSH to public.
pat-s commented 2023-11-03 07:17:10 +00:00
Member
Copy Link

@justusbunsi I wonder if should change the default to ssh.logLevel: FATAL? Otherwise these messages only swamp the logs and might cause more concern than anything else.

@justusbunsi I wonder if should change the default to `ssh.logLevel: FATAL`? Otherwise these messages only swamp the logs and might cause more concern than anything else.
justusbunsi commented 2023-11-06 19:31:51 +00:00
Member
Copy Link

@justusbunsi I wonder if should change the default to ssh.logLevel: FATAL? Otherwise these messages only swamp the logs and might cause more concern than anything else.

Sounds good.
EDIT: I was too lax with my response. It would reduce the logged events but the underlying issue would still be there. And you don't see warnings anymore. Default log levels shouldn't mute warnings. IMO.

> @justusbunsi I wonder if should change the default to `ssh.logLevel: FATAL`? Otherwise these messages only swamp the logs and might cause more concern than anything else. ~~Sounds good.~~ EDIT: I was too lax with my response. It would reduce the logged events but the underlying issue would still be there. And you don't see warnings anymore. Default log levels shouldn't mute warnings. IMO.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
has
backport

  
in progress

   
invalid
  
kind
breaking

  
kind
bug

  
kind
build

  
kind
dependency

  
kind
deployment

  
kind
docs

  
kind
enhancement

  
kind
feature

  
kind
lint

  
kind
proposal

  
kind
question

  
kind
refactor

  
kind
security

  
kind
testing

  
kind
translation

  
kind
ui

  
need
backport

  
priority
critical

  
priority
low

  
priority
maybe

  
priority
medium

  
reviewed
duplicate

  
reviewed
invalid

  
reviewed
wontfix

  
skip-changelog
  
status
blocked

  
status
needs-feedback

  
status
needs-reviews

  
status
wip

  
upstream
gitea

  
upstream
other

No Label
has
backport
in progress
invalid
kind
breaking
kind
bug
kind
build
kind
dependency
kind
deployment
kind
docs
kind
enhancement
kind
feature
kind
lint
kind
proposal
kind
question
kind
refactor
kind
security
kind
testing
kind
translation
kind
ui
need
backport
priority
critical
priority
low
priority
maybe
priority
medium
reviewed
duplicate
reviewed
invalid
reviewed
wontfix
skip-changelog
status
blocked
status
needs-feedback
status
needs-reviews
status
wip
upstream
gitea
upstream
other
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
5 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gitea/helm-chart#258
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?