gitea/helm-chart
45
55
Fork 125
Code Issues 32 Pull Requests 8 Actions Releases 87 Activity

Gitea-HA multiple replica support #205

Closed
luhahn wants to merge 8 commits from gitea-ha into main
pull from: gitea-ha
merge into: gitea:main
Conversation 29 Commits 8 Files Changed 6 +270 -3
luhahn commented 2021-07-09 14:45:01 +00:00
Member
Copy Link

This PR provides support to run Gitea with multiple replicas out of the box.

You will need to:

  • enable redis in the values
  • update replicaCount > 1
  • provide a readWriteMany PVC for gitea as existingVolume
memcached:
  enabled: false
  
redis:
  enabled: true
    
replicaCount: 2

gitea:
  persistence:
    enabled: true
    existingClaim: RWX-PVC-Gitea

This PR also adds postgresql-ha and redis-cluster as dependency to provide real HA support.

Fixes: #206

This PR provides support to run Gitea with multiple replicas out of the box. You will need to: - enable redis in the values - update replicaCount > 1 - provide a readWriteMany PVC for gitea as existingVolume ```yaml memcached: enabled: false redis: enabled: true replicaCount: 2 gitea: persistence: enabled: true existingClaim: RWX-PVC-Gitea ``` This PR also adds postgresql-ha and redis-cluster as dependency to provide real HA support. Fixes: #206
❤️ 5
luhahn added the
kind
breaking
kind
feature
 labels 2021-07-12 09:51:27 +00:00
luhahn changed title from WIP: Gitea-HA multiple replica support to Gitea-HA multiple replica support 2021-07-12 09:51:37 +00:00
luhahn commented 2021-07-12 10:46:46 +00:00
Author
Member
Copy Link

Also moved the gitea.database.builtIn and gitea.cache.builtIn section into the dependencies directly

Also moved the gitea.database.builtIn and gitea.cache.builtIn section into the dependencies directly
viceice reviewed 2021-07-14 07:24:02 +00:00
viceice left a comment
Contributor
Copy Link

What happens when memcached fails?

What happens when memcached fails?
templates/gitea/config.yaml Outdated
@ -135,0 +153,4 @@
{{- $_ := set .Values.gitea.config.queue "TYPE" "redis" -}}
{{- $_ := set .Values.gitea.config.queue "CONN_STR" (include "redis.dns" .) -}}
{{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_QUEUE_TYPE" "redis" -}}
{{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_QUEUE_CONN_STR" (include "redis.dns" .) -}}
viceice commented 2021-07-14 07:21:27 +00:00
Contributor
Copy Link

You can remove deprecated ISSUE_INDEXER_QUEUE_TYPE and ISSUE_INDEXER_QUEUE_CONN_STR. You anly need

{{- $_ := set .Values.gitea.config."queue.issue_indexer"  "TYPE" "redis" -}}

not sure how to properly quote queue.issue_indexer section.

I think you also need shared session configuration (redis or db)

{{- $_ := set .Values.gitea.config.session  "PROVIDER" "redis" -}}
{{- $_ := set .Values.gitea.config.session  "PROVIDER_CONFIG" (include "redis.dns" .) -}}
You can remove deprecated `ISSUE_INDEXER_QUEUE_TYPE` and `ISSUE_INDEXER_QUEUE_CONN_STR`. You anly need ``` {{- $_ := set .Values.gitea.config."queue.issue_indexer" "TYPE" "redis" -}} ``` not sure how to properly quote `queue.issue_indexer` section. I think you also need shared session configuration (redis or db) ``` {{- $_ := set .Values.gitea.config.session "PROVIDER" "redis" -}} {{- $_ := set .Values.gitea.config.session "PROVIDER_CONFIG" (include "redis.dns" .) -}} ```
luhahn commented 2021-07-14 07:27:07 +00:00
Author
Member
Copy Link

Thank you for your Input :)

I've already played around with shared sessions, however I ran into some Issues and forgot about it :D

Will try again and adjust the rest.

Thank you for your Input :) I've already played around with shared sessions, however I ran into some Issues and forgot about it :D Will try again and adjust the rest.
luhahn commented 2021-08-30 13:42:43 +00:00
Author
Member
Copy Link

Session now uses redis, had it configured with ingress previously.
Removed deprecated config for issue_indexer queue

Session now uses redis, had it configured with ingress previously. Removed deprecated config for issue_indexer queue
luhahn marked this conversation as resolved
values.yaml Outdated
@ -185,1 +186,3 @@
config: {}
config:
indexer:
ISSUE_INDEXER_TYPE: db
viceice commented 2021-07-14 07:23:25 +00:00
Contributor
Copy Link

Share sessions between instances, see other comment

    session:
      PROVIDER: db
Share sessions between instances, see other comment ```yaml session: PROVIDER: db ```
luhahn marked this conversation as resolved
luhahn changed title from Gitea-HA multiple replica support to WIP: Gitea-HA multiple replica support 2021-07-14 07:28:20 +00:00
luhahn commented 2021-07-14 07:49:37 +00:00
Author
Member
Copy Link

Regarding memcached, when running HA I'd currently recommend redis instead of memcached.

Regarding memcached, when running HA I'd currently recommend redis instead of memcached.
viceice commented 2021-07-14 07:53:49 +00:00
Contributor
Copy Link

Regarding memcached, when running HA I'd currently recommend redis instead of memcached.

Thats what i'm currently do.

I also have a elastic search cluster for indexing.
Is it better to use db?

> Regarding memcached, when running HA I'd currently recommend redis instead of memcached. Thats what i'm currently do. I also have a elastic search cluster for indexing. Is it better to use `db`?
luhahn commented 2021-07-14 08:20:28 +00:00
Author
Member
Copy Link

Regarding memcached, when running HA I'd currently recommend redis instead of memcached.

Thats what i'm currently do.

I also have a elastic search cluster for indexing.
Is it better to use db?

To be honest, I currently do not know which one will perform better. I plan to do some stress tests on the gitea cluster and document the results.

I choose db, because I already got postgresql-ha running, so it was the easiest to do :D

I'm thinking about adding elasticsearch as a dependency for indexing, but I'm worried, that we eventually will have too many dependencies.

> > Regarding memcached, when running HA I'd currently recommend redis instead of memcached. > > Thats what i'm currently do. > > I also have a elastic search cluster for indexing. > Is it better to use `db`? To be honest, I currently do not know which one will perform better. I plan to do some stress tests on the gitea cluster and document the results. I choose db, because I already got postgresql-ha running, so it was the easiest to do :D I'm thinking about adding elasticsearch as a dependency for indexing, but I'm worried, that we eventually will have too many dependencies.
luhahn changed title from WIP: Gitea-HA multiple replica support to Gitea-HA multiple replica support 2021-08-30 10:20:55 +00:00
luhahn commented 2021-08-30 12:52:41 +00:00
Author
Member
Copy Link

hmm still having issues with the queue.issue_indexer :/

edit: ->solved :)

hmm still having issues with the queue.issue_indexer :/ edit: ->solved :)
luhahn commented 2021-08-30 13:44:20 +00:00
Author
Member
Copy Link

This will be a breaking change, since i moved the following values

gitea.database.builtIn.mariadb.enabled -> mariadb.enabled
gitea.database.builtIn.postgresql.enabled -> postgresql.enabled
gitea.database.builtIn.mysql.enabled -> mysql.enabled
gitea.cache.builtIn.enabled -> memcached.enabled

This will be a breaking change, since i moved the following values gitea.database.builtIn.mariadb.enabled -> mariadb.enabled gitea.database.builtIn.postgresql.enabled -> postgresql.enabled gitea.database.builtIn.mysql.enabled -> mysql.enabled gitea.cache.builtIn.enabled -> memcached.enabled
luhahn commented 2021-08-30 13:45:44 +00:00
Author
Member
Copy Link

This PR is ready if anyone wants to test it :)

This PR is ready if anyone wants to test it :)
zeripath approved these changes 2021-09-23 14:25:55 +00:00
pat-s commented 2021-09-24 20:10:57 +00:00
Member
Copy Link

What is the easiest way to test this build with terraform? I cannot find an easy way to use a helm chart from a branch.

Edit: testing with minikube is probably easiest. Added #228 for instructions.

What is the easiest way to test this build with terraform? I cannot find an easy way to use a helm chart from a branch. Edit: testing with `minikube` is probably easiest. Added https://gitea.com/gitea/helm-chart/pulls/228 for instructions.
pat-s commented 2021-09-26 19:33:35 +00:00
Member
Copy Link

Please excuse for asking some possibly stupid questions and executing potentially questionable steps - relatively new to k8s replica deployments.

With this branch deployed, I see the following pods

default                gitea-0                                      1/1     Running   0               89s
default                gitea-memcached-584956987c-wdlsd             1/1     Running   0               9m16s
default                gitea-postgresql-0                           1/1     Running   0               9m16s
default                gitea-redis-master-0                         1/1     Running   0               9m16s
default                gitea-redis-replicas-0                       1/1     Running   1 (8m34s ago)   9m16s
default                gitea-redis-replicas-1                       1/1     Running   0               8m29s

I see replicas for redis but not for gitea - is gitea-memcached the second one here?
Also I see no replicaton for the postgres pod - is this expected?

Next, for gitea-redis-replicas-0 I saw the following during startup

  Normal   Started           11m (x2 over 11m)  kubelet            Started container redis
  Warning  Unhealthy         11m (x6 over 11m)  kubelet            Readiness probe failed:
Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known
  Warning  Unhealthy  11m (x5 over 11m)  kubelet  Liveness probe failed:
Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known
  Normal  Killing  11m  kubelet  Container redis failed liveness probe, will be restarted
  Normal  Pulled   11m  kubelet  Container image "docker.io/bitnami/redis:6.2.4-debian-10-r13" already present on machine

How could I effectively test HA works? I deleted the gitea-0 pod and lost the connection as port 3000 was gone (not sure if this a good test since I used kubectl port-forward).

Please excuse for asking some possibly stupid questions and executing potentially questionable steps - relatively new to k8s replica deployments. With this branch deployed, I see the following pods ``` default gitea-0 1/1 Running 0 89s default gitea-memcached-584956987c-wdlsd 1/1 Running 0 9m16s default gitea-postgresql-0 1/1 Running 0 9m16s default gitea-redis-master-0 1/1 Running 0 9m16s default gitea-redis-replicas-0 1/1 Running 1 (8m34s ago) 9m16s default gitea-redis-replicas-1 1/1 Running 0 8m29s ``` I see replicas for redis but not for gitea - is `gitea-memcached` the second one here? Also I see no replicaton for the postgres pod - is this expected? Next, for `gitea-redis-replicas-0` I saw the following during startup ``` Normal Started 11m (x2 over 11m) kubelet Started container redis Warning Unhealthy 11m (x6 over 11m) kubelet Readiness probe failed: Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known Warning Unhealthy 11m (x5 over 11m) kubelet Liveness probe failed: Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known Normal Killing 11m kubelet Container redis failed liveness probe, will be restarted Normal Pulled 11m kubelet Container image "docker.io/bitnami/redis:6.2.4-debian-10-r13" already present on machine ``` How could I effectively test HA works? I deleted the `gitea-0` pod and lost the connection as port 3000 was gone (not sure if this a good test since I used `kubectl port-forward`).
luhahn commented 2021-09-27 14:24:21 +00:00
Author
Member
Copy Link

Please excuse for asking some possibly stupid questions and executing potentially questionable steps - relatively new to k8s replica deployments.

With this branch deployed, I see the following pods

default                gitea-0                                      1/1     Running   0               89s
default                gitea-memcached-584956987c-wdlsd             1/1     Running   0               9m16s
default                gitea-postgresql-0                           1/1     Running   0               9m16s
default                gitea-redis-master-0                         1/1     Running   0               9m16s
default                gitea-redis-replicas-0                       1/1     Running   1 (8m34s ago)   9m16s
default                gitea-redis-replicas-1                       1/1     Running   0               8m29s

I see replicas for redis but not for gitea - is gitea-memcached the second one here?
Also I see no replicaton for the postgres pod - is this expected?

Next, for gitea-redis-replicas-0 I saw the following during startup

  Normal   Started           11m (x2 over 11m)  kubelet            Started container redis
  Warning  Unhealthy         11m (x6 over 11m)  kubelet            Readiness probe failed:
Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known
  Warning  Unhealthy  11m (x5 over 11m)  kubelet  Liveness probe failed:
Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known
  Normal  Killing  11m  kubelet  Container redis failed liveness probe, will be restarted
  Normal  Pulled   11m  kubelet  Container image "docker.io/bitnami/redis:6.2.4-debian-10-r13" already present on machine

How could I effectively test HA works? I deleted the gitea-0 pod and lost the connection as port 3000 was gone (not sure if this a good test since I used kubectl port-forward).

Sorry, I forgot to mention this i the initial message. Youll also need to disable memcached if you use redis. I should probably add a check here, if either one of them is enabled.

Have you set replicaCount > 1 for gitea?

> Please excuse for asking some possibly stupid questions and executing potentially questionable steps - relatively new to k8s replica deployments. > > With this branch deployed, I see the following pods > > ``` > default gitea-0 1/1 Running 0 89s > default gitea-memcached-584956987c-wdlsd 1/1 Running 0 9m16s > default gitea-postgresql-0 1/1 Running 0 9m16s > default gitea-redis-master-0 1/1 Running 0 9m16s > default gitea-redis-replicas-0 1/1 Running 1 (8m34s ago) 9m16s > default gitea-redis-replicas-1 1/1 Running 0 8m29s > > ``` > > I see replicas for redis but not for gitea - is `gitea-memcached` the second one here? > Also I see no replicaton for the postgres pod - is this expected? > > Next, for `gitea-redis-replicas-0` I saw the following during startup > > ``` > Normal Started 11m (x2 over 11m) kubelet Started container redis > Warning Unhealthy 11m (x6 over 11m) kubelet Readiness probe failed: > Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known > Warning Unhealthy 11m (x5 over 11m) kubelet Liveness probe failed: > Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known > Normal Killing 11m kubelet Container redis failed liveness probe, will be restarted > Normal Pulled 11m kubelet Container image "docker.io/bitnami/redis:6.2.4-debian-10-r13" already present on machine > ``` > > How could I effectively test HA works? I deleted the `gitea-0` pod and lost the connection as port 3000 was gone (not sure if this a good test since I used `kubectl port-forward`). Sorry, I forgot to mention this i the initial message. Youll also need to disable memcached if you use redis. I should probably add a check here, if either one of them is enabled. Have you set replicaCount > 1 for gitea?
techknowlogick added this to the 5.0.0 milestone 2021-10-15 18:11:16 +00:00
lunny commented 2021-10-20 01:58:29 +00:00
Owner
Copy Link

I think there are still some configurations absent include queue. I list them below:

  • Sessions: the values could be "memory", "file", "redis", "mysql", "postgres", "couchbase", "memcache", "db". And the default value is memory, I recommand to use db or redis. If use redis, PROVIDER_CONFIG is needed.
  • Cache: the values could be "memory", "redis", "memcache", default cache is memory. And for ha mode, redis or memcache is suitable.
  • Indexes: issues indexer supports bleve, db or elasticsearch, the default is bleve. It should be changed to db or elasticsearch.
  • code indexer supports bleve or elasticsearch, the default is bleve. It should be elasticsearch for HA mode.
  • Storages: storages for attachments, avatars, lfs, archives and so on. The default is local file system. It's better to store them to a minio instance.
I think there are still some configurations absent include `queue`. I list them below: - [x] Sessions: the values could be `"memory", "file", "redis", "mysql", "postgres", "couchbase", "memcache", "db"`. And the default value is `memory`, I recommand to use `db` or `redis`. If use `redis`, `PROVIDER_CONFIG` is needed. - [x] Cache: the values could be `"memory", "redis", "memcache"`, default cache is `memory`. And for ha mode, `redis` or `memcache` is suitable. - [x] Indexes: issues indexer supports `bleve, db or elasticsearch`, the default is `bleve`. It should be changed to `db` or `elasticsearch`. - [x] code indexer supports `bleve` or `elasticsearch`, the default is `bleve`. It should be `elasticsearch` for HA mode. - [x] Storages: storages for `attachments`, `avatars`, `lfs`, `archives` and so on. The default is local file system. It's better to store them to a `minio` instance.
👍 1
luhahn commented 2021-11-09 10:02:24 +00:00
Author
Member
Copy Link

@lunny Sorry, it took me quite a while to recover and get back to work. However

Session:
This is already included, I set redis as default including provider config, if
replicaCount > 1 and (recently added) if no session config was provided by the user.

    {{- /* multiple replicas */ -}}
    {{- if gt .Values.replicaCount 1.0 -}}
    {{- if not (hasKey .Values.gitea.config.session "PROVIDER") -}}
    {{- $_ := set .Values.gitea.config.session  "PROVIDER" "redis" -}}
    {{- end -}}
    {{- if not (hasKey .Values.gitea.config.session "PROVIDER_CONFIG") -}}
    {{- $_ := set .Values.gitea.config.session  "PROVIDER_CONFIG" (include "redis.dns" .) -}}
    {{- end -}}
    {{- end -}}

Cache:
Also already taken care of automatically, if no user config is provided for the cache:

    {{- /* cache default settings */ -}}
    {{- if or .Values.memcached.enabled (index .Values "redis-cluster").enabled .Values.redis.enabled -}}
    {{- $_ := set .Values.gitea.config.cache "ENABLED" "true" -}}
    {{- $_ := set .Values.gitea.config.cache "ADAPTER" (ternary "memcache" "redis" .Values.memcached.enabled) -}}
    {{- if not (.Values.gitea.config.cache.HOST) -}}
    {{- $_ := set .Values.gitea.config.cache "HOST" (ternary (include "memcached.dns" .) (include "redis.dns" .) .Values.memcached.enabled) -}}
    {{- end -}}
    {{- end -}}

Indexer:
Same as cache

{{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE" "db" -}}

I will check on the storage part :)

I will also need to check some parts of the config generation.

@lunny Sorry, it took me quite a while to recover and get back to work. However Session: This is already included, I set redis as default including provider config, if replicaCount > 1 and (recently added) if no session config was provided by the user. ```yaml {{- /* multiple replicas */ -}} {{- if gt .Values.replicaCount 1.0 -}} {{- if not (hasKey .Values.gitea.config.session "PROVIDER") -}} {{- $_ := set .Values.gitea.config.session "PROVIDER" "redis" -}} {{- end -}} {{- if not (hasKey .Values.gitea.config.session "PROVIDER_CONFIG") -}} {{- $_ := set .Values.gitea.config.session "PROVIDER_CONFIG" (include "redis.dns" .) -}} {{- end -}} {{- end -}} ``` Cache: Also already taken care of automatically, if no user config is provided for the cache: ```yaml {{- /* cache default settings */ -}} {{- if or .Values.memcached.enabled (index .Values "redis-cluster").enabled .Values.redis.enabled -}} {{- $_ := set .Values.gitea.config.cache "ENABLED" "true" -}} {{- $_ := set .Values.gitea.config.cache "ADAPTER" (ternary "memcache" "redis" .Values.memcached.enabled) -}} {{- if not (.Values.gitea.config.cache.HOST) -}} {{- $_ := set .Values.gitea.config.cache "HOST" (ternary (include "memcached.dns" .) (include "redis.dns" .) .Values.memcached.enabled) -}} {{- end -}} {{- end -}} ``` Indexer: Same as cache ```yaml {{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE" "db" -}} ``` I will check on the storage part :) I will also need to check some parts of the config generation.
lunny commented 2021-11-09 10:08:46 +00:00
Owner
Copy Link

@lunny Sorry, it took me quite a while to recover and get back to work. However

Session:
This is already included, I set redis as default including provider config, if
replicaCount > 1 and (recently added) if no session config was provided by the user.

    {{- /* multiple replicas */ -}}
    {{- if gt .Values.replicaCount 1.0 -}}
    {{- if not (hasKey .Values.gitea.config.session "PROVIDER") -}}
    {{- $_ := set .Values.gitea.config.session  "PROVIDER" "redis" -}}
    {{- end -}}
    {{- if not (hasKey .Values.gitea.config.session "PROVIDER_CONFIG") -}}
    {{- $_ := set .Values.gitea.config.session  "PROVIDER_CONFIG" (include "redis.dns" .) -}}
    {{- end -}}
    {{- end -}}

Cache:
Also already taken care of automatically, if no user config is provided for the cache:

    {{- /* cache default settings */ -}}
    {{- if or .Values.memcached.enabled (index .Values "redis-cluster").enabled .Values.redis.enabled -}}
    {{- $_ := set .Values.gitea.config.cache "ENABLED" "true" -}}
    {{- $_ := set .Values.gitea.config.cache "ADAPTER" (ternary "memcache" "redis" .Values.memcached.enabled) -}}
    {{- if not (.Values.gitea.config.cache.HOST) -}}
    {{- $_ := set .Values.gitea.config.cache "HOST" (ternary (include "memcached.dns" .) (include "redis.dns" .) .Values.memcached.enabled) -}}
    {{- end -}}
    {{- end -}}

Indexer:
Same as cache

{{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE" "db" -}}

I will check on the storage part :)

I will also need to check some parts of the config generation.

For indexers, there are issue indexer and code indexer. I think maybe you missed the last one.

> @lunny Sorry, it took me quite a while to recover and get back to work. However > > Session: > This is already included, I set redis as default including provider config, if > replicaCount > 1 and (recently added) if no session config was provided by the user. > > ```yaml > {{- /* multiple replicas */ -}} > {{- if gt .Values.replicaCount 1.0 -}} > {{- if not (hasKey .Values.gitea.config.session "PROVIDER") -}} > {{- $_ := set .Values.gitea.config.session "PROVIDER" "redis" -}} > {{- end -}} > {{- if not (hasKey .Values.gitea.config.session "PROVIDER_CONFIG") -}} > {{- $_ := set .Values.gitea.config.session "PROVIDER_CONFIG" (include "redis.dns" .) -}} > {{- end -}} > {{- end -}} > ``` > > Cache: > Also already taken care of automatically, if no user config is provided for the cache: > > ```yaml > {{- /* cache default settings */ -}} > {{- if or .Values.memcached.enabled (index .Values "redis-cluster").enabled .Values.redis.enabled -}} > {{- $_ := set .Values.gitea.config.cache "ENABLED" "true" -}} > {{- $_ := set .Values.gitea.config.cache "ADAPTER" (ternary "memcache" "redis" .Values.memcached.enabled) -}} > {{- if not (.Values.gitea.config.cache.HOST) -}} > {{- $_ := set .Values.gitea.config.cache "HOST" (ternary (include "memcached.dns" .) (include "redis.dns" .) .Values.memcached.enabled) -}} > {{- end -}} > {{- end -}} > ``` > > Indexer: > Same as cache > > ```yaml > {{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE" "db" -}} > ``` > > I will check on the storage part :) > > I will also need to check some parts of the config generation. For indexers, there are issue indexer and code indexer. I think maybe you missed the last one.
luhahn commented 2021-11-11 09:10:04 +00:00
Author
Member
Copy Link

@lunny regarding the storage. I think it is nice to have a minio instance for storage. However we already have a lot of dependencies in the chart, like postgresql, redis, mysql, mariadb, etc. I fear, that if we keep adding dependencies the chart will get too complicated to use. For example, the Gitlab Helm Chart also has lots and lots of dependencies and it is really horrific to use.

HA already works fine when using a RWX PVC, I think it might be better to not directly include minio.

Not blocking the Idea of minio as a dependency, but I want to discuss this :)

Same goes for the code indexer.

If I understood correctly, the code indexing feature is optional.

REPO_INDEXER_ENABLED -> false by default

I wouldn't want to add elasticsearch as a dependency, same reason as stated above :)
We should add a description which makes clear, that if HA is used, the REPO_INDEXER should be set to elasticsearch.

@lunny regarding the storage. I think it is nice to have a minio instance for storage. However we already have a lot of dependencies in the chart, like postgresql, redis, mysql, mariadb, etc. I fear, that if we keep adding dependencies the chart will get too complicated to use. For example, the Gitlab Helm Chart also has lots and lots of dependencies and it is really horrific to use. HA already works fine when using a RWX PVC, I think it might be better to not directly include minio. Not blocking the Idea of minio as a dependency, but I want to discuss this :) Same goes for the code indexer. If I understood correctly, the code indexing feature is optional. REPO_INDEXER_ENABLED -> false by default I wouldn't want to add elasticsearch as a dependency, same reason as stated above :) We should add a description which makes clear, that if HA is used, the REPO_INDEXER should be set to elasticsearch.
❤️ 1
lunny commented 2021-11-11 10:07:04 +00:00
Owner
Copy Link

@lunny regarding the storage. I think it is nice to have a minio instance for storage. However we already have a lot of dependencies in the chart, like postgresql, redis, mysql, mariadb, etc. I fear, that if we keep adding dependencies the chart will get too complicated to use. For example, the Gitlab Helm Chart also has lots and lots of dependencies and it is really horrific to use.

HA already works fine when using a RWX PVC, I think it might be better to not directly include minio.

Not blocking the Idea of minio as a dependency, but I want to discuss this :)

Same goes for the code indexer.

If I understood correctly, the code indexing feature is optional.

REPO_INDEXER_ENABLED -> false by default

I wouldn't want to add elasticsearch as a dependency, same reason as stated above :)
We should add a description which makes clear, that if HA is used, the REPO_INDEXER should be set to elasticsearch.

But if code indexer set to true, and it's a default value bleve. The second gitea instance maybe hang.

For storage, if it's a RWX PVC, That's no problem.

> @lunny regarding the storage. I think it is nice to have a minio instance for storage. However we already have a lot of dependencies in the chart, like postgresql, redis, mysql, mariadb, etc. I fear, that if we keep adding dependencies the chart will get too complicated to use. For example, the Gitlab Helm Chart also has lots and lots of dependencies and it is really horrific to use. > > HA already works fine when using a RWX PVC, I think it might be better to not directly include minio. > > Not blocking the Idea of minio as a dependency, but I want to discuss this :) > > Same goes for the code indexer. > > If I understood correctly, the code indexing feature is optional. > > REPO_INDEXER_ENABLED -> false by default > > I wouldn't want to add elasticsearch as a dependency, same reason as stated above :) > We should add a description which makes clear, that if HA is used, the REPO_INDEXER should be set to elasticsearch. > But if code indexer set to true, and it's a default value `bleve`. The second gitea instance maybe hang. For storage, if it's a RWX PVC, That's no problem.
justusbunsi added the
status
blocked
 label 2021-11-19 18:52:12 +00:00
justusbunsi added a new dependency 2021-11-19 18:52:33 +00:00
#243 Generic way for configuring Gitea
justusbunsi commented 2021-12-18 11:33:23 +00:00
Member
Copy Link

I'm wondering if Gitea could handle PodDisruptionBudgets as to replace one pod at a time to keep Gitea up?
Is this too risky from application perspective regarding data(base) consistency?

I'm wondering if Gitea could handle `PodDisruptionBudgets` as to _replace one pod at a time to keep Gitea up_? Is this too risky from application perspective regarding data(base) consistency?
luhahn modified the milestone from 5.0.0 to 5.0.x 2021-12-22 16:01:43 +00:00
luhahn force-pushed gitea-ha from ef43ba11be to d9d4266019 2021-12-22 16:36:27 +00:00 Compare
luhahn force-pushed gitea-ha from d9d4266019 to 5668ce4303 2021-12-22 16:37:08 +00:00 Compare
luhahn force-pushed gitea-ha from 5668ce4303 to 4869cbc54f 2021-12-22 16:47:31 +00:00 Compare
pat-s commented 2022-03-26 16:43:33 +00:00
Member
Copy Link

In case some testing/help is needed, LMK :)

In case some testing/help is needed, LMK :)
pat-s commented 2022-05-17 18:41:07 +00:00
Member
Copy Link

@luhahn @justusbunsi

I'd be interesting in testing/pushing this forward. I could start by merging master first and resolving conflicts, then continue with some testing/feedback.

@luhahn @justusbunsi I'd be interesting in testing/pushing this forward. I could start by merging `master` first and resolving conflicts, then continue with some testing/feedback.
👍 1
luhahn commented 2022-05-18 09:08:15 +00:00
Author
Member
Copy Link

I will hopefully continue next week, the last time i checked, i had some issues.

I will hopefully continue next week, the last time i checked, i had some issues.
:gitea: 2
justusbunsi commented 2022-06-11 11:50:10 +00:00
Member
Copy Link

As per discord discussion, running multiple Gitea instances on the same database and storage requires some sort of internal leader election. Without the instances being aware of each others everyone would execute cron tasks. This can lead to unexpected inconsistencies.

As per discord discussion, running multiple Gitea instances on the same database and storage requires some sort of internal leader election. Without the instances being aware of each others everyone would execute cron tasks. This can lead to unexpected inconsistencies.
justusbunsi removed the
status
blocked
 label 2022-06-11 11:50:33 +00:00
justusbunsi removed a dependency 2022-06-11 11:50:42 +00:00
#243 Generic way for configuring Gitea
justusbunsi removed this from the 5.0.x milestone 2022-06-11 11:50:59 +00:00
justusbunsi commented 2022-06-26 12:58:22 +00:00
Member
Copy Link

Just realized that when running multiple replicas, all init containers would run x times. But this might be a problem. Think of upgrading Gitea. One of the init container executes a gitea migrate. We wouldn't want to have this action done twice or in parallel.

#332

Just realized that when running multiple replicas, all init containers would run x times. But this might be a problem. Think of upgrading Gitea. One of the init container executes a `gitea migrate`. We wouldn't want to have this action done twice or in parallel. #332
viceice commented 2022-08-24 13:40:56 +00:00
Contributor
Copy Link

As per discord discussion, running multiple Gitea instances on the same database and storage requires some sort of internal leader election. Without the instances being aware of each others everyone would execute cron tasks. This can lead to unexpected inconsistencies.

I maybe see those issues sometimes now.
symtoms:

  • push a commit, which isn't visible to ci job yet
  • broken PR, which never gets it's checks ready
> As per discord discussion, running multiple Gitea instances on the same database and storage requires some sort of internal leader election. Without the instances being aware of each others everyone would execute cron tasks. This can lead to unexpected inconsistencies. I maybe see those issues sometimes now. symtoms: - push a commit, which isn't visible to ci job yet - broken PR, which never gets it's checks ready
😕 1
lunny commented 2023-03-07 13:50:34 +00:00
Owner
Copy Link

Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not?

Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not?
justusbunsi commented 2023-03-07 15:49:35 +00:00
Member
Copy Link

Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not?

That could actually be a way into the right direction. Can you share the PR reference that implemented this?

I think it is important for using that functionality, to move away from the complex structure of the init containers. We cannot tell which replica shall act as leader and run the migration. All replicas are identical regarding configuration.

app.ini creation and other preparation must be done once for all replicas. They share those files.

Another problem is that we cannot start a new Gitea version along with the old one to perform a rollover. I currently see no way to achieve it without a downtime. The database must be migrated in order to run the newer Gitea version which would break older running versions. Migrate is required for major releases afaik, maybe for patch release too.

An option would be to first scale down all non-leader replicas, then replace the leader with the new version, let it handle the migration, scale non-leader replicas up with new Gitea version. To achieve that, we have to know which instance is the leader instance.

> Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not? That could actually be a way into the right direction. Can you share the PR reference that implemented this? I think it is important for using that functionality, to move away from the complex structure of the init containers. We cannot tell which replica shall act as leader and run the migration. All replicas are identical regarding configuration. app.ini creation and other preparation must be done once for all replicas. They share those files. Another problem is that we cannot start a new Gitea version along with the old one to perform a rollover. I currently see no way to achieve it without a downtime. The database must be migrated in order to run the newer Gitea version which would break older running versions. Migrate is required for major releases afaik, maybe for patch release too. An option would be to first scale down all non-leader replicas, then replace the leader with the new version, let it handle the migration, scale non-leader replicas up with new Gitea version. To achieve that, we have to know which instance is the leader instance.
lunny commented 2023-03-07 15:59:50 +00:00
Owner
Copy Link

Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not?

That could actually be a way into the right direction. Can you share the PR reference that implemented this?

I think it is important for using that functionality, to move away from the complex structure of the init containers. We cannot tell which replica shall act as leader and run the migration. All replicas are identical regarding configuration.

app.ini creation and other preparation must be done once for all replicas. They share those files.

Another problem is that we cannot start a new Gitea version along with the old one to perform a rollover. I currently see no way to achieve it without a downtime. The database must be migrated in order to run the newer Gitea version which would break older running versions. Migrate is required for major releases afaik, maybe for patch release too.

An option would be to first scale down all non-leader replicas, then replace the leader with the new version, let it handle the migration, scale non-leader replicas up with new Gitea version. To achieve that, we have to know which instance is the leader instance.

It's https://github.com/go-gitea/gitea/pull/22053 . I also think maybe it's the only way to upgrade.

> > Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not? > > That could actually be a way into the right direction. Can you share the PR reference that implemented this? > > I think it is important for using that functionality, to move away from the complex structure of the init containers. We cannot tell which replica shall act as leader and run the migration. All replicas are identical regarding configuration. > > app.ini creation and other preparation must be done once for all replicas. They share those files. > > Another problem is that we cannot start a new Gitea version along with the old one to perform a rollover. I currently see no way to achieve it without a downtime. The database must be migrated in order to run the newer Gitea version which would break older running versions. Migrate is required for major releases afaik, maybe for patch release too. > > An option would be to first scale down all non-leader replicas, then replace the leader with the new version, let it handle the migration, scale non-leader replicas up with new Gitea version. To achieve that, we have to know which instance is the leader instance. It's https://github.com/go-gitea/gitea/pull/22053 . I also think maybe it's the only way to upgrade.
👍 1
pat-s commented 2023-04-01 10:34:42 +00:00
Member
Copy Link

Re:memcached

The bitnami chart says one can set architecture: "high-availability" and everything should be handled behind the scences.

Re:memcached The bitnami chart says one can set `architecture: "high-availability"` and everything should be handled behind the scences.
pat-s commented 2023-04-01 10:40:44 +00:00
Member
Copy Link

That could bring everything down to the following (aiming for minimal changes to the helm chart):

  • Deploy/configure built-in memcached in HA mode
  • Disable repo-indexing for bleve or use elastisearch or similar
  • Use a RWX for persistent storage
  • Use an external DB which is HA-capable (e.g. RDS on AWS)

In addition we could try to use a deployment instead of a statefulset for the gitea pods to avoid creating replicated PVs for each replica (see discussion in #426).

I will go ahead and create a new branch with some required changes and test the above config in a toy cluster.

That could bring everything down to the following (aiming for minimal changes to the helm chart): - Deploy/configure built-in `memcached` in HA mode - Disable repo-indexing for `bleve` or use `elastisearch` or similar - Use a RWX for persistent storage - Use an external DB which is HA-capable (e.g. RDS on AWS) In addition we could try to use a deployment instead of a statefulset for the gitea pods to avoid creating replicated PVs for each replica (see discussion in #426). I will go ahead and create a new branch with some required changes and test the above config in a toy cluster.
pat-s commented 2023-04-02 08:15:21 +00:00
Member
Copy Link

The bitnami chart says one can set architecture: "high-availability" and everything should be handled behind the scences.

Tried it, doesn't work. Doesn't matter it memcached itself is running in HA, the operations in the back don't work without issues.

> The bitnami chart says one can set architecture: "high-availability" and everything should be handled behind the scences. Tried it, doesn't work. Doesn't matter it `memcached` itself is running in HA, the operations in the back don't work without issues.
pat-s commented 2023-04-14 10:11:30 +00:00
Member
Copy Link

I'll close here in favor of #437 which has almost everything of this PR + more.

I'll close here in favor of #437 which has almost everything of this PR + more.
pat-s closed this pull request 2023-04-14 10:11:30 +00:00
Some checks are pending
continuous-integration/drone/push Build is passing
Details
continuous-integration/drone/pr Build is passing
Details
check-and-test / check-and-test (pull_request)
Required

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
zeripath
Labels
Clear labels   
has
backport

  
in progress

   
invalid
  
kind
breaking

  
kind
bug

  
kind
build

  
kind
dependency

  
kind
deployment

  
kind
docs

  
kind
enhancement

  
kind
feature

  
kind
lint

  
kind
proposal

  
kind
question

  
kind
refactor

  
kind
security

  
kind
testing

  
kind
translation

  
kind
ui

  
need
backport

  
priority
critical

  
priority
low

  
priority
maybe

  
priority
medium

  
reviewed
duplicate

  
reviewed
invalid

  
reviewed
wontfix

  
skip-changelog
  
status
blocked

  
status
needs-feedback

  
status
needs-reviews

  
status
wip

  
upstream
gitea

  
upstream
other

No Label
has
backport
in progress
invalid
kind
breaking
kind
bug
kind
build
kind
dependency
kind
deployment
kind
docs
kind
enhancement
kind
feature
kind
lint
kind
proposal
kind
question
kind
refactor
kind
security
kind
testing
kind
translation
kind
ui
need
backport
priority
critical
priority
low
priority
maybe
priority
medium
reviewed
duplicate
reviewed
invalid
reviewed
wontfix
skip-changelog
status
blocked
status
needs-feedback
status
needs-reviews
status
wip
upstream
gitea
upstream
other
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
7 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gitea/helm-chart#205
No description provided.
Delete Branch "gitea-ha"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?