Provide gitea-ha support -> multiple replicas #206

Closed
opened 2021-07-09 14:49:23 +00:00 by luhahn · 9 comments
Member

As of now it is possible to run gitea with multiple replicas with the helm chart,
however a lot of configuration is needed.

This could be included in the helm chart.

As of now it is possible to run gitea with multiple replicas with the helm chart, however a lot of configuration is needed. This could be included in the helm chart.
luhahn added the
kind
feature
label 2021-07-09 14:49:23 +00:00
Member

Is this already possible? I've set replicaCount: 2 and it seems to work as expected for now - did not do extensive testing yet though.

Is this already possible? I've set `replicaCount: 2` and it seems to work as expected for now - did not do extensive testing yet though.
Contributor

Yes, it's working fine, but needs lots of configuration. ?
My Gitea values.yaml is about 300 LOC for this. ?

Yes, it's working fine, but needs lots of configuration. ? My Gitea `values.yaml` is about 300 LOC for this. ?

Can you provide example on how to configure gitea for HA, like did you configure redis in the config?

Thank you

Can you provide example on how to configure gitea for HA, like did you configure redis in the config? Thank you
Contributor

Removed some unrelated settings:

replicaCount: 2

image:
  repository: gitea/gitea
  tag: 1.15.7
  rootless: true

gitea:
  database:
    builtIn:
      postgresql:
        enabled: false
  cache:
    builtIn:
      enabled: false

  config:
    database:
      DB_TYPE: postgres
      HOST: postgresql.db:5432
      NAME: gitea
      USER: gitea
      LOG_SQL: false
      MAX_IDLE_CONNS: 10
      CONN_MAX_LIFE_TIME: 5m
    server:
      DISABLE_SSH: true
      LANDING_PAGE: login


    cache:
      ADAPTER: redis
      ITEM_TTL: 72h

    queue:
      TYPE: redis

    session:
      PROVIDER: redis
      COOKIE_SECURE: true

    indexer:
      ISSUE_INDEXER_TYPE: elasticsearch
      ISSUE_INDEXER_CONN_STR: http://${search-server}:9200
      REPO_INDEXER_ENABLED: true
      REPO_INDEXER_TYPE: elasticsearch
      REPO_INDEXER_CONN_STR: http://${search-server}:9200


    attachment:
      MAX_SIZE: 50
      STORAGE_TYPE: minio
    lfs:
      STORAGE_TYPE: minio
    picture:
      AVATAR_STORAGE_TYPE: minio
      REPOSITORY_AVATAR_STORAGE_TYPE: minio

    storage:
      MINIO_ENDPOINT: s3.xxxx.de
      MINIO_LOCATION: md-hq-1
      MINIO_ACCESS_KEY_ID: xxxxx
      MINIO_BUCKET: gitea
      MINIO_USE_SSL: true

    storage.repo-archive:
      STORAGE_TYPE: minio

initPreScript: |
  [ -d /data/gitea ] && chmod 700 /data/gitea  

Other params are set via terraform

resource "helm_release" "gitea" {
  depends_on = [
    helm_release.keydb,
    helm_release.opensearch,
  ]

  name      = "gitea"
  namespace = local.namespace

  repository = "https://dl.gitea.io/charts"
  chart      = "gitea"
  version    = "4.1.1"

  max_history = 10

  values = [
    templatefile("${path.module}/gitea/values.yaml", {
      search-server     = "${helm_release.opensearch.name}-master"
      cm-files          = kubernetes_config_map.gitea_files.metadata.0.name
      cm-files-checksum = base64sha256(jsonencode(kubernetes_config_map.gitea_files.data))
    })
  ]


  set_sensitive {
    name  = "gitea.config.database.PASSWD"
    value = var.passwords.postgresql.gitea
  }

  set_sensitive {
    name  = "gitea.config.security.SECRET_KEY"
    value = local.gitea_sk
  }

  set_sensitive {
    name  = "gitea.config.storage.MINIO_SECRET_ACCESS_KEY"
    value = var.passwords.minio.gitea
  }

  set_sensitive {
    name  = "gitea.config.oauth2.JWT_SECRET"
    value = var.passwords.gitea.jwt_token
  }

  set_sensitive {
    name  = "gitea.config.cache.HOST"
    value = local.redis_cs
  }

  set_sensitive {
    name  = "gitea.config.queue.CONN_STR"
    value = local.redis_cs
  }

  set_sensitive {
    name  = "gitea.config.session.PROVIDER_CONFIG"
    value = local.redis_cs
  }

  set {
    name  = "gitea.admin.existingSecret"
    value = kubernetes_secret.gitea_admin.metadata.0.name
  }

  set {
    name  = "gitea.ldap.existingSecret"
    value = kubernetes_secret.gitea_ldap.metadata.0.name
  }


  set {
    name  = "persistence.existingClaim"
    value = kubernetes_persistent_volume_claim.gitea_data.metadata.0.name
  }
}

locals {
  redis_cs        = "redis://${local.redis_password}@${helm_release.keydb.name}:6379/0?pool_size=100&idle_timeout=180s"
}

I'm using keydb as redis and opensearch as elasticsearch replacement

Removed some unrelated settings: ```yml replicaCount: 2 image: repository: gitea/gitea tag: 1.15.7 rootless: true gitea: database: builtIn: postgresql: enabled: false cache: builtIn: enabled: false config: database: DB_TYPE: postgres HOST: postgresql.db:5432 NAME: gitea USER: gitea LOG_SQL: false MAX_IDLE_CONNS: 10 CONN_MAX_LIFE_TIME: 5m server: DISABLE_SSH: true LANDING_PAGE: login cache: ADAPTER: redis ITEM_TTL: 72h queue: TYPE: redis session: PROVIDER: redis COOKIE_SECURE: true indexer: ISSUE_INDEXER_TYPE: elasticsearch ISSUE_INDEXER_CONN_STR: http://${search-server}:9200 REPO_INDEXER_ENABLED: true REPO_INDEXER_TYPE: elasticsearch REPO_INDEXER_CONN_STR: http://${search-server}:9200 attachment: MAX_SIZE: 50 STORAGE_TYPE: minio lfs: STORAGE_TYPE: minio picture: AVATAR_STORAGE_TYPE: minio REPOSITORY_AVATAR_STORAGE_TYPE: minio storage: MINIO_ENDPOINT: s3.xxxx.de MINIO_LOCATION: md-hq-1 MINIO_ACCESS_KEY_ID: xxxxx MINIO_BUCKET: gitea MINIO_USE_SSL: true storage.repo-archive: STORAGE_TYPE: minio initPreScript: | [ -d /data/gitea ] && chmod 700 /data/gitea ``` Other params are set via terraform ```hcl resource "helm_release" "gitea" { depends_on = [ helm_release.keydb, helm_release.opensearch, ] name = "gitea" namespace = local.namespace repository = "https://dl.gitea.io/charts" chart = "gitea" version = "4.1.1" max_history = 10 values = [ templatefile("${path.module}/gitea/values.yaml", { search-server = "${helm_release.opensearch.name}-master" cm-files = kubernetes_config_map.gitea_files.metadata.0.name cm-files-checksum = base64sha256(jsonencode(kubernetes_config_map.gitea_files.data)) }) ] set_sensitive { name = "gitea.config.database.PASSWD" value = var.passwords.postgresql.gitea } set_sensitive { name = "gitea.config.security.SECRET_KEY" value = local.gitea_sk } set_sensitive { name = "gitea.config.storage.MINIO_SECRET_ACCESS_KEY" value = var.passwords.minio.gitea } set_sensitive { name = "gitea.config.oauth2.JWT_SECRET" value = var.passwords.gitea.jwt_token } set_sensitive { name = "gitea.config.cache.HOST" value = local.redis_cs } set_sensitive { name = "gitea.config.queue.CONN_STR" value = local.redis_cs } set_sensitive { name = "gitea.config.session.PROVIDER_CONFIG" value = local.redis_cs } set { name = "gitea.admin.existingSecret" value = kubernetes_secret.gitea_admin.metadata.0.name } set { name = "gitea.ldap.existingSecret" value = kubernetes_secret.gitea_ldap.metadata.0.name } set { name = "persistence.existingClaim" value = kubernetes_persistent_volume_claim.gitea_data.metadata.0.name } } locals { redis_cs = "redis://${local.redis_password}@${helm_release.keydb.name}:6379/0?pool_size=100&idle_timeout=180s" } ``` I'm using `keydb` as `redis` and `opensearch` as `elasticsearch` replacement - https://artifacthub.io/packages/helm/enapter/keydb - https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch
Owner

Removed some unrelated settings:

replicaCount: 2

image:
  repository: gitea/gitea
  tag: 1.15.7
  rootless: true

gitea:
  database:
    builtIn:
      postgresql:
        enabled: false
  cache:
    builtIn:
      enabled: false

  config:
    database:
      DB_TYPE: postgres
      HOST: postgresql.db:5432
      NAME: gitea
      USER: gitea
      LOG_SQL: false
      MAX_IDLE_CONNS: 10
      CONN_MAX_LIFE_TIME: 5m
    server:
      DISABLE_SSH: true
      LANDING_PAGE: login


    cache:
      ADAPTER: redis
      ITEM_TTL: 72h

    queue:
      TYPE: redis

    session:
      PROVIDER: redis
      COOKIE_SECURE: true

    indexer:
      ISSUE_INDEXER_TYPE: elasticsearch
      ISSUE_INDEXER_CONN_STR: http://${search-server}:9200
      REPO_INDEXER_ENABLED: true
      REPO_INDEXER_TYPE: elasticsearch
      REPO_INDEXER_CONN_STR: http://${search-server}:9200


    attachment:
      MAX_SIZE: 50
      STORAGE_TYPE: minio
    lfs:
      STORAGE_TYPE: minio
    picture:
      AVATAR_STORAGE_TYPE: minio
      REPOSITORY_AVATAR_STORAGE_TYPE: minio

    storage:
      MINIO_ENDPOINT: s3.xxxx.de
      MINIO_LOCATION: md-hq-1
      MINIO_ACCESS_KEY_ID: xxxxx
      MINIO_BUCKET: gitea
      MINIO_USE_SSL: true

    storage.repo-archive:
      STORAGE_TYPE: minio

initPreScript: |
  [ -d /data/gitea ] && chmod 700 /data/gitea  

Other params are set via terraform

resource "helm_release" "gitea" {
  depends_on = [
    helm_release.keydb,
    helm_release.opensearch,
  ]

  name      = "gitea"
  namespace = local.namespace

  repository = "https://dl.gitea.io/charts"
  chart      = "gitea"
  version    = "4.1.1"

  max_history = 10

  values = [
    templatefile("${path.module}/gitea/values.yaml", {
      search-server     = "${helm_release.opensearch.name}-master"
      cm-files          = kubernetes_config_map.gitea_files.metadata.0.name
      cm-files-checksum = base64sha256(jsonencode(kubernetes_config_map.gitea_files.data))
    })
  ]


  set_sensitive {
    name  = "gitea.config.database.PASSWD"
    value = var.passwords.postgresql.gitea
  }

  set_sensitive {
    name  = "gitea.config.security.SECRET_KEY"
    value = local.gitea_sk
  }

  set_sensitive {
    name  = "gitea.config.storage.MINIO_SECRET_ACCESS_KEY"
    value = var.passwords.minio.gitea
  }

  set_sensitive {
    name  = "gitea.config.oauth2.JWT_SECRET"
    value = var.passwords.gitea.jwt_token
  }

  set_sensitive {
    name  = "gitea.config.cache.HOST"
    value = local.redis_cs
  }

  set_sensitive {
    name  = "gitea.config.queue.CONN_STR"
    value = local.redis_cs
  }

  set_sensitive {
    name  = "gitea.config.session.PROVIDER_CONFIG"
    value = local.redis_cs
  }

  set {
    name  = "gitea.admin.existingSecret"
    value = kubernetes_secret.gitea_admin.metadata.0.name
  }

  set {
    name  = "gitea.ldap.existingSecret"
    value = kubernetes_secret.gitea_ldap.metadata.0.name
  }


  set {
    name  = "persistence.existingClaim"
    value = kubernetes_persistent_volume_claim.gitea_data.metadata.0.name
  }
}

locals {
  redis_cs        = "redis://${local.redis_password}@${helm_release.keydb.name}:6379/0?pool_size=100&idle_timeout=180s"
}

I'm using keydb as redis and opensearch as elasticsearch replacement

The example is perfect. Only one thing which is Gitea's problem is every instance will run the cron jobs. A simple method is to add a command line flag to disable or enable all cron jobs which is not implemented by v1.15.7. A better case is Gitea cluster, all Gitea replications could know each other and will elect one to run cron jobs.

> Removed some unrelated settings: > ```yml > replicaCount: 2 > > image: > repository: gitea/gitea > tag: 1.15.7 > rootless: true > > gitea: > database: > builtIn: > postgresql: > enabled: false > cache: > builtIn: > enabled: false > > config: > database: > DB_TYPE: postgres > HOST: postgresql.db:5432 > NAME: gitea > USER: gitea > LOG_SQL: false > MAX_IDLE_CONNS: 10 > CONN_MAX_LIFE_TIME: 5m > server: > DISABLE_SSH: true > LANDING_PAGE: login > > > cache: > ADAPTER: redis > ITEM_TTL: 72h > > queue: > TYPE: redis > > session: > PROVIDER: redis > COOKIE_SECURE: true > > indexer: > ISSUE_INDEXER_TYPE: elasticsearch > ISSUE_INDEXER_CONN_STR: http://${search-server}:9200 > REPO_INDEXER_ENABLED: true > REPO_INDEXER_TYPE: elasticsearch > REPO_INDEXER_CONN_STR: http://${search-server}:9200 > > > attachment: > MAX_SIZE: 50 > STORAGE_TYPE: minio > lfs: > STORAGE_TYPE: minio > picture: > AVATAR_STORAGE_TYPE: minio > REPOSITORY_AVATAR_STORAGE_TYPE: minio > > storage: > MINIO_ENDPOINT: s3.xxxx.de > MINIO_LOCATION: md-hq-1 > MINIO_ACCESS_KEY_ID: xxxxx > MINIO_BUCKET: gitea > MINIO_USE_SSL: true > > storage.repo-archive: > STORAGE_TYPE: minio > > initPreScript: | > [ -d /data/gitea ] && chmod 700 /data/gitea > > ``` > > Other params are set via terraform > > ```hcl > resource "helm_release" "gitea" { > depends_on = [ > helm_release.keydb, > helm_release.opensearch, > ] > > name = "gitea" > namespace = local.namespace > > repository = "https://dl.gitea.io/charts" > chart = "gitea" > version = "4.1.1" > > max_history = 10 > > values = [ > templatefile("${path.module}/gitea/values.yaml", { > search-server = "${helm_release.opensearch.name}-master" > cm-files = kubernetes_config_map.gitea_files.metadata.0.name > cm-files-checksum = base64sha256(jsonencode(kubernetes_config_map.gitea_files.data)) > }) > ] > > > set_sensitive { > name = "gitea.config.database.PASSWD" > value = var.passwords.postgresql.gitea > } > > set_sensitive { > name = "gitea.config.security.SECRET_KEY" > value = local.gitea_sk > } > > set_sensitive { > name = "gitea.config.storage.MINIO_SECRET_ACCESS_KEY" > value = var.passwords.minio.gitea > } > > set_sensitive { > name = "gitea.config.oauth2.JWT_SECRET" > value = var.passwords.gitea.jwt_token > } > > set_sensitive { > name = "gitea.config.cache.HOST" > value = local.redis_cs > } > > set_sensitive { > name = "gitea.config.queue.CONN_STR" > value = local.redis_cs > } > > set_sensitive { > name = "gitea.config.session.PROVIDER_CONFIG" > value = local.redis_cs > } > > set { > name = "gitea.admin.existingSecret" > value = kubernetes_secret.gitea_admin.metadata.0.name > } > > set { > name = "gitea.ldap.existingSecret" > value = kubernetes_secret.gitea_ldap.metadata.0.name > } > > > set { > name = "persistence.existingClaim" > value = kubernetes_persistent_volume_claim.gitea_data.metadata.0.name > } > } > > locals { > redis_cs = "redis://${local.redis_password}@${helm_release.keydb.name}:6379/0?pool_size=100&idle_timeout=180s" > } > > ``` > > I'm using `keydb` as `redis` and `opensearch` as `elasticsearch` replacement > - https://artifacthub.io/packages/helm/enapter/keydb > - https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch The example is perfect. Only one thing which is Gitea's problem is every instance will run the cron jobs. A simple method is to add a command line flag to disable or enable all cron jobs which is not implemented by v1.15.7. A better case is Gitea cluster, all Gitea replications could know each other and will elect one to run cron jobs.
Member

A better case is Gitea cluster, all Gitea replications could know each other and will elect one to run cron jobs.

That would be good. Do you think it's a thing to implement in the Chart or in Gitea itself?

> A better case is Gitea cluster, all Gitea replications could know each other and will elect one to run cron jobs. That would be good. Do you think it's a thing to implement in the Chart or in Gitea itself?

That would be awesome! I just found this issue on their github: https://github.com/go-gitea/gitea/issues/13791

So I guess they would eventually make it possible. I'm not sure we can implement it in the Chart before they did it in Gitea itself

That would be awesome! I just found this issue on their github: https://github.com/go-gitea/gitea/issues/13791 So I guess they would eventually make it possible. I'm not sure we can implement it in the Chart before they did it in Gitea itself
Member

I've spent some time revisiting the current state, especially #205 and #206.
I'll try to replicate the setup posted from @viceice in the next days/weeks and see how far I can go.

Of course there is still #205 which aims to provide a built-in solution for HA. While this would of course be nice (!), I wonder if we should rather focus on documenting HA requirements and configuration changes given all the considerations one has to make when going HA here (indexer, memcache, attachment storage, etc.).
If this is done and people have a guide how to go HA with their setup, one could focus on a "simple" default config within the chart as in #206 for a "best effort" config solution that does not require multiple additional chart deployments?

I think it could be worth adding a dedicated markdown doc (e.g. ha-setup.md) into here (next to the README or within docs/ or config/) and explain what is needed and what are possible options to achieve this.

For example, outlining that elasticsearch (or similar) is required as a code indexer which itself requires quite some resources on a cluster (4-8 GB and multiple nodes) AFAICS. Such information might already put the topic off the table for some as their cluster is not that large.

I really see the need to consolidate information from #205 and #206 as otherwise it will get lost over time and it is really hard for users to grasp the current state. I am aware of the Gitea-internal issue of cron jobs running multiple times as described in #205, but I guess for most users this might be acceptable in a trade-off for a HA setup.

AFAICS, and mainly putting things together from @lunny comment in #205 (comment):

Additional/External requirements for Gitea HA

(Happy to edit this list or put it into a dedicated markdown doc in the repo)

  • A HA-ready code indexer (elasticsearch or similar, 4-8 GB in memory)
  • A HA-ready external object storage (minio)
  • A HA-ready memcache (e.g. redis or similar)
  • A RWX file-system (e.g. NFS or EFS (AWS), or similar)
I've spent some time revisiting the current state, especially #205 and #206. I'll try to replicate the setup posted from @viceice in the next days/weeks and see how far I can go. Of course there is still #205 which aims to provide a built-in solution for HA. While this would of course be nice (!), I wonder if we should rather focus on documenting HA requirements and configuration changes given all the considerations one has to make when going HA here (indexer, memcache, attachment storage, etc.). If this is done and people have a guide how to go HA with their setup, one could focus on a "simple" default config within the chart as in #206 for a "best effort" config solution that does not require multiple additional chart deployments? I think it could be worth adding a dedicated markdown doc (e.g. `ha-setup.md`) into here (next to the README or within `docs/` or `config/`) and explain what is needed and what are possible options to achieve this. For example, outlining that `elasticsearch` (or similar) is required as a code indexer which itself requires quite some resources on a cluster (4-8 GB and multiple nodes) AFAICS. Such information might already put the topic off the table for some as their cluster is not that large. I really see the need to consolidate information from #205 and #206 as otherwise it will get lost over time and it is really hard for users to grasp the current state. I am aware of the Gitea-internal issue of cron jobs running multiple times as described in #205, but I guess for most users this might be acceptable in a trade-off for a HA setup. AFAICS, and mainly putting things together from @lunny comment in https://gitea.com/gitea/helm-chart/pulls/205#issuecomment-593662: ### Additional/External requirements for Gitea HA (Happy to edit this list or put it into a dedicated markdown doc in the repo) - A HA-ready code indexer (`elasticsearch` or similar, 4-8 GB in memory) - A HA-ready external object storage (`minio`) - A HA-ready memcache (e.g. `redis` or similar) - A RWX file-system (e.g. NFS or EFS (AWS), or similar)
Member

Done in #437.

Done in #437.
pat-s closed this issue 2023-07-17 20:00:06 +00:00
Sign in to join this conversation.
No Milestone
No Assignees
6 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gitea/helm-chart#206
No description provided.