use postgresql-ha login 500 or root user cannot be created #502

New Issue

ggbar · 2023-09-05T08:06:07Z

ggbar commented

2023-09-05 08:06:07 +00:00

https://github.com/go-gitea/gitea/issues/26858

justusbunsi commented

2023-09-05 19:25:20 +00:00

Same question as https://github.com/go-gitea/gitea/issues/26858#issuecomment-1702639940

Is Gitea trying to connect to a readonly instance of the db?

Also: You said it works well with non-ha postgres, right? Is the have you checked whether there are differences in values definition between postgres and postgres-ha chart?

Same question as https://github.com/go-gitea/gitea/issues/26858#issuecomment-1702639940 Is Gitea trying to connect to a readonly instance of the db? Also: You said it works well with non-ha postgres, right? Is the have you checked whether there are differences in values definition between postgres and postgres-ha chart?

ggbar commented

2023-09-06 01:07:24 +00:00

Is Gitea trying to connect to a readonly instance of the db?

Yes, gitea is randomly connected to postgresql instances

Also: You said it works well with non-ha postgres, right? Is the have you checked whether there are differences in values definition between postgres and postgres-ha chart?

I used the default configuration

> Is Gitea trying to connect to a readonly instance of the db? Yes, gitea is randomly connected to postgresql instances > Also: You said it works well with non-ha postgres, right? Is the have you checked whether there are differences in values definition between postgres and postgres-ha chart? I used the default configuration

Delta1977 commented

2023-09-08 08:32:19 +00:00

same here with chart 9.3.0

pat-s commented

2023-09-09 15:32:22 +00:00

To add some context, I just tried to bootstrap in a fresh namespace and this is what the logs say

Failed to initialize ORM engine: migrate: sync: pq: cannot execute CREATE TABLE in a read-only transaction

Besides the issue at hand, I wonder how we can test for something like this in the future. The deployment worked a few weeks ago so something must have changed in the bitnami chart presumably.
I guess I should look into the chart-testing action again.

To add some context, I just tried to bootstrap in a fresh namespace and this is what the logs say ``` Failed to initialize ORM engine: migrate: sync: pq: cannot execute CREATE TABLE in a read-only transaction ``` Besides the issue at hand, I wonder how we can test for something like this in the future. The deployment worked a few weeks ago so something must have changed in the bitnami chart presumably. I guess I should look into the chart-testing action again.

pat-s added the

labels 2023-09-09 15:32:36 +00:00

pat-s commented

2023-09-09 15:58:47 +00:00

What helped was scaling down postgresql-ha to 1 replica.
There was a similar issue reported here: https://github.com/bitnami/charts/issues/15702

postgresql-ha:
  enabled: true
  postgresql:
    replicaCount: 1

However when scaling up to replicaCount: 3 again, the error occurs again.
Unclear why atm. I would suggest to use the non-HA chart for the moment until we have gathered more information.

What helped was scaling down `postgresql-ha` to 1 replica. There was a similar issue reported here: https://github.com/bitnami/charts/issues/15702 ```yml postgresql-ha: enabled: true postgresql: replicaCount: 1 ``` However when scaling up to `replicaCount: 3` again, the error occurs again. Unclear why atm. I would suggest to use the non-HA chart for the moment until we have gathered more information.

pat-s commented

2023-10-10 12:40:13 +00:00

I've just tested the following again using postgresql-ha 11.9.2:

Bootstrapping a new install using replicaCount: 1 -> worked fine
Bootstrapping a new install using replicaCount: 3 -> worked fine

EKS 1.28.1

@ggbar @Delta1977 Is this still an issue for you?

I've just tested the following again using postgresql-ha 11.9.2: - Bootstrapping a new install using `replicaCount: 1` -> worked fine - Bootstrapping a new install using `replicaCount: 3` -> worked fine EKS 1.28.1 @ggbar @Delta1977 Is this still an issue for you?

ggbar commented

2023-10-11 03:47:14 +00:00

Do you mean postgresql-ha 11.9.3 fixes this issue? None postgresql is currently used. I'll test it later

pat-s commented

2023-10-12 09:03:56 +00:00

No, I tested with 11.9.2 and couldn't replicate the issue anymore (in contrast to my tries one month ago).

None postgresql is currently used.

What do you mean by that?

No, I tested with 11.9.2 and couldn't replicate the issue anymore (in contrast to my tries one month ago). > None postgresql is currently used. What do you mean by that?

ggbar commented

2023-10-12 09:07:37 +00:00

a slip of the pen
postgresql-ha 11.9.3 -> postgresql-ha 11.9.2
postgresql - postgresql-ha

a slip of the pen postgresql-ha 11.9.3 -> postgresql-ha 11.9.2 postgresql - postgresql-ha

ggbar commented

2023-10-12 09:22:19 +00:00

Maybe postgresql-ha 11.9.2 fixes this issue, I am using gitea helm version 9.2.1 postgresql-ha version 11.7.9

pat-s commented

2023-10-13 16:10:14 +00:00

I'd say give it a try. I was under the impression you already used 11.9.2 as it was released with 07fe17caf4 - which was released just right before you opened the issue.

We're importing downstream releases as they come in - there might always be small issues/fixes which we are not directly in control of.

I'd say give it a try. I was under the impression you already used 11.9.2 as it was released with 07fe17caf44401eb1da9dd364373030590bbc621 - which was released just right before you opened the issue. We're importing downstream releases as they come in - there might always be small issues/fixes which we are not directly in control of.

justusbunsi referenced a pull request that will close this issue

2023-10-14 15:36:42 +00:00

[postgresql-ha] Use pgpool service as db connection proxy #542

pat-s referenced a pull request that will close this issue

2023-10-14 16:05:41 +00:00

[postgresql-ha] Use pgpool service as db connection proxy #542

pat-s referenced this issue from a commit

2023-10-14 16:06:00 +00:00

[postgresql-ha] Use pgpool service as db connection proxy (#542)

pat-s closed this issue

2023-10-14 16:06:01 +00:00

PIE commented

2023-10-17 15:44:33 +00:00

I've just done a fresh install of 9.5.0 using pretty much the default Values (adding oauth, metrics, existing secret for admin) and gitea is failing to start because the configure-gitea init containers reporting:

No admin user 'gitea_admin' found. Creating now...
2023-10-17T16:38:51.797649581+01:00 Command error: CreateUser: pq: cannot set transaction read-write mode during recovery

Fixed it by deleting the pgpool pod which, I presume, forced it to reconnect to the db and this time it picked the primary/active db pod.

I've just done a fresh install of 9.5.0 using pretty much the default Values (adding oauth, metrics, existing secret for admin) and gitea is failing to start because the `configure-gitea` init containers reporting: ``` No admin user 'gitea_admin' found. Creating now... 2023-10-17T16:38:51.797649581+01:00 Command error: CreateUser: pq: cannot set transaction read-write mode during recovery ``` Fixed it by deleting the pgpool pod which, I presume, forced it to reconnect to the db and this time it picked the primary/active db pod.

justusbunsi commented

2023-10-17 15:48:47 +00:00

The fix is not yet released. If you like, you can test the current master branch version. I'd be happy for feedback that this actually fixes your problem.

PIE commented

2023-10-17 15:51:59 +00:00

Apologies, with the issue being closed I thought it'd been released, I should used a little more attention on the issue history.

justusbunsi commented

2023-10-17 15:53:00 +00:00

No worries. It was autoclosed with merging the referenced PR.

Sign in to join this conversation.

No Milestone

No Assignees

5 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gitea/helm-chart#502