Reverse-Logic wiki

Felhasználói eszközök

Eszközök a webhelyen

Nyomvonal: • monitoring
kubernetes:stackgres:monitoring

Ez a dokumentum egy előző változata!

Tartalomjegyzék

Kubernetes StackGres

Állapot ellenőrzés, monitorozás

Kubernetes állapot ellenőrzés

A következő utasítások futtatása előtt be kell állítani a névteret. 

export NAMESPACE=<névtér>
POD-ok ellenőrzése
$ kubectl get pods -n $NAMESPACE -o wide
NAME           READY   STATUS    RESTARTS     AGE   IP           NODE     NOMINATED NODE   READINESS GATES
pg-cluster-0   5/5     Running   6 (5d ago)   37d   10.244.0.5   kube01   <none>           <none>
pg-cluster-1   5/5     Running   7 (5d ago)   37d   10.244.2.5   kube03   <none>           <none>
pg-cluster-2   5/5     Running   8 (5d ago)   37d   10.244.1.8   kube02   <none>           <none>

Amennyiben hiba van (bármelyik POD állapota nem Running, akkor az alábbi ellenőrzések szükségesek a hibás POD nevét megadva (a példában ez a pg-cluster-0 nevű pod). 

$ kubectl describe pod pg-cluster-0 -n $NAMESPACE
Name:             pg-cluster-0
Namespace:        lapodb-postgres
Priority:         0
Service Account:  pg-cluster-patroni
Node:             kube01/192.168.110.161
Start Time:       Sun, 03 May 2026 17:31:59 +0200
Labels:           app=StackGresCluster
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=pg-cluster-7f856d76dc
                  nofailover=false
                  noloadbalance=false
                  role=primary
                  stackgres.io/cluster=true
                  stackgres.io/cluster-name=pg-cluster
                  stackgres.io/cluster-scope=pg-cluster
                  stackgres.io/cluster-uid=27cdba8f-3477-4740-848d-6f82c45c405a
                  stackgres.io/disruptible=true
                  statefulset.kubernetes.io/pod-name=pg-cluster-0
Annotations:      stackgres.io/cluster-controller-version: 1.18.6
                  stackgres.io/operatorVersion: 1.18.6
                  stackgres.io/patroni-version: 4.1.0
                  stackgres.io/pgbouncer-version: 1.25.1
                  stackgres.io/postgresql-version: 18.3
                  stackgres.io/prometheus-postgres-exporter-version: 0.19.0
                  status:
                    {"conn_url":"postgres://10.244.0.5:5432/postgres","api_url":"http://10.244.0.5:8009/patroni","state":"running","role":"primary","version":...
Status:           Running
IP:               10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  StatefulSet/pg-cluster
...
Node-Selectors:              node-role.kubernetes.io/postgres=true
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

POD-ban található konténerek nevének meghatározása. 

$ kubectl get pod pg-cluster-0 -n $NAMESPACE -o jsonpath='{.spec.containers[*].name}{"\n"}'
patroni pgbouncer prometheus-postgres-exporter postgres-util cluster-controller

A PostgreSQL példány a patroni konténerben fut. 

$ kubectl logs pg-cluster-0 -c patroni -n $NAMESPACE
...
2026-06-10 08:26:26 UTC [596361]: db=postgres,user=postgres,app=[unknown],client=127.0.0.1 LOG:  connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
2026-06-10 08:26:26 UTC [596361]: db=postgres,user=postgres,app=PostgreSQL JDBC Driver,client=127.0.0.1 LOG:  disconnection: session time: 0:00:00.026 user=postgres database=postgres host=127.0.0.1 port=48302
2026-06-10 08:26:33,319 INFO: no action. I am (pg-cluster-0), the leader with the lock
StackGres objektumok ellenőrzése
$ kubectl get sgcluster -n $NAMESPACE -o wide
NAME         VERSION   FLAVOR   INSTANCES   PROFILE              DISK   AGE   REPLICATION   BACKUP OBJECT STORAGE   BACKUP SCHEDULE
pg-cluster   18.3               3           pg-cluster-default   8Gi    37d   async                                 

$ kubectl describe sgcluster pg-cluster -n $NAMESPACE
$ kubectl describe sgcluster pg-cluster -n $NAMESPACE
Name:         pg-cluster
Namespace:    lapodb-postgres
Labels:       <none>
Annotations:  stackgres.io/operatorVersion: 1.18.6
API Version:  stackgres.io/v1
Kind:         SGCluster
Metadata:
  Creation Timestamp:  2026-05-03T15:25:56Z
  Generation:          17
  Resource Version:    8510436
  UID:                 27cdba8f-3477-4740-848d-6f82c45c405a
Spec:
  Configurations:
    Observability:
      Disable Metrics:      false
      Prometheus Autobind:  false
    Sg Pooling Config:      pg-cluster-default
    Sg Postgres Config:     pg-config
  Instances:                3
  Managed Sql:
    Continue On SGScript Error:  false
    Scripts:
      Id:         0
      Sg Script:  pg-cluster-default
  Pods:
    Disable Connection Pooling:  false
    Disable Envoy:               true
    Disable Metrics Exporter:    false
    Disable Postgres Util:       false
    Management Policy:           OrderedReady
    Persistent Volume:
      Size:           8Gi
      Storage Class:  local-postgres
    Scheduling:
      Node Selector:
        node-role.kubernetes.io/postgres:  true
      Pod Anti Affinity:
        Required During Scheduling Ignored During Execution:
          Label Selector:
            Match Labels:
              stackgres.io/cluster:  pg-cluster
          Topology Key:              kubernetes.io/hostname
    Update Strategy:
      Method:  InPlace
      Type:    OnlyDbOps
  Postgres:
    Flavor:  vanilla
    Ssl:
      Enabled:  true
    Version:    18
  Postgres Services:
    Primary:
      Enabled:  true
      Type:     ClusterIP
    Replicas:
      Enabled:  true
      Type:     ClusterIP
  Profile:      production
  Replication:
    Mode:               async
    Role:               ha-read
  Sg Instance Profile:  pg-cluster-default
Status:
  Arch:  x86_64
  Binding:
    Name:         pg-cluster-binding
  Build Version:  6.49
  Conditions:
    Last Transition Time:  2026-05-03T15:35:52.134132Z
    Reason:                ClusterInitialScriptApplied
    Status:                True
    Type:                  InitialScriptsApplied
    Last Transition Time:  2026-05-03T15:35:52.071432Z
    Reason:                ClusterBootstrapped
    Status:                True
    Type:                  Bootstrapped
    Last Transition Time:  2026-05-03T15:31:59.887576Z
    Reason:                FalsePendingUpgrade
    Status:                False
    Type:                  PendingUpgrade
    Last Transition Time:  2026-05-03T15:31:59.887531Z
    Reason:                FalsePendingRestart
    Status:                False
    Type:                  PendingRestart
    Last Transition Time:  2026-05-03T15:31:59.313942Z
    Reason:                FalseFailed
    Status:                False
    Type:                  Failed
  Extensions:
  Instances:       3
  Label Selector:  app=StackGresCluster,stackgres.io/cluster-name=pg-cluster,stackgres.io/cluster-scope=pg-cluster,stackgres.io/cluster=true,stackgres.io/cluster-uid=27cdba8f-3477-4740-848d-6f82c45c405a
  Managed Sql:
    Scripts:
      Completed At:  2026-05-03T15:35:47.613868Z
      Id:            0
      Scripts:
        Id:        0
        Version:   0
        Id:        4
        Version:   0
      Started At:  2026-05-03T15:35:44.096419Z
      Updated At:  2026-05-03T15:35:47.193412Z
  Os:              linux
  Pod Statuses:
    Installed Postgres Extensions:
    Name:               pg-cluster-0
    Node Name:          kube01
    Pending Restart:    false
    Primary:            true
    Replication Group:  0
    Installed Postgres Extensions:
    Name:               pg-cluster-1
    Node Name:          kube03
    Pending Restart:    false
    Primary:            false
    Replication Group:  0
    Installed Postgres Extensions:
    Name:               pg-cluster-2
    Node Name:          kube02
    Pending Restart:    false
    Primary:            false
    Replication Group:  0
  Postgres Version:     18.3
Events:                 <none>

$ kubectl get events -n $NAMESPACE --sort-by=.lastTimestamp
LAST SEEN   TYPE      REASON               OBJECT                           MESSAGE
1s          Warning   BackupConfigFailed   sgbackup/bk2026-05-03-17-52-04   SGBackup reconciliation cycle failed: SGCluster pg-cluster has no backup configured
StatefulSet ellenőrzése
$ kubectl get statefulset -n $NAMESPACE -o wide
NAME         READY   AGE   CONTAINERS                                                                        IMAGES
pg-cluster   3/3     37d   patroni,pgbouncer,prometheus-postgres-exporter,postgres-util,cluster-controller   quay.io/ongres/patroni:v4.1.0-pg18.3-build-6.49,quay.io/ongres/pgbouncer:v1.25.1-build-6.49,quay.io/ongres/prometheus-postgres-exporter:v0.19.0-build-6.49,quay.io/ongres/postgres-util:v18.3-build-6.49,quay.io/stackgres/cluster-controller:1.18.6
PVC-k ellenőrzése
$ kubectl get pvc -n $NAMESPACE -o wide
NAME                           STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE   VOLUMEMODE
pg-cluster-data-pg-cluster-0   Bound    local-pg-kube01   8Gi        RWO            local-postgres   <unset>                 37d   Filesystem
pg-cluster-data-pg-cluster-1   Bound    local-pg-kube03   8Gi        RWO            local-postgres   <unset>                 37d   Filesystem
pg-cluster-data-pg-cluster-2   Bound    local-pg-kube02   8Gi        RWO            local-postgres   <unset>                 37d   Filesystem

Patroni állapot ellenőrzés

Normál állapot
$ kubectl exec -n $NAMESPACE -it pg-cluster-0 -c patroni -- patronictl list
+ Cluster: pg-cluster (7635689612010042939) ------+----+-------------+-----+------------+-----+
| Member       | Host       | Role    | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------------+------------+---------+-----------+----+-------------+-----+------------+-----+
| pg-cluster-0 | 10.244.0.5 | Leader  | running   |  2 |             |     |            |     |
| pg-cluster-1 | 10.244.2.5 | Replica | streaming |  2 |   0/40002B0 |   0 |  0/40002B0 |   0 |
| pg-cluster-2 | 10.244.1.8 | Replica | streaming |  2 |   0/40002B0 |   0 |  0/40002B0 |   0 |
+--------------+------------+---------+-----------+----+-------------+-----+------------+-----+

Egy replika node kiesése 

$ kubectl exec -n $NAMESPACE -it pg-cluster-0 -c patroni -- patronictl list
+ Cluster: pg-cluster (7635689612010042939) ------+----+-------------+-----+------------+-----+
| Member       | Host       | Role    | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------------+------------+---------+-----------+----+-------------+-----+------------+-----+
| pg-cluster-0 | 10.244.0.5 | Leader  | running   |  2 |             |     |            |     |
| pg-cluster-1 | 10.244.2.5 | Replica | streaming |  2 |   0/40002B0 |   0 |  0/40002B0 |   0 |
| pg-cluster-2 |            | Replica |           |    |     unknown |     |    unknown |     |
+--------------+------------+---------+-----------+----+-------------+-----+------------+-----+
Leader node kiesése
# kubectl get nodes
NAME     STATUS     ROLES                    AGE   VERSION
kube01   NotReady   control-plane,postgres   38d   v1.36.0
kube02   Ready      control-plane,postgres   38d   v1.36.0
kube03   Ready      control-plane,postgres   38d   v1.36.0

# kubectl get pods -n $NAMESPACE
NAME           READY   STATUS        RESTARTS       AGE
pg-cluster-0   5/5     Terminating   6 (5d2h ago)   37d
pg-cluster-1   5/5     Running       7 (5d2h ago)   37d
pg-cluster-2   5/5     Running       1 (15m ago)    16m

# kubectl exec -n $NAMESPACE pg-cluster-1 -c patroni -- patronictl list
2026-06-10 09:43:17,883 - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError("HTTPSConnection(host='10.96.0.1', port=443): Failed to establish a new connection: [Errno 113] No route to host")': /api/v1/namespaces/lapodb-postgres/endpoints?labelSelector=app%3DStackGresCluster%2Cstackgres.io%2Fcluster%3Dtrue%2Cstackgres.io%2Fcluster-scope%3Dpg-cluster
+ Cluster: pg-cluster (7635689612010042939) ------+----+-------------+-----+------------+-----+
| Member       | Host       | Role    | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------------+------------+---------+-----------+----+-------------+-----+------------+-----+
| pg-cluster-0 | 10.244.0.5 | Replica | running   |  2 |     unknown |     |  0/40002B0 |   0 |
| pg-cluster-1 | 10.244.2.5 | Leader  | running   |  3 |             |     |            |     |
| pg-cluster-2 | 10.244.1.9 | Replica | streaming |  3 |   0/40004C0 |   0 |  0/40004C0 |   0 |
+--------------+------------+---------+-----------+----+-------------+-----+------------+-----+

# kubectl exec -n $NAMESPACE pg-cluster-2 -c patroni -- patronictl list
+ Cluster: pg-cluster (7635689612010042939) ------+----+-------------+-----+------------+-----+
| Member       | Host       | Role    | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------------+------------+---------+-----------+----+-------------+-----+------------+-----+
| pg-cluster-0 | 10.244.0.5 | Replica | running   |  2 |     unknown |     |  0/40002B0 |   0 |
| pg-cluster-1 | 10.244.2.5 | Leader  | running   |  3 |             |     |            |     |
| pg-cluster-2 | 10.244.1.9 | Replica | streaming |  3 |   0/40004C0 |   0 |  0/40004C0 |   0 |
+--------------+------------+---------+-----------+----+-------------+-----+------------+-----+

Patroni automatikus failover után (10-60 másodperc) elérhető a szolgáltatás. 

$ kubectl run psql-test -n $NAMESPACE --rm -it --image=postgres:17 -- bash
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.

root@psql-test:/# pg_isready -h pg-cluster-primary -p 5432
pg-cluster-primary:5432 - accepting connections

root@psql-test:/# exit
exit
Session ended, resume using 'kubectl attach psql-test -c psql-test -n lapodb-postgres -i -t' command
pod "psql-test" deleted from lapodb-postgres namespace

$ kubectl exec -n $NAMESPACE -it pg-cluster-1 -c patroni -- patronictl list
+ Cluster: pg-cluster (7635689612010042939) -------+----+-------------+-----+------------+-----+
| Member       | Host        | Role    | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------------+-------------+---------+-----------+----+-------------+-----+------------+-----+
| pg-cluster-0 | 10.244.0.9  | Replica | streaming |  4 |   0/40014F0 |   0 |  0/40014F0 |   0 |
| pg-cluster-1 | 10.244.2.16 | Replica | streaming |  4 |   0/40014F0 |   0 |  0/40014F0 |   0 |
| pg-cluster-2 | 10.244.1.9  | Leader  | running   |  4 |             |     |            |     |
+--------------+-------------+---------+-----------+----+-------------+-----+------------+-----+

PostgreSQL állapot ellenőrzés

Streaming replikáció állapota
$ kubectl exec -n $NAMESPACE pg-cluster-1 -c patroni -- patronictl list
+ Cluster: pg-cluster (7635689612010042939) -------+----+-------------+-----+------------+-----+
| Member       | Host        | Role    | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------------+-------------+---------+-----------+----+-------------+-----+------------+-----+
| pg-cluster-0 | 10.244.0.9  | Replica | streaming |  4 |   0/40014F0 |   0 |  0/40014F0 |   0 |
| pg-cluster-1 | 10.244.2.16 | Replica | streaming |  4 |   0/40014F0 |   0 |  0/40014F0 |   0 |
| pg-cluster-2 | 10.244.1.9  | Leader  | running   |  4 |             |     |            |     |
+--------------+-------------+---------+-----------+----+-------------+-----+------------+-----+

A leaderen kell ellenőrizni a replikáció állapotát 

$ kubectl exec -n $NAMESPACE -it pg-cluster-2 -c patroni -- psql -U postgres -c "SELECT application_name, client_addr, state, sync_state FROM pg_stat_replication"
Pager usage is off.
 application_name | client_addr |   state   | sync_state 
------------------+-------------+-----------+------------
 pg-cluster-0     | 10.244.0.9  | streaming | async
 pg-cluster-1     | 10.244.2.16 | streaming | async
(2 rows)

Részletesebb megjelenítés 

$ kubectl exec -n $NAMESPACE -it pg-cluster-2 -c patroni -- psql -U postgres -x -c "SELECT * FROM pg_stat_replication"
Pager usage is off.
-[ RECORD 1 ]----+------------------------------
pid              | 9267
usesysid         | 16384
usename          | replicator
application_name | pg-cluster-0
client_addr      | 10.244.0.9
client_hostname  | 
client_port      | 41370
backend_start    | 2026-06-10 11:05:15.664685+00
backend_xmin     | 
state            | streaming
sent_lsn         | 0/40014F0
write_lsn        | 0/40014F0
flush_lsn        | 0/40014F0
replay_lsn       | 0/40014F0
write_lag        | 
flush_lag        | 
replay_lag       | 
sync_priority    | 0
sync_state       | async
reply_time       | 2026-06-10 11:42:47.125+00
-[ RECORD 2 ]----+------------------------------
pid              | 9962
usesysid         | 16384
usename          | replicator
application_name | pg-cluster-1
client_addr      | 10.244.2.16
client_hostname  | 
client_port      | 47432
backend_start    | 2026-06-10 11:13:44.348486+00
backend_xmin     | 
state            | streaming
sent_lsn         | 0/40014F0
write_lsn        | 0/40014F0
flush_lsn        | 0/40014F0
replay_lsn       | 0/40014F0
write_lag        | 
flush_lag        | 
replay_lag       | 
sync_priority    | 0
sync_state       | async
reply_time       | 2026-06-10 11:42:47.158991+00

pid WAL sender PostgreSQL folyamat PID-je a leaderen usesysid Kapcsolódó PostgreSQL felhasználó azonosítója usename Replikációs kapcsolatot használó felhasználó (replicator) application_name Replika neve. Patroni használatakor a pod neve client_addr Replika IP címe client_hostname DNS név, amennyiben feloldható client_port Replika TCP forrásportja backend_start Replikációs kapcsolat létrejöttének ideje backend_xmin Hot standby VACUUM védelemhez használt xmin.

kubernetes/stackgres/monitoring.1781092216.txt.gz · legutóbbi módosítás: – Szerkesztő: riba.zoltan