Troubleshooting v1
In this page, you can find some basic information on how to troubleshoot EDB Postgres for Kubernetes in your Kubernetes cluster deployment.
Hint
As a Kubernetes administrator, you should have the
kubectl
Cheat Sheet page
bookmarked!
Before you start
Kubernetes environment
What can make a difference in a troubleshooting activity is to provide clear information about the underlying Kubernetes system.
Make sure you know:
- the Kubernetes distribution and version you are using
- the specifications of the nodes where PostgreSQL is running
- as much as you can about the actual storage, including storage class and benchmarks you have done before going into production.
- which relevant Kubernetes applications you are using in your cluster (i.e. Prometheus, Grafana, Istio, Certmanager, ...)
- the situation of continuous backup, in particular if it's in place and working correctly: in case it is not, make sure you take an emergency backup before performing any potential disrupting operation
Useful utilities
On top of the mandatory kubectl
utility, for troubleshooting, we recommend the
following plugins/utilities to be available in your system:
cnp
plugin forkubectl
jq
, a lightweight and flexible command-line JSON processorgrep
, searches one or more input files for lines containing a match to a specified pattern. It is already available in most *nix distros. If you are on Windows OS, you can usefindstr
as an alternative togrep
or directly usewsl
and install your preferred *nix distro and use the tools mentioned above.
First steps
To quickly get an overview of the cluster or installation, the kubectl
plugin
is the primary tool to use:
- the status subcommand provides an overview of a cluster
- the report subcommand provides the manifests
for clusters and the operator deployment. It can also include logs using
the
--logs
option. The report generated via the plugin will include the full cluster manifest.
The plugin can be installed on air-gapped systems via packages. Please refer to the plugin document for complete instructions.
Are there backups?
After getting the cluster manifest with the plugin, you should verify if backups are set up and working.
In a cluster with backups set up, you will find, in the cluster Status, the fields
lastSuccessfulBackup
and firstRecoverabilityPoint
. You should make sure
there is a recent lastSuccessfulBackup
.
A cluster lacking the .spec.backup
stanza won't have backups.
An insistent message will appear in the PostgreSQL logs:
Before proceeding with troubleshooting operations, it may be advisable to perform an emergency backup depending on your findings regarding backups. Refer to the following section for instructions.
It is extremely risky to operate a production database without keeping regular backups.
Emergency backup
In some emergency situations, you might need to take an emergency logical
backup of the main app
database.
Important
The instructions you find below must be executed only in emergency situations
and the temporary backup files kept under the data protection policies
that are effective in your organization. The dump file is indeed stored
in the client machine that runs the kubectl
command, so make sure that
all protections are in place and you have enough space to store the
backup file.
The following example shows how to take a logical backup of the app
database
in the cluster-example
Postgres cluster, from the cluster-example-1
pod:
Note
You can easily adapt the above command to backup your cluster, by providing the names of the objects you have used in your environment.
The above command issues a pg_dump
command in custom format, which is the most
versatile way to take logical backups in PostgreSQL.
The next step is to restore the database. We assume that you are operating
on a new PostgreSQL cluster that's been just initialized (so the app
database
is empty).
The following example shows how to restore the above logical backup in the
app
database of the new-cluster-example
Postgres cluster, by connecting to
the primary (new-cluster-example-1
pod):
Important
The example in this section assumes that you have no other global objects
(databases and roles) to dump and restore, as per our recommendation. In case
you have multiple roles, make sure you have taken a backup using pg_dumpall -g
and you manually restore them in the new cluster. In case you have multiple
databases, you need to repeat the above operation one database at a time, making
sure you assign the right ownership. If you are not familiar with PostgreSQL,
we advise that you do these critical operations under the guidance of
a professional support company.
The above steps might be integrated into the cnp
plugin at some stage in the future.
Logs
Every resource created and controlled by EDB Postgres for Kubernetes logs to
standard output, as expected by Kubernetes, and directly in JSON
format. As a result, you should rely on the kubectl logs
command to retrieve logs from a given resource.
For more information, type:
Hint
JSON logs are great for machine reading, but hard to read for human beings.
Our recommendation is to use the jq
command to improve usability. For
example, you can pipe the kubectl logs
command with | jq -C
.
Note
In the sections below, we will show some examples on how to retrieve logs about different resources when it comes to troubleshooting EDB Postgres for Kubernetes.
Operator information
By default, the EDB Postgres for Kubernetes operator is installed in the
postgresql-operator-system
namespace in Kubernetes as a Deployment
(see the "Details about the deployment" section
for details).
You can get a list of the operator pods by running:
Note
Under normal circumstances, you should have one pod where the operator is
running, identified by a name starting with postgresql-operator-controller-manager-
.
In case you have set up your operator for high availability, you should have more entries.
Those pods are managed by a deployment named postgresql-operator-controller-manager
.
Collect the relevant information about the operator that is running in pod
<POD>
with:
Then get the logs from the same pod by running:
Gather more information about the operator
Get logs from all pods in EDB Postgres for Kubernetes operator Deployment (in case you have a multi operator deployment) by running:
Tip
You can add -f
flag to above command to follow logs in real time.
Save logs to a JSON file by running:
Get EDB Postgres for Kubernetes operator version by using kubectl-cnp
plugin:
Output:
Cluster information
You can check the status of the <CLUSTER>
cluster in the NAMESPACE
namespace with:
Output:
The above example reports a healthy PostgreSQL cluster of 3 instances, all in
ready state, and with <CLUSTER>-1
being the primary.
In case of unhealthy conditions, you can discover more by getting the manifest
of the Cluster
resource:
Another important command to gather is the status
one, as provided by the
cnp
plugin:
Tip
You can print more information by adding the --verbose
option.
Note
Besides knowing cluster status, you can also do the following things with the cnp plugin:
Promote a replica.
Manage certificates.
Make a rollout restart cluster to apply configuration changes.
Make a reconciliation loop to reload and apply configuration changes.
For more information, please see cnp
plugin documentation.
Get EDB PostgreSQL Advanced Server (EPAS) / PostgreSQL container image version:
Output:
Note
Also you can use kubectl-cnp status -n <NAMESPACE> <CLUSTER_NAME>
to get the same information.
Pod information
You can retrieve the list of instances that belong to a given PostgreSQL cluster with:
Output:
You can check if/how a pod is failing by running:
You can get all the logs for a given PostgreSQL instance with:
If you want to limit the search to the PostgreSQL process only, you can run:
The following example also adds the timestamp in a user-friendly format:
Gather and filter extra information about PostgreSQL pods
Check logs from a specific pod that has crashed:
Get FATAL errors from a specific PostgreSQL pod:
Output:
Filter PostgreSQL DB error messages in logs for a specific pod:
Output:
Get messages matching err
word from a specific pod:
Output:
Get all logs from PostgreSQL process from a specific pod:
Output:
Get pod logs filtered by fields with values and join them separated by |
running:
Output:
Backup information
You can list the backups that have been created for a named cluster with:
Important
Backup labelling has been introduced in version 1.10.0 of EDB Postgres for Kubernetes. So only those resources that have been created with that version or a higher one will contain such a label.
Storage information
Sometimes is useful to double-check the StorageClass used by the cluster to have some more context during investigations or troubleshooting, like this:
We are taking the StorageClass from one of the cluster pod here since often clusters are created using the default StorageClass.
Node information
Kubernetes nodes is where ultimately PostgreSQL pods will be running. It's strategically important to know as much as we can about them.
You can get the list of nodes in your Kubernetes cluster with:
Additionally, you can gather the list of nodes where the pods of a given cluster are running with:
The latter is important to understand where your pods are distributed - very useful if you are using affinity/anti-affinity rules and/or tolerations.
Conditions
Like many native kubernetes
objects like here,
Cluster exposes status.conditions
as well. This allows one to 'wait' for a particular
event to occur instead of relying on the overall cluster health state. Available conditions as of now are:
- LastBackupSucceeded
- ContinuousArchiving
- Ready
LastBackupSucceeded
is reporting the status of the latest backup. If set to True
the
last backup has been taken correctly, it is set to False
otherwise.
ContinuousArchiving
is reporting the status of the WAL archiving. If set to True
the
last WAL archival process has been terminated correctly, it is set to False
otherwise.
Ready
is True
when the cluster has the number of instances specified by the user
and the primary instance is ready. This condition can be used in scripts to wait for
the cluster to be created.
How to wait for a particular condition
- Backup:
- ContinuousArchiving:
- Ready (Cluster is ready or not):
Below is a snippet of a cluster.status
that contains a failing condition.
Networking
EDB Postgres for Kubernetes requires basic networking and connectivity in place. You can find more information in the networking section.
If installing EDB Postgres for Kubernetes in an existing environment, there might be network policies in place, or other network configuration made specifically for the cluster, which could have an impact on the required connectivity between the operator and the cluster pods and/or the between the pods.
You can look for existing network policies with the following command:
There might be several network policies set up by the Kubernetes network administrator.
PostgreSQL core dumps
Although rare, PostgreSQL can sometimes crash and generate a core dump
in the PGDATA
folder. When that happens, normally it is a bug in PostgreSQL
(and most likely it has already been solved - this is why it is important
to always run the latest minor version of PostgreSQL).
EDB Postgres for Kubernetes allows you to control what to include in the core dump through
the k8s.enterprisedb.io/coredumpFilter
annotation.
Info
Please refer to "Labels and annotations" for more details on the standard annotations that EDB Postgres for Kubernetes provides.
By default, the k8s.enterprisedb.io/coredumpFilter
is set to 0x31
in order to
exclude shared memory segments from the dump, as this is the safest
approach in most cases.
Info
Please refer to
"Core dump filtering settings" section of "The /proc
Filesystem" page of the Linux Kernel documentation.
for more details on how to set the bitmask that controls the core dump filter.
Important
Beware that this setting only takes effect during Pod startup and that changing the annotation doesn't trigger an automated rollout of the instances.
Although you might not personally be involved in inspecting core dumps,
you might be asked to provide them so that a Postgres expert can look
into them. First, verify that you have a core dump in the PGDATA
directory with the following command (please run it against the
correct pod where the Postgres instance is running):
Under normal circumstances, this should return an empty set. Suppose, for example, that we have a core dump file:
Once you have verified the space on disk is sufficient, you can collect the
core dump on your machine through kubectl cp
as follows:
You now have the file. Make sure you free the space on the server by removing the core dumps.
Some common issues
Storage is full
If one or more pods in the cluster are in CrashloopBackoff
and logs
suggest this could be due to a full disk, you probably have to increase the
size of the instance's PersistentVolumeClaim
. Please look at the
"Volume expansion" section in the documentation.
Pods are stuck in Pending
state
In case a Cluster's instance is stuck in the Pending
phase, you should check
the pod's Events
section to get an idea of the reasons behind this:
Some of the possible causes for this are:
- No nodes are matching the
nodeSelector
- Tolerations are not correctly configured to match the nodes' taints
- No nodes are available at all: this could also be related to
cluster-autoscaler
hitting some limits, or having some temporary issues
In this case, it could also be useful to check events in the namespace:
Replicas out of sync when no backup is configured
Sometimes replicas might be switched off for a bit of time due to maintenance
reasons (think of when a Kubernetes nodes is drained). In case your cluster
does not have backup configured, when replicas come back up, they might
require a WAL file that is not present anymore on the primary (having been
already recycled according to the WAL management policies as mentioned in
"The postgresql
section"), and
fall out of synchronization.
Similarly, when pg_rewind
might require a WAL file that is not present
anymore in the former primary, reporting pg_rewind: error: could not open file
.
In these cases, pods cannot become ready anymore, and you are required to delete the PVC and let the operator rebuild the replica.
If you rely on dynamically provisioned Persistent Volumes, and you are confident in deleting the PV itself, you can do so with:
Cluster stuck in Creating new replica
Cluster is stuck in "Creating a new replica", while pod logs don't show relevant problems. This has been found to be related to the next issue on connectivity. From releases 1.20.1, 1.19.3, and 1.18.5, networking issues will be more clearly reflected in the status column as follows:
Networking is impaired by installed Network Policies
As pointed out in the networking section, local network policies could prevent some of the required connectivity.
A tell-tale sign that connectivity is impaired is the presence in the operator logs of messages like:
You should list the network policies, and look for any policies restricting connectivity.
For example, in the listing above, default-deny-ingress
seems a likely culprit.
You can drill into it:
In the networking page you can find a network policy file
that you can customize to create a NetworkPolicy
explicitly allowing the
operator to connect cross-namespace to cluster pods.
Error while bootstrapping the data directory
If your Cluster's initialization job crashes with a "Bus error (core dumped) child process exited with exit code 135", you likely need to fix the Cluster hugepages settings.
The reason is the incomplete support of hugepages in the cgroup v1 that should be fixed in v2. For more information, check the PostgreSQL BUG #17757: Not honoring huge_pages setting during initdb causes DB crash in Kubernetes.
To check whether hugepages are enabled, run grep HugePages /proc/meminfo
on
the Kubernetes node and check if hugepages are present, their size, and how many
are free.
If the hugepages are present, you need to configure how much hugepages memory every PostgreSQL pod should have available.
For example:
Please remember that you must have enough hugepages memory available to schedule every Pod in the Cluster (in the example above, at least 512MiB per Pod must be free).