JVM based dumps for troubleshooting WSO2 products related issues in a Kubernetes environment?

We all know that we are not analyzing heap dumps, thread dumps and thread usage dumps to debug JVM based related issues in a containerized environment. However if you need to get those dumps to analyze such an issue for WSO2 products, we can get them using below strategy.

Most of the time pod will be killed during a situation like out of memory. you can solve this problem in a Kubernetes environment using pod lifecycle events and some dumping threads at here with a shared file location.

Configure WSO2 product to collect heap dump

In a WSO2 product, you can configure product to create a heap dump in a mounted location during OOM situation which can access outside from the container using below config in wso2server.sh

-XX:HeapDumpPath=”/home/wso2carbon/dumps/Heap_dump_$HOSTNAME.hprof”

Here “/home/wso2carbon/dumps/” location is mounted to NFS based persistent volume where I can access from outside to download all these dumps.

Configure Kubernetes pod lifecycle event to trigger dumping scripts

You can get all these dumps before killing a pod as below,

You can use below lifecycle event definition to trigger dumping script in a pod killing situation.

preStop:
exec:
command: [‘/bin/bash’, ‘-c’, ‘/home/wso2carbon/scripts/prestop.sh’]

All these scripts can be found here and you can use same script under amis directory for WSO2 API manager and WSO2 IS based products. Please use scripts under analytics directory for WSO2 API manager analytics related products.

You can build your image by adding these scripts to the base image.

Basically you can utilize these scripts for almost all the java based application by adjusting parameters as you need for troubleshooting purposes.