Steps to capture QMetry Logs to troubleshoot performance issues or downtimes.

In case of QMetry Downtime or performance issue, QMetry support team needs the below information and logs to analyze and troubleshoot further. This information would be helpful to analyze memory utilization on server, exports/imports in progress (if any), running database processes and to capture relevant logs, before restarting the services.

Note : These steps apply only to docker based installations.

Create a directory "capturedlogs" under /opt/

• sudo su -
• cd /opt/
• mkdir capturedlogs
• Give required permissions as per the user capturing the logs
  • chmod 755 -R capturedlogs
  • chown root:root -R capturedlogs

Capture MySQL Logs

1. Execute the commands below to login to the MySQL docker
   • sudo su -
   • docker exec -it qmetry_rds_1 bash
2. Capture MySQL Processes
   • Login to MySQL and execute the below commands:
     • mysql -u[username] -p[password] qmetry -A
     • tee /opt/mysqlprocesses.txt
     • show full processlist;
     • SHOW OPEN TABLES WHERE `Database` LIKE 'qmetry' AND In_use > 0;
     • show engine innodb status;
     • select * from information_schema.innodb_trx;
     • select * from information_schema.innodb_locks;
     • select * from information_schema.INNODB_LOCK_WAITS;
     • exit
   • Use exit command come out of docker
   • sudo docker cp qmetry_rds_1:/opt/mysqlprocesses.txt  /opt/capturedlogs
   • Share the this mysqlprocesses.txt with support copied with other logs in /opt/capturedlogs directory.

Capture Tomcat Logs

1. Execute the commands below to login to the QMetry tomcat application docker
   • sudo su -
   • docker exec -it qmetry_app_1 bash
2. Execute top command and take complete screenshot.
3. Execute htop command and take complete screenshot. In case htop is not installed, run command : sudo yum install htop -y
4. Execute free -m and check free memory available & ps aux | grep tomcat and take screenshot including both outputs.
5. Go to QMetry logs directory:
   • Exit dockers if using command : exit and go to the below logs location outside the docker
   • cd /var/lib/docker/volumes/qmetry_logs/_data/
   • ls -ltr to view the files list. Copy all logs created for a given date along with catalina.out file.
   • cp <file_name1> <file_name2> /opt/capturedlogs
   • cd /var/lib/docker/volumes/qmetry_logs/_data/qmetry-log/
   • ls -ltr to view the files list. Copy all logs created for a given date along with qmetry-access.log and qmetry.log
   • cp <file_name1> <file_name2> /opt/capturedlogs
6. Copy all the logs files noted above for the given date to /opt/capturedlogs.
7. Get Heap Memory dump and Thread dump by following below steps.
   • Go to the JDK directory in the docker
     • sudo su -
     • docker exec -it qmetry_app_1 bash
     • cd /opt/jdk-11.0.2    (Change depending on the jdk folder name)
   • Get Heap Memory dump and Thread dump.
     • Get the java process ID using command : ps aux | grep java , in case of multiple PIDs, capture logs for both.
     • Make sure you are in the jdk bin directory.
     • Replace the $PID in this command to capture heap dump:
       •  /opt/jdk-11.0.2/bin/jcmd $PID Thread.print > jcmd-threaddump.tdump
       •  /opt/jdk-11.0.2/bin/jmap -dump:format=b,file=jmap-heapdump.hdump $PID
     • Run pwd command and copy location where dump file are stored along with the file names
     • Using exit command come out of docker
     • Copy dump file to main host  using below command.
       • sudo docker cp qmetry_app_1:/{{path we got in PWD command }}/jcmd-threaddump.tdump /opt/capturedlogs/
       • sudo docker cp qmetry_app_1:/ {{path we got in PWD command }}/jmap-heapdump.hdump /opt/capturedlogs/
8. Enable automatic capturing of GC Logs and Heap Logs during on OutOfMemory Error on server (requires docker restart)
   • Execute the commands below to login to the tomcat docker: 
     • sudo su -
     • docker exec -it qmetry_app_1 bash
   • Run command
     • cat /home/qmetry/QMetry/bin/setenv.sh
     • Important - Note the values for -Xms and -Xmx
   • Download the attached setenv.sh file place it under : /home/qmetry/QMetry/bin/setenv.sh
   • Important - Replace the values for -Xms and -Xmx noted in the above steps
   • Restart Docker Services.
     • Type exit, to exit the docker
     • docker stop qmetry_app_1 qmetry_rds_1
     • docker start qmetry_rds_1
     • docker start qmetry_app_1
   • With this change the logs will be generated under /home/qmetry/QMetry/qmetry-logs/ directory in the docker. Provide us the log files “gc_$(date +%Y_%m_%d-%H_%M).log" and “heapdump-$(date +%Y_%m_%d-%H_%M)" generated under : /home/qmetry/QMetry/qmetry-logs/ directory.
   • Note down the file names for GC logs and type exit to exit the docker
   • Copy the GC logs using commands : docker cp qmetry85_qtm_1:/home/qmetry/QMetry/qmetry-logs/<FILENAME> /opt/capturedlogs

Copy the screenshots from steps 2,3 and 4, logs from /opt/capturedlogs directory along with the GC logs specified above.

Once this information and logs captured, share this with QMetry support team to investigate further.