Steps to capture QMetry Logs in event of Downtime - For Docker Based Installation (5) (5)

Owned by Mrunalini Potnis
Apr 13, 2020
3 min read

In case of QMetry Downtime or performance issue, QMetry support team needs the below information and logs to analyze and troubleshoot further. This information would be helpful to analyze memory utilization on server, exports/imports in progress (if any), running database processes and to capture relevant logs, before restarting the services.

Note : These steps are valid only for docker based installations.

Capture Tomcat Logs

  1. Create a directory "capturedlogs" under /opt/
    1. sudo su -
    2. cd /opt/
    3. mkdir capturedlogs
    4. Give required permissions as per the user capturing the logs
      • chmod 755 -R capturedlogs
      • chown root:root -R capturedlogs
  2. Execute the command : docker ps to view existing dockers running on the server. To go the the QMetry Tomcat Docker execute command : docker exec -it <docker_name> bash, example : docker exec -it qmetry85_qtm_1 bash
  3. Execute top command and take complete screenshot.
  4. Execute htop command and take complete screenshot. In case htop is not installed, run command : sudo yum install htop -y
  5. Execute free -m and check free memory available & ps aux | grep tomcat and take screenshot including both outputs.
  6. Take QMetry logs using the below steps.
    1. Go to QMetry logs directory:
      1. cd /home/qmetry/QMetry/qmetry-logs/qmetry/
      2. Copy / Note all the file names for the day that need to be copied and qmetry.log file
      3. cd /home/qmetry/QMetry/logs/
      4. Copy / Note all the file names for the day that need to be copied and catalina.out file
      5. cd /home/qmetry/QMetry/qmetry-logs/accessLog/      (available from 8.5 Build 5)
      6. Copy / Note all the file names for the given date and qmetry-access.log
      7. Type "exit" and quit docker.
    2. Copy all the logs files noted above for the given date to /opt/capturedlogs from the docker - replace docker name : eg - qmetry85_qtm_1 and <FILENAME> in below commads while copying all logs.
  7. Download JDK inside as below and Get Heap Memory dump and Thread dump:
    • Download JDK 
    • Get Heap Memory dump and Thread dump.
      • Get the java process ID using command : ps aux | grep java , in case of multiple PIDs, capture logs for both.
      • Make sure you are in the jdk bin directory.
      • Replace the $PID in this command to capture heap dump:
        •  /opt/jdk1.8.0_211/bin/jcmd $PID Thread.print > jcmd-threaddump.tdump
        •  /opt/jdk1.8.0_211/bin/jmap -dump:format=b,file=jmap-heapdump.hdump $PID
      • Run pwd command and copy location where dump file are stored along with the file names
      • Using exit command come out of docker
      • Copy dump file to main host  using below command.
        • sudo docker cp qmetry85_qtm_1:/{{path we got in PWD command }}/jcmd-threaddump.tdump /opt/capturedlogs/
        • sudo docker cp qmetry85_qtm_1:/ {{path we got in PWD command }}/jmap-heapdump.hdump /opt/capturedlogs/
  8. Enable automatic capturing of GC Logs and Heap Logs during on OutOfMemory Error on server (requires docker restart)
    • Execute the command : docker ps to view existing dockers running on the server.
    • To go the the QMetry Tomcat Docker execute command : docker exec -it <docker_name> bash, example : docker exec -it qmetry85_qtm_1 bash
    • Run command
      • cat /home/qmetry/QMetry/bin/setenv.sh
      • Important - Note the values for -Xms and -Xmx
    • Download the attached setenv.sh file place it under : /home/qmetry/QMetry/bin/setenv.sh
    • Important - Replace the values for -Xms and -Xmx noted in the above steps
    • Restart Docker Services.
      • Type exit, to exit the docker
      • docker ps to view list of dockers
      • docker stop qmetry85_qtm_1
      • docker start qmetry85_qtm_1
    • With this change the logs will be generated under /home/qmetry/QMetry/qmetry-logs/ directory in the docker. Provide us the log files “gc_$(date +%Y_%m_%d-%H_%M).log" and “heapdump-$(date +%Y_%m_%d-%H_%M)" generated under : /home/qmetry/QMetry/qmetry-logs/ directory.
    • Note down the file names for GC logs and type exit to exit the docker
    • Copy the GC logs using commands : docker cp qmetry85_qtm_1:/home/qmetry/QMetry/qmetry-logs/<FILENAME> /opt/capturedlogs

Capture MySQL Logs

  1. Execute the command : docker ps to view existing dockers running on the server.
  2. To go the the QMetry MySQL Docker execute command : docker exec -it <docker_name> bash, example : docker exec -it qmetry85_mysql_1 bash
  3. Capture MySQL Processes
    • Login to MySQL and execute the below commands:
      • mysql -u[username] -p[password] qmetry -A
      • tee /opt/mysqlprocesses.txt
      • show full processlist;
      • SHOW OPEN TABLES WHERE `Database` LIKE 'qmetry' AND In_use > 0;
      • show engine innodb status;
      • select * from information_schema.innodb_trx;
      • select * from information_schema.innodb_locks;
      • select * from information_schema.INNODB_LOCK_WAITS;
      • exit
    • Use exit command come out of docker
    • sudo docker cp qmetry85_mysql_1:/opt/mysqlprocesses.txt  /opt/capturedlogs
    • Share the this mysqlprocesses.txt with support copied with other logs in /opt/capturedlogs directory.

Copy the screenshots from steps 3,4 and 5, logs from /opt/capturedlogs directory along with the GC logs specified above.

Once this information and logs captured, share this with QMetry support team to investigate further.