Steps to capture QMetry Logs in event of Downtime - For non docker based installation (5) (5)

Owned by Mrunalini Potnis
Apr 13, 2020
3 min read

In case of QMetry Downtime or performance issue, QMetry support team needs the below information and logs to analyze and troubleshoot further. This information would be helpful to analyze memory utilization on server, exports/imports in progress (if any), running database processes and to capture relevant logs, before restarting the services.

Note : These steps are valid only for non-docker based installations.

Capture Tomcat Logs

  1. Create a directory "capturedlogs" under /opt/
    1. sudo su -
    2. cd /opt/
    3. mkdir capturedlogs
    4. Give required permissions as per the user capturing the logs
      • chmod 755 -R capturedlogs
      • chown root:root -R capturedlogs
  2. Execute top command and take complete screenshot.
  3. Execute htop command and take complete screenshot. In case htop is not installed, run command : sudo yum install htop -y to install it.
  4. Execute free -m and check free memory available & ps aux | grep tomcat and take screenshot including both outputs.
  5. Take QMetry logs by following below steps.
    1. Go to QMetry logs directory:
      1. cd /home/qmetry/QMetry/qmetry-logs/qmetry/
      2. Copy qmetry.log and all the files for the given date from here to /opt/capturedlogs
      3. cd /home/qmetry/QMetry/qmetry-logs/accessLog/      (available from 8.5 Build 5)
      4. Copy qmetry-access.log and all the files for the given date from here to /opt/capturedlogs
      5. cd /home/qmetry/QMetry/logs/
      6. Copy catalina.out and all the files for the given date from here to /opt/capturedlogs
  6. Download JDK as follows and Get Heap Memory dump and Thread dump:
    • Download JDK 
    • Get Heap Memory dump and Thread dump
      • Get the java process ID using command : ps aux | grep java , in case of multiple PIDs, capture logs for both.
      • Make sure you are in the jdk bin directory.
      • Replace the $PID in this command to capture heap dump:
        • /opt/jdk1.8.0_211/bin/jstack -l $PID > jstack-threaddump.tdump
          • If the above command doesn't respond use the below command instead to take force dump.
          • /opt/jdk1.8.0_211/bin/jstack -F -l $PID > jstack-threaddump.tdump
        • /opt/jdk1.8.0_211/bin/jcmd $PID Thread.print > jcmd-threaddump.tdump
        • /opt/jdk1.8.0_211/bin/jmap -dump:format=b,file=jmap-heapdump.hdump $PID 
      • Copy the dump files created in the /opt/jdk1.8.0_211/bin directory to the /opt/capturedlogs folder
        • cp jstack-threaddump.tdump /opt/capturedlogs/
        • cp jcmd-threaddump.tdump /opt/capturedlogs/
        • cp jmap-heapdump.hdump /opt/capturedlogs/
  7. Enable automatic capturing of GC Logs and Heap Logs during on OutOfMemory Error on server (requires Tomcat restart)
    • Run command : cat /home/qmetry/QMetry/bin/setenv.sh and note the values of -Xms and -Xmx
    • Download the attached setenv.sh file place it under : /home/qmetry/QMetry/bin/setenv.sh
    • Important - Update the values for -Xms and -Xmx noted in the above steps
    • With this change (and restart of tomcat in next step) the logs will be generated under /home/qmetry/QMetry/qmetry-logs/ directory. Provide us the log files “gc_$(date +%Y_%m_%d-%H_%M).log" and “heapdump-$(date +%Y_%m_%d-%H_%M)" generated under : /home/qmetry/QMetry/qmetry-logs/ directory.
    • Copy the logs using commands : cp /home/qmetry/QMetry/qmetry-logs/<FILENAME> /opt/capturedlogs
  8. Enabling extra logging with detailed timestamps and user information
    • Go to tomcat 'conf' directory : cd /home/qmetry/QMetry/conf. 

    • At the end of the file there is an element like below : Replace it with below values.Restart Tomcat Services : 

      • cd /home/qmetry/QMetry/bin
      • ./shutdown.sh
      • ./startup.sh

Capture MySQL Logs

  1. Capture MySQL Processes
    • Login to MySQL and execute the below commands:
      • mysql -u[username] -p[password] qmetry -A  OR /opt/mysql/bin/mysql -u[usename] -p[password] qmetry -A
      • tee /opt/mysqlprocesses.txt
      • show full processlist;
      • SHOW OPEN TABLES WHERE `Database` LIKE 'qmetry' AND In_use > 0;
      • show engine innodb status;
      • select * from information_schema.innodb_trx;
      • select * from information_schema.innodb_locks;
      • select * from information_schema.INNODB_LOCK_WAITS;
      • exit
    • Copy the mysqlprocess.txt generated under /opt to /opt/capturedlogs
    • Share the this mysqlprocesses.txt with support copied with other logs in /opt/capturedlogs directory.

Note - Copy the screenshots from steps 2,3 and 4, logs from /opt/capturedlogs directory along with the GC logs specified above.

Once this information and logs captured, share this with QMetry support team to investigate further.