Every action in OpManager leaves a trace(s) in the log files. There are two logs folder in OpManager. One under the OpManager directory and the other under OpManager tomcat folder.
The logs folder under the OpManager directory contains various logs file mentioned below:
Every log file has a certain size limit and upon reaching the specified size, a new file is created and is appended with a file number at the end. For example, stderr1.txt, stderr2.txt etc.
Each group of these logs, records traces for specific actions or features in OpManager. As a standard, every log file has the Date & Time entry before the trace. We will look into each group of these logs now.
As the name suggests, these log files record actions/events related to OpManager alarms. Every time an alarm in triggered in OpManager an entry is made in the alert_audit files.
As you can see, the Date & Time is recorded along with the Severity Change (i.e Attention from Clear, Trouble from Attention, etc), the Device Name and the Alarm message.
These logs contain CLI related traces.
03. ConfChange_AdventNet_ManageEngine_OpManager.log & 04. ConfChangeErr_AdventNet_ManageEngine_OpManager.log
These logs contain entries for actions occuring during an upgrade. So, if there are any upgrade related issues, these files can be used to debug the problem.
Very evident from the file name, these logs contain records for Discovery related actions in OpManager. When you are faced with any device/interface addition issues, you can analyze these logs.
06. disterr.txt & 07. distout.txt
These files contain Central-Probe communication related log traces.
Again, like the file name suggests, these files contain log entries for HyperV related actions.
09. mserr.txt & 10. msout.txt
11. nmserr.txt & 12. nmsout.txt
OpManager is built on WebNMS framework. These files maintain framework related logs & errors.
These logs contain records for every notification (email, SMS etc) that is being triggered from OpManager.
15. opmanager_serverErr.txt & opmanager_serverOut.txt
These contain logs related to OpManager DB's table creation, backend queries for Reports generated in OpManager. Also, any traces for debugs enabled for troubleshooting purposes are also recorded here.
These contain IT Automation/Workflow related logs.
A standard log file referred the most in OpManager. This contains traces for most of the actions done in OpManager, though there are feature specific logs (like alert_audit, notificationLogs, hyperverr etc). When ever a feature does not work as expected, you are sure to find error traces in these logs that will help you debug the issue.
These logs are mostly used to debug issues related to Reports. The SQL query is printed in these files for every report generated in OpManager.
Records Transaction log activities if the OpManager database is on MSSQL backend.
These logs contain entries for actions occuring during an upgrade like the ConfChange_AdventNet_ManageEngine_OpManager.log & ConfChangeErr_AdventNet_ManageEngine_OpManager.log. Combining these 3 logs, any upgrade related issue can be debug easily.
This log is useful to debug OpManager start-up issues.
The logs folder under the OpManager\tomcat directory contains the below mentioned files:
Unlike the previous set of the logs where the file number is appended at the end, here it is appended with the Date. For example, catalina.2012-07-09.log & localhost.2012-07-09.log
The Catalina & localhost files record logs when the OpManager UI redirects itself to an Error page or when any Action buttons (like Add Monitors, Test Credential etc) in the UI remains inactive.
Now, having got a brief idea on the files and their logging properties, lets look at some exceptions/error traces which can be found in the logs and the steps to troubleshoot/fix them.
11:38:54:578 AM java.sql.SQLException: Table '.opmanagerdbpolleddata' is marked as crashed and should be repaired
11:38:54:578 AM at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
11:38:54:578 AM at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985)
11:38:54:578 AM at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)
11:38:54:578 AM at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)
11:38:54:578 AM at com.mysql.jdbc.Connection.execSQL(Connection.java:3277)
In the above trace, you can clearly see the message that PolledData of the OpManagerDB has crashed. This issue can be resolved by doing a Database repair. To repair the database, execute RepairDB.batfound under OpManagerin folder. If this does not resolve the issue, follow the steps from the link : http://support.opmanager.com/sd/AddSolution.sd?solID=29701
06:29:18:796 AM com.adventnet.nms.store.NmsStorageException: The transaction log for database 'OpManager' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases
06:29:18:796 AM at com.adventnet.management.transaction.ConnectionPool.executeUpdate(ConnectionPool.java:1798)
Here, the Transaction logs for the OpManager database is full. This issue will cause all the dials in OpManager to be greyed out and it will look like the data-collection has stopped.
You can follow the steps in the link to resolve this case: http://support.opmanager.com/sd/AddSolution.sd?solID=24901