A few days ago we had an outage on our OpManager software which meant that we stopped being alerted by the system for almost 24 hours before it was noticed that the services had crashed out (once restarted we then had to run repairdb.bat), this has meant we have lost availability data, but more importantly hadn't been alerted to faults on our infrastructure for this time.

In response to this we'd like to monitor the opmanager services and growth of log files or DB to ensure in future we are made aware of this through another monitoring system - do you have any recommendations for putting together a tool like this, ie. which files should we be monitoring and how often should they be updating?

Finally, are there any steps we can take to avoid future occurences of this problem?