Opmanager Won't Start (Java EXCEPTION_ACCESS_VIOLATION error?)

Opmanager Won't Start (Java EXCEPTION_ACCESS_VIOLATION error?)

After a server reboot OpManager will no longer load up.  The service is running on the server (Win2003) but nothing will come up via a web browser.  A look at the wrapper.log indicate this may be a Java problem:

  2. INFO   | jvm 1    | 2014/02/12 11:56:26 | # An unexpected error has been detected by Java Runtime Environment:
  3. INFO   | jvm 1    | 2014/02/12 11:56:26 | #
  4. INFO   | jvm 1    | 2014/02/12 11:56:26 | #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x4d0910be, pid=6528, tid=7780
  5. INFO   | jvm 1    | 2014/02/12 11:56:26 | #
  6. INFO   | jvm 1    | 2014/02/12 11:56:26 | # Java VM: Java HotSpot(TM) Client VM (11.0-b16 mixed mode windows-x86)
  7. INFO   | jvm 1    | 2014/02/12 11:56:26 | # Problematic frame:
  8. INFO   | jvm 1    | 2014/02/12 11:56:26 | # C  0x4d0910be
  9. INFO   | jvm 1    | 2014/02/12 11:56:26 | #
  10. INFO   | jvm 1    | 2014/02/12 11:56:26 | # An error report file with more information is saved as:
  11. INFO   | jvm 1    | 2014/02/12 11:56:26 | # C:\MANAGE~1\OPMANA~1\hs_err_pid6528.log
The log file mentioned above shows the following information:

  2. #
  3. # An unexpected error has been detected by Java Runtime Environment:
  4. #
  5. #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x4d0910be, pid=6528, tid=7780
  6. #
  7. # Java VM: Java HotSpot(TM) Client VM (11.0-b16 mixed mode windows-x86)
  8. # Problematic frame:
  9. # C  0x4d0910be
  10. #
  11. # If you would like to submit a bug report, please visit:
  12. #   http://java.sun.com/webapps/bugreport/crash.jsp
  13. # The crash happened outside the Java Virtual Machine in native code.
  14. # See problematic frame for where to report the bug.
  15. #

  17. ---------------  T H R E A D  ---------------

  19. Current thread (0x4dca7800):  JavaThread "Secondary finalizer" daemon [_thread_in_native, id=7780, stack(0x4d190000,0x4d1e0000)]

  21. siginfo: ExceptionCode=0xc0000005, reading address 0x4d0910be

  23. Registers:
  24. EAX=0x4d0910be, EBX=0x428b8db8, ECX=0x4dca7914, EDX=0x428b95a8
  25. ESP=0x4d1df8bc, EBP=0x4d1df8ec, ESI=0x428b8db8, EDI=0x4dca7800
  26. EIP=0x4d0910be, EFLAGS=0x00010287

  28. Top of Stack: (sp=0x4d1df8bc)
  29. 0x4d1df8bc:   00619d11 4dca7914 4d1df8fc 4d1df8fc
  30. 0x4d1df8cc:   4d1df8cc 00000000 4d1df8fc 428b95a8
  31. 0x4d1df8dc:   00000000 428b8db8 00000000 4d1df8fc
  32. 0x4d1df8ec:   4d1df920 00612da1 00000000 00618189
  33. 0x4d1df8fc:   027fc218 4d1df900 428b8c61 4d1df930
  34. 0x4d1df90c:   428b95a8 00000000 428b8cb0 4d1df8fc
  35. 0x4d1df91c:   4d1df930 4d1df950 006102ca 00000000
  36. 0x4d1df92c:   00000000 027fc218 6d943934 4d1df9ac 

  38. Instructions: (pc=0x4d0910be)
  39. 0x4d0910ae:   
  40. [error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xc0000005]

  42. Stack: [0x4d190000,0x4d1e0000],  sp=0x4d1df8bc,  free space=318k
  43. Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
  44. C  0x4d0910be
  45. j  org.tanukisoftware.wrapper.WrapperResources.finalize()V+1
  46. v  ~StubRoutines::call_stub

  48. Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
  49. j  org.tanukisoftware.wrapper.WrapperResources.nativeDestroyResource()V+0
  50. j  org.tanukisoftware.wrapper.WrapperResources.finalize()V+1
  51. v  ~StubRoutines::call_stub
  52. j  java.lang.ref.Finalizer.invokeFinalizeMethod(Ljava/lang/Object;)V+0
  53. j  java.lang.ref.Finalizer.runFinalizer()V+45
  54. j  java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;)V+1
  55. j  java.lang.ref.Finalizer$3.run()V+38
  56. j  java.lang.Thread.run()V+11
  57. v  ~StubRoutines::call_stub

  59. ---------------  P R O C E S S  ---------------

  61. Java Threads: ( => current thread )
  62. =>0x4dca7800 JavaThread "Secondary finalizer" daemon [_thread_in_native, id=7780, stack(0x4d190000,0x4d1e0000)]
  63.   0x4dcab000 JavaThread "TimerQueue" daemon [_thread_blocked, id=7748, stack(0x4d4f0000,0x4d540000)]
  64.   0x4cfdd400 JavaThread "Thread-7" [_thread_in_native, id=4140, stack(0x4db40000,0x4db90000)]
  65.   0x4cfaa800 JavaThread "Thread-5" [_thread_blocked, id=5556, stack(0x4daf0000,0x4db40000)]
  66.   0x4cfa9c00 JavaThread "Timer-0" [_thread_blocked, id=6588, stack(0x4daa0000,0x4daf0000)]
  67.   0x4cf86000 JavaThread "AWT-EventQueue-0" [_thread_in_native, id=6584, stack(0x4da50000,0x4daa0000)]
  68.   0x4cf84800 JavaThread "AWT-Shutdown" [_thread_blocked, id=4976, stack(0x4da00000,0x4da50000)]
  69.   0x4ceb1c00 JavaThread "Thread-3" [_thread_blocked, id=5360, stack(0x4d910000,0x4d960000)]
  70.   0x4cea3000 JavaThread "NmsMainBE" [_thread_blocked, id=2664, stack(0x4d870000,0x4d8c0000)]
  71.   0x4cea5c00 JavaThread "ShutDown Listener Thread" [_thread_in_native, id=4460, stack(0x4d820000,0x4d870000)]
  72.   0x4ce90000 JavaThread "TrayIconThread" [_thread_blocked, id=5272, stack(0x4d740000,0x4d790000)]
  73.   0x4ce2e800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=4940, stack(0x4d6f0000,0x4d740000)]
  74.   0x002a9000 JavaThread "DestroyJavaVM" [_thread_blocked, id=6540, stack(0x005c0000,0x00610000)]
  75.   0x4ce19400 JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_blocked, id=4448, stack(0x4d0f0000,0x4d140000)]
  76.   0x4c7a6c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=216, stack(0x4c9f0000,0x4ca40000)]
  77.   0x4c7a4400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=6580, stack(0x4c9a0000,0x4c9f0000)]
  78.   0x4c79f800 JavaThread "Attach Listener" daemon [_thread_blocked, id=6500, stack(0x4c950000,0x4c9a0000)]
  79.   0x4c78cc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=6496, stack(0x4c900000,0x4c950000)]
  80.   0x4c787c00 JavaThread "Finalizer" daemon [_thread_blocked, id=6504, stack(0x4c8b0000,0x4c900000)]
  81.   0x4c798800 JavaThread "Reference Handler" daemon [_thread_blocked, id=6544, stack(0x4c860000,0x4c8b0000)]

  83. Other Threads:
  84.   0x4c780c00 VMThread [stack: 0x4c810000,0x4c860000] [id=6524]
  85.   0x4c7b1800 WatcherThread [stack: 0x4ca40000,0x4ca90000] [id=2880]

  87. VM state:not at safepoint (normal execution)

  89. VM Mutex/Monitor currently owned by a thread: None

  91. Heap
  92.  def new generation   total 36288K, used 30414K [0x02690000, 0x04df0000, 0x07550000)
  93.   eden space 32256K,  94% used [0x02690000, 0x04443870, 0x04610000)
  94.   from space 4032K,   0% used [0x04610000, 0x04610000, 0x04a00000)
  95.   to   space 4032K,   0% used [0x04a00000, 0x04a00000, 0x04df0000)
  96.  tenured generation   total 483968K, used 0K [0x07550000, 0x24df0000, 0x42690000)
  97.    the space 483968K,   0% used [0x07550000, 0x07550000, 0x07550200, 0x24df0000)
  98.  compacting perm gen  total 65536K, used 12262K [0x42690000, 0x46690000, 0x4a690000)
  99.    the space 65536K,  18% used [0x42690000, 0x43289970, 0x43289a00, 0x46690000)
  100. No shared spaces configured.

  102. Dynamic libraries:
  103. 0x00400000 - 0x00424000 C:\MANAGE~1\OPMANA~1\jre\bin\java.exe
  104. 0x7c800000 - 0x7c8c3000 C:\WINDOWS\system32\ntdll.dll
  105. 0x77e40000 - 0x77f42000 C:\WINDOWS\system32\kernel32.dll
  106. 0x7d1e0000 - 0x7d27c000 C:\WINDOWS\system32\ADVAPI32.dll
  107. 0x77c50000 - 0x77cf0000 C:\WINDOWS\system32\RPCRT4.dll
  108. 0x76f50000 - 0x76f63000 C:\WINDOWS\system32\Secur32.dll
  109. 0x7c340000 - 0x7c396000 C:\MANAGE~1\OPMANA~1\jre\bin\msvcr71.dll
  110. 0x6d800000 - 0x6da56000 C:\MANAGE~1\OPMANA~1\jre\bin\client\jvm.dll
  111. 0x77380000 - 0x77411000 C:\WINDOWS\system32\USER32.dll
  112. 0x77c00000 - 0x77c49000 C:\WINDOWS\system32\GDI32.dll
  113. 0x76aa0000 - 0x76acd000 C:\WINDOWS\system32\WINMM.dll
  114. 0x76290000 - 0x762ad000 C:\WINDOWS\system32\IMM32.DLL
  115. 0x6d280000 - 0x6d288000 C:\MANAGE~1\OPMANA~1\jre\bin\hpi.dll
  116. 0x76b70000 - 0x76b7b000 C:\WINDOWS\system32\PSAPI.DLL
  117. 0x6d7b0000 - 0x6d7bc000 C:\MANAGE~1\OPMANA~1\jre\bin\verify.dll
  118. 0x6d320000 - 0x6d33f000 C:\MANAGE~1\OPMANA~1\jre\bin\java.dll
  119. 0x6d7f0000 - 0x6d7ff000 C:\MANAGE~1\OPMANA~1\jre\bin\zip.dll
  120. 0x6d560000 - 0x6d569000 C:\ManageEngine\OpManager\jre\bin\management.dll
  121. 0x77ba0000 - 0x77bfa000 C:\WINDOWS\system32\msvcrt.dll
  122. 0x6d610000 - 0x6d623000 C:\ManageEngine\OpManager\jre\bin\net.dll
  123. 0x71c00000 - 0x71c17000 C:\WINDOWS\system32\WS2_32.dll
  124. 0x71bf0000 - 0x71bf8000 C:\WINDOWS\system32\WS2HELP.dll
  125. 0x55600000 - 0x55635000 C:\Program Files\Microsoft Firewall Client 2004\FwcWsp.dll
  126. 0x7d0e0000 - 0x7d16b000 C:\WINDOWS\system32\OLEAUT32.dll
  127. 0x77670000 - 0x777a9000 C:\WINDOWS\system32\ole32.dll
  128. 0x71b20000 - 0x71b61000 C:\WINDOWS\system32\mswsock.dll
  129. 0x5f270000 - 0x5f2ca000 C:\WINDOWS\system32\hnetcfg.dll
  130. 0x7d180000 - 0x7d1d2000 C:\WINDOWS\system32\SHLWAPI.dll
  131. 0x77420000 - 0x77523000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.4770_x-ww_05FDF087\comctl32.dll
  132. 0x71ae0000 - 0x71ae8000 C:\WINDOWS\System32\wshtcpip.dll
  133. 0x76ed0000 - 0x76efa000 C:\WINDOWS\system32\DNSAPI.dll
  134. 0x76f70000 - 0x76f77000 C:\WINDOWS\System32\winrnr.dll
  135. 0x76f10000 - 0x76f3e000 C:\WINDOWS\system32\WLDAP32.dll
  136. 0x76f80000 - 0x76f85000 C:\WINDOWS\system32\rasadhlp.dll
  137. 0x6d000000 - 0x6d138000 C:\ManageEngine\OpManager\jre\bin\awt.dll
  138. 0x73070000 - 0x73097000 C:\WINDOWS\system32\WINSPOOL.DRV
  139. 0x6d220000 - 0x6d274000 C:\ManageEngine\OpManager\jre\bin\fontmanager.dll
  140. 0x7c8d0000 - 0x7d0cf000 C:\WINDOWS\system32\shell32.dll
  141. 0x6d630000 - 0x6d639000 C:\ManageEngine\OpManager\jre\bin\nio.dll
  142. 0x71b70000 - 0x71ba6000 C:\WINDOWS\system32\UxTheme.dll
  143. 0x6d190000 - 0x6d1b3000 C:\ManageEngine\OpManager\jre\bin\dcpr.dll

  145. VM Arguments:
  146. jvm_args: -Dcatalina.home=tomcat -Ddb.home=pgsql -Djava.library.path=lib -Dwebserver.rootdir=tomcat -Djava.rmi.server.codebase=tomcat/conf/workers.properties -Dcom.adventnet.me.opmanager.showprogress=true -Dcom.adventnet.me.opmanager.service=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=tomcat/conf/logging.properties -XX:PermSize=64m -XX:MaxPermSize=128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Dfile.encoding=Cp1252 -Duser.language=en -Xms512m -Xmx1024m -Djava.library.path=lib/ -Dwrapper.key=S_VdJnX3GkhObMqEhumOULzKfm1-WYWq -Dwrapper.port=32001 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=5700 -Dwrapper.version=3.5.15-pro -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=2995 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper 
  147. java_command: org.tanukisoftware.wrapper.WrapperSimpleApp com.adventnet.me.opmanager.server.StartOpManagerJdbc TOMCAT_DIR ./tomcat/ ROOT_DIR . NATIVE_PING true NMS_BE_FAILOVER true SERVICE true
  148. Launcher Type: SUN_STANDARD

  150. Environment Variables:
  151. PATH=D:\ESET File Security\;C:\Program Files\Support Tools\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Dell\SysMgt\oma\bin;C:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\Program Files\Windows Imaging\;C:\Program Files\Spiceworks\Nmap
  152. USERNAME=******
  153. OS=Windows_NT
  154. PROCESSOR_IDENTIFIER=x86 Family 15 Model 4 Stepping 3, GenuineIntel
Has anyone come across this error before?