After a failover switch in OpManager, the APM Plugin on the Standby server throws authtoken encryption errors and plugin functionalities such as monitoring, alerts, and data collection do not work.

How to Confirm ?

When accessing the Applications tab in OpManager UI, the following error is displayed:

The following errors also appear in the APM Plugin logs on the Standby server:

java.lang.NullPointerException
 at com.adventnet.appmanager.util.UserScopeUtil.getEARKeyForDBPoll(UserScopeUtil.java:513)
 at com.adventnet.appmanager.util.UserScopeUtil.regenerateUserToken(UserScopeUtil.java:368)

java.lang.Exception: keyToken/plainText is invalid.
 at com.adventnet.appmanager.util.AMEncryption.encrypt_AES_CBC_withKey(AMEncryption.java:293)

PluginUtil.getPluginAuthTokenFromOpManagerDB() | EARKey is null |
Returning to avoid setting encrypted authtoken in cache

Cause

Both Primary and Standby APM Plugin servers share the same database, but each installation generates its own MasterEncryptionKey in <APM_Plugin_Home>\conf\DB_Crypt.properties. During failover, the Standby cannot decrypt data that was encrypted by the Primary's key, resulting in the errors above.

Resolution Steps

Ensure both Primary and Standby APM Plugin servers use the same MasterEncryptionKey so encrypted data in the shared database can be decrypted by either server.

A. Steps for Fresh Installation

For a freshly installed APM Plugin on the Standby server, the steps depend on whether the Standby server has already been started at least once.

Case 1: Standby server has NOT been started yet

If the Standby APM Plugin has never been started, DB_Crypt.properties does not exist yet. Perform these steps before starting the Standby server or triggering failover:

1. Navigate to <APM_Plugin_Home>\conf\ on the Primary server.
2. Copy the entire DB_Crypt.properties file.
3. Place it in the same location <APM_Plugin_Home>\conf\ on the Standby server.

Case 2: Standby server has already been started

If the Standby APM Plugin has been started at least once, DB_Crypt.properties already exists with its own unique AMCryptKey and MasterEncryptionKey. The AMCryptKey has already been used to encrypt instance-specific data (e.g., database passwords) on the Standby server. Only the MasterEncryptionKey needs to be synchronized:

1. Open <APM_Plugin_Home>\conf\DB_Crypt.properties on the Primary server.
2. Copy the value of MasterEncryptionKey.
3. Open the same file on the Standby server.
4. Replace the MasterEncryptionKey value with the one copied from Primary.
5. Save the file.

B. Steps for Existing Installation (Post-Upgrade)

From build 179200, the MasterEncryptionKey is used for encrypting shared database data. Since each server already has its own different key, they must be synchronized. Copy the key from the server that was started first after the upgrade, as that server's key is used to encrypt data in the shared database:

If the Primary server was started first after upgrade:

1. Open <APM_Plugin_Home>\conf\DB_Crypt.properties on the Primary server.
2. Copy the value of MasterEncryptionKey.
3. Open the same file on the Standby server.
4. Replace the MasterEncryptionKey value with the one copied from Primary.
5. Save the file.

If the Standby server was started first after upgrade:

1. Open <APM_Plugin_Home>\conf\DB_Crypt.properties on the Standby server.
2. Copy the value of MasterEncryptionKey.
3. Open the same file on the Primary server.
4. Replace the MasterEncryptionKey value with the one copied from Standby.
5. Save the file.

Verification

After applying the fix, trigger a failover and confirm:

1. The APM Plugin web console is accessible on the Standby server.
2. No NullPointerException or "EARKey is null" errors in the logs.
3. Monitors are active and collecting data.