Oracle Cloud monitors show "Maximum Rate Exceeded" error

Oracle Cloud monitors show "Maximum Rate Exceeded" error

The Maximum Rate Exceeded error occurs in Oracle Cloud and its child monitors when too many APIs are accessed at the same time. This error happens usually in an environment where there are large number of Oracle Cloud child monitors like Instances and Storage. 

Reason: In Oracle Cloud, the API rate is very low by default. The default API rate limit is 10 Transactions per Second. When polling happens for more than 10 OCI monitors at the same time, this limit is exceeded and hence you might encounter the Maximum rate exceeded error.

Resolution: Follow the steps given below to avoid this error.
  1. Increase Rate Limit:
    1. To increase the rate limit ie. TPS (Transactions per Second), raise a request in the OCI portal.
    2. In the Support section, you will find an option to raise a request to increase the rate limit.
    3. It is recommended that you raise a request to increase the rate of the reported API to 100 TPS
    4. Sample format to share with Oracle Cloud support:
      1. Tenancy: <Tenancy Name> & <Tenancy OCID>
      2. Region: <Region of the service>
      3. Compartment OCID(s): <OCIDs of compartment>
      4. Limit minimum required: 100 TPS (Can request more TPS based on the monitoring scope)
      5. If the error message is 'Unable to collect some of the performance metrics' (This is the API that is used the most as it fetches all the performance metrics in all the OCI monitor types.)
        1. API Service name: Monitoring (Telemetry)
        2. Endpoint: https://telemetry.<REGION>.oraclecloud.com
          1. Eg: https://telemetry.ap-sydney-1.oraclecloud.com
        3. Operation: SummarizeMetricsData — POST /20180401/metrics/actions/summarizeMetricsData
      6. For other error message:
        1. API Service name: <Service Name> (Eg: Oracle Cloud Instance, Oracle AI Autonomous Database, etc)
        2. Endpoint & Operation: Ask Oracle Cloud support to check your account API logs.

  2. Disable fast data collection: 
    1. Go to Admin/Settings -> Performance Polling -> Optimize Data Collection -> Monitor Type -> Oracle Cloud -> Metric Name -> Performance.
    2. If the 'Enable faster data collection' checkbox is enabled, then disable the same.
    3. If this option is enabled, then multiple API calls happen parallelly for a single monitor and when multiple monitors are polled at the same time, the rate might exceed and cause error.

  3. Maintain optimum thread count:
    1. In Applications Manager if the value of the KeyValueMonitor Thread count is high, then there are more chances of multiple monitors being polled at the same time which leads to high number of requests at the same time.
    2. To check this, go to Admin/Settings -> Server Settings -> Threads Configuration page -> KeyValueMonitor Thread.
      1. Reduce the thread count as per the monitors in the probe.
    3. If your environment has large number of OCI monitors, it is recommended to move the OCI monitors to a separate Managed server and maintain optimum (less) thread count than the current value.