AWS EKS Troubleshooting FAQs

AWS EKS Troubleshooting FAQs

AWS EKS monitor can be configured only if you have already added an Amazon monitor in Applications Manager.
Ensure that you have met the pre-requisites for the Amazon monitor before monitoring Elastic Kubernetes Service in Applications Manager.

How to verify if all the pre-requisites have been met?

  1. Ensure that pre-requisites required to monitor EKS cluster in Applications Manager is met.
  2. Verify if you are able to connect to the EKS cluster with the AWS credentials of the IAM user used in Applications Manager.
    1. For troubleshooting errors while connecting to the EKS cluster, refer the troubleshooting errors section.
    2. For more troubleshooting, refer here.
If the issue still persists, please reach out via mail to appmanager-support@manageengine.com with the below information:
  1. Latest Support Information File (SIF), ensure that 'print all logs' is enabled.
  2. Screenshot of the below mentioned information from prerequisites:
    1. Command: aws --version (To connect to AWS account) --> Sample Output (For windows 10): aws-cli/2.1.29 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
    2. Command: kubectl version --client (To connect to EKS cluster) --> Sample Output: Client Version: v1.19.15-eks-ad4801
    3. IAM username of the user whose credentials used in Applications Manager for EKS cluster monitoring.
    4. EKS config map where the IAM user is added to provide access to the cluster.
    5. Connect to the EKS cluster and run the below commands:
      1. kubectl get nodes
      2. kubectl get pods -A

When the data will be populated for an EKS monitor?

  1. We initiate data collection only when the EKS cluster is in ACTIVE state.

    1. Possible states : CREATING, ACTIVE, DELETING, FAILED, UPDATING

If the cluster is in CREATING/UPDATING state, we will only update the "Cluster Information" group in "Overview" tab.

How to connect to EKS Cluster?

  1. Keep the AWS credentials of IAM user such as Access Key and Secret Access Key used in Applications Manager Amazon monitor ready.
  2. Login using the AWS credentials by executing the below command and entering the relevant details. Make sure to provide the proper region name of the EKS cluster: 
    aws configure
  3. Create the KUBECONFIG file for the required cluster using the below command:  
    aws eks --region {REGION} update-kubeconfig --name {CLUSTER_NAME} --kubeconfig {FILEPATH}

    In the above command,
      1. replace {REGION} with the region of the EKS cluster
      2. replace {CLUSTER_NAME} with the EKS cluster name
      3. replace {FILEPATH} with the path where the KUBECONFIG file should be created. 

        Example        : aws eks --region us-east-1 update-kubeconfig --name EKS-CLUSTER --kubeconfig /home/user/kubeconfig
    1. Note: The current user (running the commands) should have the permission to create and access the KUBECONFIG file.
  4. Execute the below commands to verify if the user is able to access the EKS cluster, replace <FILEPATH> with the path where the KUBECONFIG file should be created:
    1. kubectl get nodes --kubeconfig <FILEPATH>
    2. kubectl get pods -A --kubeconfig <FILEPATH>
  5. If any error occurs, refer the Possible Errors in EKS Monitor and their troubleshooting section and implement necessary steps.

Possible Errors in EKS Monitor and their troubleshooting

Unable to get data from 'eks:DescribeCluster' API (OR)
Access is denied for 'eks:DescribeCluster' API (OR)
Unable to collect data for <SERVICE_NAME>. Access denied for <API_NAME>.
  1. Ensure that you have met the pre-requisites to discover and monitor Elastic Kubernetes Service clusters in Applications Manager.
  2. Login to the AWS console and verify if the IAM user whose credentials used in Applications Manager has permission to the APIs in the pre-requisites.
  3. Check the user name in the EKS config map.
Data collection has stopped as kubectl is not installed. (OR)
Data collection has stopped as aws-cli is not installed.
  1. Ensure that you have followed and done all the pre-requisites.
    1. Steps 2 and 3 are mandatory.
  2. Restart Applications Manager once and check if the environment changes are reflected in Applications Manager.
Data collection has stopped. The user is not authorized to fetch data via kubectl. (OR) Any Unauthorized errors.

  1. Ensure that the user whose credentials were used to add the EKS monitor has root permissions.

  2. To resolve unauthorized error and to add an IAM user to enable access to EKS cluster

    1. Refer here to allow the user to access the cluster.

      1. If the IAM user is the EKS cluster admin, follow the section You're the cluster creator.

      2. If the IAM user is not the EKS cluster admin, follow the section You're not the cluster creator.

Data collection has stopped. Unable to connect to EKS cluster endpoint via kubectl.

  1. If you are using a private EKS endpoint and you encounter this error, refer here to run kubectl commands using private endpoint access.

    1. If you want to add access to kubernetes API server for a specific IP address alone, refer here.

  2. Ensure that the DNS hostnames are enabled in VPC of the EKS Cluster. To enable DNS hostnames and DNS resolution, refer here.

  3. Ensure if the security group's inbound policy is properly configured.

Timeout exceeded when executing kubectl command.

    1. Increase the timeout value. (The default and minimum value for timeout is 30 seconds). To increase the value, go to EKS monitor page, monitor actions -> Edit monitor -> give the appropriate value and update.

    2. Ensure your system has a stable network connectivity.

    3. Run the following kubectl commands to get to know the approximate amount of time it takes to receive a response as these commands take.

      1. kubectl get nodes

      2. kubectl get pods --all-namespaces


If Container Insights data is not being populated:

  1. Verify if you have met all the pre-requisites by verifying using above method.
  2. Ensure whether amazon-cloudwatch pods are in the running state. If they are and if there's still no data in cloudwatch of AWS console, ensure that the CloudWatchAgentServerPolicy has been attached to the node (EC2 instance) in which the amazon-cloudwatch pods are running. For more info, click here.
Make sure container insights data is populated in cloudwatch console.


After clicking a resource, choose 1h and check if data is being collected for the cluster as shown in the below screenshot.


Frequently Asked Questions

How to disable Proxy:

  1. In Applications manager, navigate to Admin > Product settings > Connectivity > Proxy server.
  2. First verify if Proxy server has been configured.
  3. If yes, to exclude kubectl from using proxy, navigate to EKS monitor page > Overview tab > Cluster information and get Cluster Endpoint information. Give the endpoint information in the Do not use proxy for addresses beginning with field without protocol(http/https) 
 
Proxy is only supported from kubectl version 1.19 and above.

How to disable automatic alerts for a specific node condition

Disable the specific condition by navigating to Admin > Performance Polling > Optimize Data Collection > Elastic Kubernetes Service.



If a node is not in the ready state, its health will be automatically affected. If node is in ready state, then we will check the above node conditions and affect the node health based on the configuration.

List of kubectl commands used in APM in EKS monitor

  1. All the commands will be appended with  --kubeconfig <FILEPATH>.
  2. If proxy is configured, then run the below command by replacing the clustername and proxyurl 
    kubectl config set clusters.clustername.proxy-url proxyurl
  3. Below are the list of commands used:
    1. kubectl config set clusters.%s.proxy-url %s
    2. kubectl version -o json
    3. kubectl get namespaces -o json
    4. kubectl get componentstatuses -o json --all-namespaces
    5. kubectl get nodes -o json
    6. kubectl get pods -o json --all-namespaces
    7. kubectl get services -o json --all-namespaces
    8. kubectl get deployments -o json --all-namespaces
    9. kubectl get persistentvolumes -o json --all-namespaces
    10. kubectl get persistentvolumeclaims -o json --all-namespaces
    11. kubectl describe nodes

List of aws commands used in APM in EKS monitor

  1. aws configure set aws_access_key_id {access_key}
  2. aws configure set aws_secret_access_key {secret_access_key}
  3. aws configure set default.region {region}
  4. aws eks --region {region} update-kubeconfig --name {cluster_name} --kubeconfig <FILEPATH>


If any issue persists, please reach out via mail to appmanager-support@manageengine.com with the below information:
  1. Latest Support Information File (SIF), ensuring that 'print all logs' is enabled.
  2. Screenshot of the below mentioned information from prerequisites:
    1. Command: aws --version (To connect to AWS account) --> Sample Output (For windows 10): aws-cli/2.1.29 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
    2. Command: kubectl version --client (To connect to EKS cluster) --> Sample Output: Client Version: v1.19.15-eks-ad4801
    3. IAM username of the user whose credentials used in Applications Manager for EKS cluster monitoring.
    4. EKS config map where the IAM user is added to provide access to the cluster.
    5. Connect to the EKS cluster and run the below commands:
      1. kubectl get nodes
      2. kubectl get pods -A


      New to M365 Manager Plus?
      New to M365 Manager Plus?
        New to RecoveryManager Plus?
        New to RecoveryManager Plus?
          New to Exchange Reporter Plus?
          New to Exchange Reporter Plus?
            New to SharePoint Manager Plus?
            New to SharePoint Manager Plus?
                See all integrations
                New to ADManager Plus?
                See all integrations
                New to ADManager Plus?

                  New to ADSelfService Plus?

                  Start your free trial
                  Start your free trial

                    • Related Articles

                    • AWS Monitor Addition - FAQs

                      Best Practices while adding an Amazon monitor in Applications Manager: Provide proper 'Display name' for the Amazon monitor. Select the Amazon services you want to discover from the Amazon services drop box. By default, all the services will be ...

                    • Real User Monitor (RUM) - Troubleshooting

                      If the monitor has not polled data for a long time, follow the steps below for troubleshooting: Step 1: Check the RUM Agent configuration Real User Monitor requires the RUM Agent to be installed and mapped to the Applications Manager. Refer the help ...

                    • Azure Kubernetes Service(AKS) Troubleshooting FAQs

                      Azure Kubernetes Service (AKS) monitor can be configured only if you have already added a Microsoft Azure monitor in Applications Manager. Ensure that you have met all the prerequisites for the Microsoft Azure monitor before monitoring Azure ...

                    • APM Insight Troubleshooting - FAQs

                      Monitor Addition - FAQs 1. How to add an APM Insight Monitor? After you deploy the APM Insight agent in your Application Server with suitable Applications Manager credentials in the apminsight.conf file, APM Insight monitors will automatically be ...

                    • Troubleshooting Failed Diagnostic Tests

                      This KB is intended for troubleshooting Diagnostic Tests with Failed results. If you encounter any errors displayed in the monitor that prevent the execution of Diagnostic tests, or if there is no data available for Diagnostic Tests, please refer ...

                      Related Products