AWS EKS Troubleshooting FAQs

AWS EKS Troubleshooting FAQs

AWS EKS monitor can be configured only if you have already added an Amazon monitor in Applications Manager.
Ensure that you have met the pre-requisites for the Amazon monitor before monitoring Elastic Kubernetes Service in Applications Manager.

How to verify if all the pre-requisites have been met?

  1. To check if aws-cli is installed, execute the command: aws --version
    Sample Output (For windows 10): aws-cli/2.1.29 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
  2. To check if kubectl is installed, execute the command: kubectl version --short --client
    Sample Output: Client Version: v1.19.15-eks-ad4801
  3. Login to the AWS console and check if the IAM user whose credential used in Applications Manager has access to all the APIs listed in the pre-requisites.
  4. Verify if you are able to connect to the EKS cluster with the AWS credentials used in Applications Manager.

When the data will be populated for an EKS monitor?

  1. We initiate data collection only when the EKS cluster is in ACTIVE state.

    1. Possible states : CREATING, ACTIVE, DELETING, FAILED, UPDATING

If the cluster is in CREATING/UPDATING state, we will only update the "Cluster Information" group in "Overview" tab.

Why isn't there any data being displayed in my AWS EKS monitor?

If data is not being shown (No data/No UI message) in your AWS EKS Monitor, it could be due to any of the following reasons:
  1. Incomplete prerequisites: Make sure that the pre-requisites required to monitor AWS EKS in Applications Manager is met.
  2. Insufficient permission: Check whether the Applications Manager-installed user has root permission to access installed kubectl/aws-cli by verifying the pre-requisites mentioned above.
  3. Inactive Cluster Status: Ensure that the EKS Cluster is in the ACTIVE state.
  4. Improper Cluster Connectivity: Verify whether you are able to connect to the EKS cluster with the AWS credentials used in Applications Manager.
If the issue still persists, please reach out via mail to appmanager-support@manageengine.com with the latest Support Information File (SIF) from Applications Manager, ensuring that 'print all logs' is enabled. Additionally, include screenshots of the steps outlined above for further analysis.
Note: Applications Manager initiates data collection only when the EKS cluster is in the ACTIVE state. If the cluster is in the CREATING/UPDATING state, data will be updated in the Cluster Information group in the Overview tab alone.

How to connect to EKS Cluster?

  1. Keep the AWS credentials such as Access Key and Secret Access Key used in Applications Manager Amazon monitor ready.
  2. Login using the AWS credentials by executing the below command and entering the relevant details. Make sure to provide the proper region name of the EKS cluster: 
    aws configure
  3. Create the KUBECONFIG file for the required cluster using the below command:  
    aws eks --region {REGION} update-kubeconfig --name {CLUSTER_NAME} --kubeconfig {FILEPATH}

    In the above command,
      1. replace {REGION} with the region of the EKS cluster
      2. replace {CLUSTER_NAME} with the EKS cluster name
      3. replace {FILEPATH} with the path where the KUBECONFIG file should be created. 

        Example
        aws eks --region us-east-1 update-kubeconfig --name EKS-CLUSTER --kubeconfig /home/user/kubeconfig
    1. Note: The current user (running the commands) should have the permission to create and access the KUBECONFIG file.
  4. Execute the below commands to verify if the user is able to access the EKS cluster, replace <FILEPATH> with the path where the KUBECONFIG file should be created:
    1. kubectl get nodes --kubeconfig <FILEPATH>
    2. kubectl get pods -A --kubeconfig <FILEPATH>
  5. If any error occurs, refer the Possible Errors in EKS Monitor and their troubleshooting section and implement necessary steps.
If the issue still persists, please reach out via mail to appmanager-support@manageengine.com with the latest Support Information File (SIF) from Applications Manager, ensuring that 'print all logs' is enabled. Additionally, include screenshots of the steps outlined above for further analysis.

If Container Insights data is not being populated:

  1. Verify if you have met all the pre-requisites by verifying using above method.
  2. Ensure whether amazon-cloudwatch pods are in the running state. If they are and if there's still no data in cloudwatch of AWS console, ensure that the CloudWatchAgentServerPolicy has been attached to the node (EC2 instance) in which the amazon-cloudwatch pods are running. For more info, click here.
Make sure container insights data is populated in cloudwatch console.


After clicking a resource, choose 1h and check if data is being collected for the cluster as shown in the below screenshot.


If you want to disable Proxy:

  1. In Applications manager, navigate to Admin > Product settings > Connectivity > Proxy server.
  2. First verify if Proxy server has been configured.
  3. If yes, to exclude kubectl from using proxy, navigate to EKS monitor page > Overview tab > Cluster information and get Cluster Endpoint information. Give the endpoint information in the Do not use proxy for addresses beginning with field without protocol(http/https) 
 
Proxy is only supported from kubectl version 1.19 and above.

If you want to disable automatic alerts for a specific node condition


Disable the specific condition by navigating to Admin > Performance Polling > Optimize Data Collection > Elastic Kubernetes Service.



If a node is not in the ready state, its health will be automatically affected. If node is in ready state, then we will check the above node conditions and affect the node health based on the configuration.

Possible Errors in EKS Monitor and their troubleshooting

Unable to get data from 'eks:DescribeCluster' API (OR)
Access is denied for 'eks:DescribeCluster' API (OR)
Unable to collect data for <SERVICE_NAME>. Access denied for <API_NAME>.
  1. Ensure that you have met the pre-requisites to discover and monitor Elastic Kubernetes Service clusters in Applications Manager.
  2. Login to the AWS console and verify if the IAM user whose credentials used in Applications Manager has permission to the APIs in the pre-requisites.
  3. Check the user name in the EKS config map.
Data collection has stopped as kubectl is not installed. (OR)
Data collection has stopped as aws-cli is not installed.
  1. Ensure that you have followed and done all the pre-requisites.
  2. Verify if you have done the pre-requisites by performing steps mentioned above.
  3. Restart Applications Manager once and check if the environment changes are reflected in Applications Manager.
Data collection has stopped. Unable to connect to EKS cluster endpoint via kubectl.
    1. If you are using a private EKS endpoint and you encounter this error, refer here to run kubectl commands using private endpoint access.

      1. If you want to add access to kubernetes API server for a specific IP address alone, refer here.

    2. Ensure that the DNS hostnames are enabled in VPC of the EKS Cluster. To enable DNS hostnames and DNS resolution, refer here.

    3. Ensure if the security group's inbound policy is properly configured.

Data collection has stopped. The user is not authorized to fetch data via kubectl. (OR) Any Unauthorized errors.
    1. Ensure that the user whose credentials were used to add the EKS monitor has root permissions.

    2. To resolve unauthorized error, refer here.

    3. To add an IAM user to enable access to EKS cluster, refer here.

Timeout exceeded when executing kubectl command.

    1. Increase the timeout value. (The default and minimum value for timeout is 30 seconds). To increase the value, go to EKS monitor page, monitor actions -> Edit monitor -> give the appropriate value and update.

    2. Ensure your system has a stable network connectivity.

    3. Run the following kubectl commands to get to know the approximate amount of time it takes to receive a response as these commands take.

      1. kubectl get nodes

      2. kubectl get pods --all-namespaces


List of kubectl commands used in APM

  1. All the commands will be appended with  --kubeconfig <FILEPATH>.
  2. kubectl config set clusters.%s.proxy-url %s
  3. kubectl version -o json
  4. kubectl get namespaces -o json
  5. kubectl get componentstatuses -o json --all-namespaces
  6. kubectl get nodes -o json
  7. kubectl get pods -o json --all-namespaces
  8. kubectl get services -o json --all-namespaces
  9. kubectl get deployments -o json --all-namespaces
  10. kubectl get persistentvolumes -o json --all-namespaces
  11. kubectl get persistentvolumeclaims -o json --all-namespaces
  12. kubectl describe nodes

List of aws commands used

  1. aws configure set aws_access_key_id {access_key}
  2. aws configure set aws_secret_access_key {secret_access_key}
  3. aws configure set default.region {region}
  4. aws eks --region {region} update-kubeconfig --name {cluster_name} --kubeconfig <FILEPATH>


If any issue persists, please reach out to appmanager-support@manageengine.com. Provide the latest Support Information File (SIF) from Applications Manager, ensuring that 'print all logs' is enabled. Additionally, include screenshots of the steps outlined above for further analysis.


                    New to ADSelfService Plus?

                      • Related Articles

                      • AWS Monitor Addition - FAQs

                        Best Practices while adding an Amazon monitor in Applications Manager: Provide proper 'Display name' for the Amazon monitor. Select the Amazon services you want to discover from the Amazon services drop box. By default, all the services will be ...
                      • Real User Monitor (RUM) - Troubleshooting

                        If the monitor has not polled data for a long time, follow the steps below for troubleshooting: Step 1: Check prerequisites to be done Real User Monitor requires the RUM Agent to be installed and mapped to the Applications Manager. Refer the help ...
                      • APM Insight Troubleshooting - FAQs

                        Monitor Addition - FAQs 1. How to add an APM Insight Monitor? After you deploy the APM Insight agent in your Application Server with suitable Applications Manager credentials in the apminsight.conf file, APM Insight monitors will automatically be ...
                      • Azure Kubernetes Service(AKS) Troubleshooting FAQs

                        Azure Kubernetes Service(AKS) monitor can be configured only if you have already added a Microsoft Azure monitor in Applications Manager. Ensure that you have met all the prerequisites for the Microsoft Azure monitor before monitoring Azure ...
                      • Script Monitor FAQs

                        Common queries, errors and troubleshooting: 1. To execute scripts other than .bat /.vbs  in Windows and .sh / .bash in Linux: It is better to execute the script in the form of a command. Eg. To execute python script, you can use command py ...