AWS EKS Troubleshooting FAQs

AWS EKS Troubleshooting FAQs

AWS EKS monitor can be configured only if you have already added an Amazon monitor in Applications Manager.
Ensure that you have met the pre-requisites for the Amazon monitor before monitoring Elastic Kubernetes Service in Applications Manager.

How to verify if all the pre-requisites have been met?
  1. To check if aws-cli is installed, execute the command: aws --version
    Sample Output (For windows 10): aws-cli/2.1.29 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
  2. To check if kubectl is installed, execute the command: kubectl version --short --client
    Sample Output: Client Version: v1.19.15-eks-ad4801
  3. Login to the AWS console and check if the IAM user whose credential used in Applications Manager has access to all the APIs listed in the pre-requisites.

When the data will be populated for an EKS monitor?

  1. We initiate data collection only when the EKS cluster is in ACTIVE state.

    1. Possible states : CREATING, ACTIVE, DELETING, FAILED, UPDATING

If the cluster is in CREATING/UPDATING state, we will only update the "Cluster Information" group in "Overview" tab.


If data isn't being populated (No data/No UI message) in your AWS EKS Monitor, It could be due to any of the following reasons:
  1. You haven't met the pre-requisites required to monitor AWS EKS in Applications Manager.
  2. Check if the APM installed user has root permission to access installed kubectl/aws-cli by verifying pre-requisites mentioned above.
  3. The EKS Cluster is not in the ACTIVE state.
Applications Manager initiates data collection only when the EKS cluster is in the ACTIVE state. If the cluster is in the CREATING/UPDATING state, data will be updated in the Cluster Information group in the Overview tab alone.

If Container Insights data is not being populated:
  1. Verify if you have met all the pre-requisites by verifying using above method.
  2. Ensure whether amazon-cloudwatch pods are in the running state. If they are and if there's still no data in cloudwatch of AWS console, ensure that the CloudWatchAgentServerPolicy has been attached to the node (EC2 instance) in which the amazon-cloudwatch pods are running. For more info, click here.
Make sure container insights data is populated in cloudwatch console.


After clicking a resource, choose 1h and check if data is being collected for the cluster as shown in the below screenshot.


If you want to disable Proxy:
  1. In Applications manager, navigate to Admin > Product settings > Connectivity > Proxy server.
  2. First verify if Proxy server has been configured.
  3. If yes, to exclude kubectl from using proxy, navigate to EKS monitor page > Overview tab > Cluster information and get Cluster Endpoint information. Give the endpoint information in the Do not use proxy for addresses beginning with field without protocol(http/https) 
 
Proxy is only supported from kubectl version 1.19 and above.

If you want to disable automatic alerts for a specific node condition

Disable the specific condition by navigating to Admin > Performance Polling > Optimize Data Collection > Elastic Kubernetes Service.



If a node is not in the ready state, its health will be automatically affected. If node is in ready state, then we will check the above node conditions and affect the node health based on the configuration.


Possible Errors in EKS Monitor and their troubleshooting

Unable to get data from 'eks:DescribeCluster' API (OR)
Access is denied for 'eks:DescribeCluster' API (OR)
Unable to collect data for <SERVICE_NAME>. Access denied for <API_NAME>.
  1. Ensure that you have met the pre-requisites to discover and monitor Elastic Kubernetes Service clusters in Applications Manager.
  2. Login to the AWS console and verify if the IAM user whose credentials used in Applications Manager has permission to the APIs in the pre-requisites.
  3. Check the user name in the EKS config map.
Data collection has stopped as kubectl is not installed. (OR)
Data collection has stopped as aws-cli is not installed.
  1. Ensure that you have followed and done all the pre-requisites.
  2. Verify if you have done the pre-requisites by performing steps mentioned above.
  3. Restart Applications Manager once and check if the environment changes are reflected in Applications Manager.
Data collection has stopped. Unable to connect to EKS cluster endpoint via kubectl.

    1. If you are using a private EKS endpoint and you encounter this error, refer here to run kubectl commands using private endpoint access.

      1. If you want to add access to kubernetes API server for a specific IP address alone, refer here.

    2. Ensure that the DNS hostnames are enabled in VPC of the EKS Cluster. To enable DNS hostnames and DNS resolution, refer here.

    3. Ensure if the security group's inbound policy is properly configured.

Data collection has stopped. The user is not authorized to fetch data via kubectl.

    1. Ensure that the user whose credentials were used to add the EKS monitor has root permissions.

    2. To resolve unauthorized error, refer here.

    3. To add an IAM user to enable access to EKS cluster, refer here.

Timeout exceeded when executing kubectl command.

    1. Increase the timeout value. (The default and minimum value for timeout is 30 seconds). To increase the value, go to EKS monitor page, monitor actions -> Edit monitor -> give the appropriate value and update.

    2. Ensure your system has a stable network connectivity.

    3. Run the following kubectl commands to get to know the approximate amount of time it takes to receive a response as these commands take.

      1. kubectl get nodes

      2. kubectl get pods --all-namespaces


Possible Errors in EKS Monitor and their troubleshooting


List of kubectl commands used in APM

  1. All the commands will be appended with  --kubeconfig <FILEPATH>.
  2. kubectl config set clusters.%s.proxy-url %s
  3. kubectl version -o json
  4. kubectl get namespaces -o json
  5. kubectl get componentstatuses -o json --all-namespaces
  6. kubectl get nodes -o json
  7. kubectl get pods -o json --all-namespaces
  8. kubectl get services -o json --all-namespaces
  9. kubectl get deployments -o json --all-namespaces
  10. kubectl get persistentvolumes -o json --all-namespaces
  11. kubectl get persistentvolumeclaims -o json --all-namespaces
  12. kubectl describe nodes

List of aws commands used

  1. aws configure set aws_access_key_id {access_key}
  2. aws configure set aws_secret_access_key {secret_access_key}
  3. aws configure set default.region {region}
  4. aws eks --region {region} update-kubeconfig --name {cluster_name} --kubeconfig <FILEPATH>







      New to ADManager Plus?

        New to ADSelfService Plus?

        Start your free trial

          • Related Articles

          • AWS Monitor Addition - FAQs

            Best Practices while adding an Amazon monitor in Applications Manager:  Provide proper 'Display name' for the Amazon monitor. Select the Amazon services you want to discover from the Amazon services drop box. By default, all the services will be ...

          • Real User Monitor (RUM) - Troubleshooting

            If the monitor has not polled data for long time. Follow below steps for troubleshooting Check prerequisites to done : Real User Monitor requires RUM Agent to be installed and mapped to Applications Manager. Refer the help page to know how to setup ...

          • APM Insight Troubleshooting - FAQs

            Monitor Addition - FAQs 1. How to add an APM Insight Monitor? After you deploy APM Insight agent in your Application Server with suitable Applications Manager credentials in apminsight.conf, APM Insight monitors will be added automatically when ...

          • Azure Kubernetes Service(AKS) Troubleshooting FAQs

            Azure Kubernetes Service(AKS) monitor can be configured only if you have already added a Microsoft Azure monitor in Applications Manager. Ensure that you have met all the prerequisites for the Microsoft Azure monitor before monitoring Azure ...

          • Script Monitor FAQs

            Common queries, errors and troubleshooting: 1. To execute scripts other than .bat /.vbs  in Windows and .sh / .bash in Linux: It is better to execute the script in the form of a command. Eg. To execute python script, you can use command py ...

            Related Products