1

I have deployed grafana in eks using the steps provided in this link After deployment of grafana, the pod is not in running state.

 kubectl get po -n grafana
NAME                       READY   STATUS    RESTARTS   AGE
grafana-6db7758575-pfqdg   0/1     Pending   0          31m

pod logs shown nothing.

kubectl logs grafana-6db7758575-pfqdg -n grafana
Defaulted container "grafana" out of: grafana, init-chown-data (init)

events.

kubectl get ev -n grafana
LAST SEEN   TYPE      REASON                 OBJECT                          MESSAGE
61s         Warning   FailedScheduling       pod/grafana-6db7758575-pfqdg    running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
31m         Normal    SuccessfulCreate       replicaset/grafana-6db7758575   Created pod: grafana-6db7758575-pfqdg
31m         Normal    WaitForFirstConsumer   persistentvolumeclaim/grafana   waiting for first consumer to be created before binding
31m         Normal    EnsuringLoadBalancer   service/grafana                 Ensuring load balancer
31m         Normal    ScalingReplicaSet      deployment/grafana              Scaled up replica set grafana-6db7758575 to 1
62s         Normal    ExternalProvisioning   persistentvolumeclaim/grafana   waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
31m         Normal    EnsuredLoadBalancer    service/grafana                 Ensured load balancer

pod description, gave below output, in the events, it is showing about pvc.

vagrant@vagrant:~$ kubectl describe po -n grafana
Name:             grafana-6db7758575-pfqdg
Namespace:        grafana
Priority:         0
Service Account:  grafana
Node:             <none>
Labels:           app.kubernetes.io/instance=grafana
                  app.kubernetes.io/name=grafana
                  pod-template-hash=6db7758575
Annotations:      checksum/config: efd2d840644320308147c6b71db3d00a4d55a6a22ed3146c32e597c1127e9a1e
                  checksum/dashboards-json-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
                  checksum/sc-dashboard-provider-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
                  checksum/secret: 60fa63d41d5ab20111541f3a82af4fd684b98cac4f0dfef57e20698fb99b63e3
                  kubernetes.io/psp: eks.privileged
Status:           Pending
IP:
IPs:              <none>
Controlled By:    ReplicaSet/grafana-6db7758575
Init Containers:
  init-chown-data:
    Image:      busybox:1.31.1
    Port:       <none>
    Host Port:  <none>
    Command:
      chown
      -R
      472:472
      /var/lib/grafana
    Environment:  <none>
    Mounts:
      /var/lib/grafana from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcp4z (ro)
Containers:
  grafana:
    Image:      grafana/grafana:9.1.1
    Port:       3000/TCP
    Host Port:  0/TCP
    Liveness:   http-get http://:3000/api/health delay=60s timeout=30s period=10s #success=1 #failure=10
    Readiness:  http-get http://:3000/api/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      GF_SECURITY_ADMIN_USER:      <set to the key 'admin-user' in secret 'grafana'>      Optional: false
      GF_SECURITY_ADMIN_PASSWORD:  <set to the key 'admin-password' in secret 'grafana'>  Optional: false
      GF_PATHS_DATA:               /var/lib/grafana/
      GF_PATHS_LOGS:               /var/log/grafana
      GF_PATHS_PLUGINS:            /var/lib/grafana/plugins
      GF_PATHS_PROVISIONING:       /etc/grafana/provisioning
    Mounts:
      /etc/grafana/grafana.ini from config (rw,path="grafana.ini")
      /etc/grafana/provisioning/datasources/datasources.yaml from config (rw,path="datasources.yaml")
      /var/lib/grafana from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcp4z (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      grafana
    Optional:  false
  storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  grafana
    ReadOnly:   false
  kube-api-access-fcp4z:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  118s (x3 over 22m)  default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

get pvc output

 kubectl get pvc -n grafana
NAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
grafana   Pending                                      gp2            33m




kubectl describe pvc -n grafana
    Name:          grafana
    Namespace:     grafana
    StorageClass:  gp2
    Status:        Pending
    Volume:
    Labels:        app.kubernetes.io/instance=grafana
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=grafana
                   app.kubernetes.io/version=9.1.1
                   helm.sh/chart=grafana-6.36.2
    Annotations:   meta.helm.sh/release-name: grafana
                   meta.helm.sh/release-namespace: grafana
                   volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
                   volume.kubernetes.io/selected-node: ip-192-168-23-114.ec2.internal
                   volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:
    Access Modes:
    VolumeMode:    Filesystem
    Used By:       grafana-6db7758575-pfqdg
    Events:
      Type    Reason                Age                    From                         Message
      ----    ------                ----                   ----                         -------
      Normal  ExternalProvisioning  116s (x3982 over 16h)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Improve this question
3
  • 1
    What does kubectl describe pvc say?
    – Daigo
    Commented Sep 6, 2022 at 6:32
  • added output above, can you please check
    – user2331760
    Commented Sep 6, 2022 at 10:06
  • what about your prometheus PVC? that one worked? From what you shared so far, we can see the "ebs.csi.aws.com" provisioner should be creating an EBS volume, and it's somehow stuck. Check storageclasses in your cluster: is "gp2" available (didn't they move to gp3?) -- please share "kubectl get sc". If you can find a gp2 storageclass, and your prometheus PVC was provisioned: look for the ebs provisioner pod logs, there may be some error explaining what's going on (eg: ebs volumes quota reached, maybe?!)
    – SYN
    Commented Oct 2, 2022 at 7:37

2 Answers 2

Reset to default
1

I am just working on a similar issue in EKS cluster and it points to PersistentVolume not being provisioned and it times out. You need ebs csi driver as per this documentation - https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html

Improve this answer
1
  • yes, it is configured but still same issue.
    – user2331760
    Commented Sep 27, 2022 at 5:55
0

Check if the node is out of cpu/memory

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .