|
| 1 | +# Resolve Pod Deployment Issue |
| 2 | + |
| 3 | +A junior DevOps team member encountered difficulties deploying a stack on the Kubernetes cluster. The pod fails to start, presenting errors. Let's troubleshoot and rectify the issue promptly. |
| 4 | + |
| 5 | +- There is a pod named `webserver`, and the container within it is named `nginx-container`, its utilizing the `nginx:latest` image. |
| 6 | +- Additionally, there's a sidecar container named `sidecar-container` using the ubuntu:latest image. |
| 7 | +- Identify and address the issue to ensure the pod is in the `running` state and the application is accessible. |
| 8 | + |
| 9 | +> Note: The kubectl utility on jump_host is configured to interact with the Kubernetes cluster. |
| 10 | +
|
| 11 | +## Steps |
| 12 | + |
| 13 | +1. Let's get the status of pods |
| 14 | + |
| 15 | + ```sh |
| 16 | + kubectl get pod webserver |
| 17 | + ``` |
| 18 | + |
| 19 | + ```shell |
| 20 | + kubectl get pod webserver |
| 21 | + NAME READY STATUS RESTARTS AGE |
| 22 | + webserver 1/2 ImagePullBackOff 0 10m |
| 23 | + ``` |
| 24 | + |
| 25 | + > we can see there is a problem with container image pulling |
| 26 | + |
| 27 | +2. Let's see in details |
| 28 | +
|
| 29 | + ```sh |
| 30 | + kubectl describe pod webserver |
| 31 | + ``` |
| 32 | +
|
| 33 | + > We can see events details at the end: |
| 34 | +
|
| 35 | + ```shell |
| 36 | + Events: |
| 37 | + Type Reason Age From Message |
| 38 | + ---- ------ ---- ---- ------- |
| 39 | + Normal Scheduled 13m default-scheduler Successfully assigned default/webserver to kodekloud-control-plane |
| 40 | + Normal Pulling 13m kubelet Pulling image "ubuntu:latest" |
| 41 | + Normal Pulled 13m kubelet Successfully pulled image "ubuntu:latest" in 3.148272011s (3.148286457s including waiting) |
| 42 | + Normal Created 13m kubelet Created container sidecar-container |
| 43 | + Normal Started 13m kubelet Started container sidecar-container |
| 44 | + Normal Pulling 12m (x3 over 13m) kubelet Pulling image "nginx:latests" |
| 45 | + Warning Failed 12m (x3 over 13m) kubelet Failed to pull image "nginx:latests": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:latests": failed to resolve reference "docker.io/library/nginx:latests": docker.io/library/nginx:latests: not found |
| 46 | + Warning Failed 12m (x3 over 13m) kubelet Error: ErrImagePull |
| 47 | + Warning Failed 12m (x6 over 13m) kubelet Error: ImagePullBackOff |
| 48 | + Normal BackOff 3m32s (x43 over 13m) kubelet Back-off pulling image "nginx:latests" |
| 49 | + ``` |
| 50 | +
|
| 51 | + > We see there are two container where nignx is getting failed to pull. It seems image tag issue. |
| 52 | +
|
| 53 | +3. Let's create a `pod.yml` file from the `webserver` pod |
| 54 | + |
| 55 | + ```sh |
| 56 | + kubectl get pod webserver -o yaml > pod.yml |
| 57 | + ``` |
| 58 | + |
| 59 | +4. Update the nginx container image tag |
| 60 | + |
| 61 | + ```sh |
| 62 | + vi pod.yml |
| 63 | + ``` |
| 64 | + |
| 65 | + > let's change nginx container image: `nginx:latests` to `nginx:latest` |
| 66 | +
|
| 67 | + ```YAML |
| 68 | + apiVersion: v1 |
| 69 | + kind: Pod |
| 70 | + metadata: |
| 71 | + annotations: |
| 72 | + kubectl.kubernetes.io/last-applied-configuration: | |
| 73 | + {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"web-app"},"name":"webserver","namespace":"default"},"spec":{"containers":[{"image":"nginx:latests","name":"nginx-container","volumeMounts":[{"mountPath":"/var/log/nginx","name":"shared-logs"}]},{"command":["sh","-c","while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log; sleep 30; done"],"image":"ubuntu:latest","name":"sidecar-container","volumeMounts":[{"mountPath":"/var/log/nginx","name":"shared-logs"}]}],"volumes":[{"emptyDir":{},"name":"shared-logs"}]}} |
| 74 | + creationTimestamp: "2025-09-16T15:08:31Z" |
| 75 | + labels: |
| 76 | + app: web-app |
| 77 | + name: webserver |
| 78 | + namespace: default |
| 79 | + resourceVersion: "2169" |
| 80 | + uid: a9c988ea-ed2f-4f6d-92eb-ed282f273c6c |
| 81 | + spec: |
| 82 | + containers: |
| 83 | + - image: nginx:latest |
| 84 | + imagePullPolicy: IfNotPresent |
| 85 | + name: nginx-container |
| 86 | + resources: {} |
| 87 | + terminationMessagePath: /dev/termination-log |
| 88 | + terminationMessagePolicy: File |
| 89 | + volumeMounts: |
| 90 | + - mountPath: /var/log/nginx |
| 91 | + name: shared-logs |
| 92 | + - mountPath: /var/run/secrets/kubernetes.io/serviceaccount |
| 93 | + name: kube-api-access-cmx2n |
| 94 | + readOnly: true |
| 95 | + - command: |
| 96 | + - sh |
| 97 | + - -c |
| 98 | + - while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log; sleep |
| 99 | + 30; done |
| 100 | + image: ubuntu:latest |
| 101 | + imagePullPolicy: Always |
| 102 | + name: sidecar-container |
| 103 | + resources: {} |
| 104 | + terminationMessagePath: /dev/termination-log |
| 105 | + terminationMessagePolicy: File |
| 106 | + volumeMounts: |
| 107 | + - mountPath: /var/log/nginx |
| 108 | + name: shared-logs |
| 109 | + - mountPath: /var/run/secrets/kubernetes.io/serviceaccount |
| 110 | + name: kube-api-access-cmx2n |
| 111 | + readOnly: true |
| 112 | + dnsPolicy: ClusterFirst |
| 113 | + enableServiceLinks: true |
| 114 | + nodeName: kodekloud-control-plane |
| 115 | + preemptionPolicy: PreemptLowerPriority |
| 116 | + priority: 0 |
| 117 | + restartPolicy: Always |
| 118 | + schedulerName: default-scheduler |
| 119 | + securityContext: {} |
| 120 | + serviceAccount: default |
| 121 | + serviceAccountName: default |
| 122 | + terminationGracePeriodSeconds: 30 |
| 123 | + tolerations: |
| 124 | + - effect: NoExecute |
| 125 | + key: node.kubernetes.io/not-ready |
| 126 | + operator: Exists |
| 127 | + tolerationSeconds: 300 |
| 128 | + - effect: NoExecute |
| 129 | + key: node.kubernetes.io/unreachable |
| 130 | + operator: Exists |
| 131 | + tolerationSeconds: 300 |
| 132 | + volumes: |
| 133 | + - emptyDir: {} |
| 134 | + name: shared-logs |
| 135 | + - name: kube-api-access-cmx2n |
| 136 | + projected: |
| 137 | + defaultMode: 420 |
| 138 | + sources: |
| 139 | + - serviceAccountToken: |
| 140 | + expirationSeconds: 3607 |
| 141 | + path: token |
| 142 | + - configMap: |
| 143 | + items: |
| 144 | + - key: ca.crt |
| 145 | + path: ca.crt |
| 146 | + name: kube-root-ca.crt |
| 147 | + - downwardAPI: |
| 148 | + items: |
| 149 | + - fieldRef: |
| 150 | + apiVersion: v1 |
| 151 | + fieldPath: metadata.namespace |
| 152 | + path: namespace |
| 153 | + status: |
| 154 | + conditions: |
| 155 | + - lastProbeTime: null |
| 156 | + lastTransitionTime: "2025-09-16T15:08:31Z" |
| 157 | + status: "True" |
| 158 | + type: Initialized |
| 159 | + - lastProbeTime: null |
| 160 | + lastTransitionTime: "2025-09-16T15:08:31Z" |
| 161 | + message: 'containers with unready status: [nginx-container]' |
| 162 | + reason: ContainersNotReady |
| 163 | + status: "False" |
| 164 | + type: Ready |
| 165 | + - lastProbeTime: null |
| 166 | + lastTransitionTime: "2025-09-16T15:08:31Z" |
| 167 | + message: 'containers with unready status: [nginx-container]' |
| 168 | + reason: ContainersNotReady |
| 169 | + status: "False" |
| 170 | + type: ContainersReady |
| 171 | + - lastProbeTime: null |
| 172 | + lastTransitionTime: "2025-09-16T15:08:31Z" |
| 173 | + status: "True" |
| 174 | + type: PodScheduled |
| 175 | + containerStatuses: |
| 176 | + - image: nginx:latests |
| 177 | + imageID: "" |
| 178 | + lastState: {} |
| 179 | + name: nginx-container |
| 180 | + ready: false |
| 181 | + restartCount: 0 |
| 182 | + started: false |
| 183 | + state: |
| 184 | + waiting: |
| 185 | + message: Back-off pulling image "nginx:latests" |
| 186 | + reason: ImagePullBackOff |
| 187 | + - containerID: containerd://675404859dcc69803025764a26dafaf3d651c8827af1c6379ad10091f77b048d |
| 188 | + image: docker.io/library/ubuntu:latest |
| 189 | + imageID: docker.io/library/ubuntu@sha256:590e57acc18d58cd25d00254d4ca989bbfcd7d929ca6b521892c9c904c391f50 |
| 190 | + lastState: {} |
| 191 | + name: sidecar-container |
| 192 | + ready: true |
| 193 | + restartCount: 0 |
| 194 | + started: true |
| 195 | + state: |
| 196 | + running: |
| 197 | + startedAt: "2025-09-16T15:08:35Z" |
| 198 | + hostIP: 172.17.0.2 |
| 199 | + phase: Pending |
| 200 | + podIP: 10.244.0.5 |
| 201 | + podIPs: |
| 202 | + - ip: 10.244.0.5 |
| 203 | + qosClass: BestEffort |
| 204 | + startTime: "2025-09-16T15:08:31Z" |
| 205 | + ``` |
| 206 | +
|
| 207 | +5. lets re-create the pod |
| 208 | +
|
| 209 | + ```sh |
| 210 | + kubectl delete pod webserver |
| 211 | + kubectl apply -f pod.yml |
| 212 | + ``` |
| 213 | +
|
| 214 | +6. We can verify: |
| 215 | +
|
| 216 | + ```sh |
| 217 | + kubectl get pod webserver |
| 218 | + ``` |
| 219 | +
|
| 220 | + > we can see the pod is running |
| 221 | +
|
| 222 | +7. Access webpage |
| 223 | +
|
| 224 | + ```sh |
| 225 | + curl localhost:8080 |
| 226 | + ``` |
0 commit comments