Doordash share their hard-earned experience of configuring pro | CatOps

Doordash share their hard-earned experience of configuring probes in Kubernetes.

One of the incidents they had was caused by a readinessProbe that triggered a connection to a downstream resource. Unfortunately, it’s very common to see health checks that are not scoped to a service itself but also rely on service’s dependencies.

The trickiest part is that it’s not entirely bad thing. Sometimes it makes sense to indicate that your service won’t work unless a dependency is present. However, service owners should be fully aware of the consequences.

Here are the takeaways from this article neatly collected by Doordash themselves:

1.       Understand the different applications of the various Kubernetes probes. Ensure the entire department is aware of these use cases.
2.       Verify the applications and options regarding any third-party health check endpoints. Consider disabling certain features on third-party tools.
3.       Treat health check endpoints as Tier 0 by instrumenting them with various observability methods and ensuring they are not ignored by the observability tooling. If health checks are providing too much data, consider sampling them or reducing the volume of data they share.
4.       Having a health-checks depend on a backend dependency can be problematic, as an outage within your dependency can cause you to have an outage as Kubernetes restarts your containers.

#kubernetes