--- title: "Postmortem: Two Self-Inflicted Service Mesh Outages from Config, Not Software" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/incidents/post/mesh-config-ordering-outages --- [Home](/)›[Incidents](/incidents)›[All Categories](/incidents/categories)›[Reliability](/incidents/categories/reliability) Incidents [Prev in ReliabilityThree Dashboards Reported Zero Revenue, and Every Job Exited 0](/incidents/post/cron-silent-zero-revenue-failure)[Next in ReliabilityA Regional Control-Plane Failure Took a Payments API Offline](/incidents/post/single-region-payments-api-regional-outage) [Reliability](/incidents/categories/reliability)[Kubernetes](/incidents/categories/kubernetes) # Two Outages Where We Told Envoy Something We Did Not Mean [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 21 Oct 2026Updated: 26 Oct 202603 Mins read05 Mins listen [Markdown for AI(opens in a new tab)](/incidents/post/mesh-config-ordering-outages/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR - A service mesh rollout caused two outages. Both were self-inflicted and both were configuration rather than software. - One was a `VirtualService` applied before its `DestinationRule`. The other was a `PeerAuthentication` scoped to the wrong namespace, which enforced strict mTLS against a service whose caller was not yet meshed. - Retry policy stacked too: mesh-level and code-level retries together turned a slow dependency into a hammered one. > The mesh did what it was told. That was the problem. ### Two Outages Where We Told Envoy Something We Did Not Mean Contents [Summary](#summary)[Impact](#impact)[Timeline](#timeline)[Root cause](#root-cause)[Contributing factors](#contributing-factors)[What changed](#what-changed)[Did it hold](#did-it-hold)[What I would do differently](#what-i-would-do-differently) ## [Summary](#summary) Did the sidecars cause outages? Two, both self-inflicted, and both configuration rather than software. One was a `DestinationRule` ordering mistake. The other was a `PeerAuthentication` scoped to the wrong namespace, which enforced strict mTLS against a service whose caller was not yet meshed. Neither was Envoy failing; both were us telling it something we did not mean. ## [Impact](#impact) - Every request routed to an undefined subset failed, on the first outage. The error names the destination service, so the alert fired on `payments`, the on-call engineer paged the payments team, and the payments team correctly reported that they were fine. - A service whose caller was not yet meshed lost its callers, on the second, when strict mTLS arrived in a namespace it did not belong in. - Before the mesh existed at all, the diagnosis cost was its own incident class: finding which hop was slow took hours, sometimes a day, because each team had instrumented their own service differently, or not at all. ## [Timeline](#timeline) No clock for either outage; both were short, caught by their own blast radius, and reverted. What the record holds is the mechanism, and both mechanisms are deterministic enough to deserve the space: - **Outage one:** a `VirtualService` (“send 5% to subset v2”) applied before the `DestinationRule` that defines what `v2` means. Every request routed to the undefined subset fails immediately. - **Outage two:** a `PeerAuthentication` scoped to the wrong namespace, enforcing strict mTLS against a service whose caller was not yet meshed. ## [Root cause](#root-cause) Configuration expressed real intent poorly. The `VirtualService` and the `DestinationRule` are two halves of one routing decision that Kubernetes happily applies in either order, and only one order is correct. The namespace scope on the `PeerAuthentication` was written once and read everywhere it mattered. ## [Contributing factors](#contributing-factors) **mTLS arrived by accumulation in reverse.** The project started because an auditor asked how we encrypted traffic between services and the honest answer was that we mostly did not: everything inside the VPC talked over plain HTTP, roughly 15% encrypted before the rollout. **The migration had no owner for three months.** The mesh went un-upgraded through two releases while it sat on the platform team’s list somewhere below everything with a deadline. A mesh with no owner becomes the thing every team blames when something is slow and nobody has the standing to change. **Retries at two layers.** We already had retries and timeouts in application code. Adding them at the mesh layer meant that during one incident a request was being retried three times by the sidecar and twice by the caller, turning a slow dependency into a hammered one. ## [What changed](#what-changed) The full rollout is the case study: eleven weeks of `PERMISSIVE` mode, flipped to `STRICT` only after a daily plain-text check had returned nothing for five consecutive days. The incident-specific changes: 1. **Apply the `DestinationRule` first, always.** The ordering rule is now written into the rollout procedure, not carried in one person’s head. 2. **Namespace scoping reviewed per policy**, not inherited. 3. **Mesh-level retries removed; the code-level ones kept.** One layer owns retry behaviour. 4. **Ownership named before the next migration**, after three months proved what ownerless infrastructure decays into. ## [Did it hold](#did-it-hold) Flipping `PERMISSIVE` to `STRICT` across everything was a one-line change and a non-event, because the telemetry had been checked first. Encrypted service-to-service traffic went from roughly 15% to 100%. Time to identify a slow hop went from hours, sometimes a day, to minutes, measured against four incidents after the rollout. Seven canary-caught regressions in six weeks never became incidents at all. ## [What I would do differently](#what-i-would-do-differently) Treat config ordering as part of the interface. “Two objects that only work in one order” is a design smell in any system, and in a mesh it is an outage with a two-week fuse. The full rollout, the mTLS migration path, and the measured numbers are written up in [Service Mesh Rollout: When It Is Worth It](/case-studies/post/service-mesh-rollout-when-worth-it). Was this useful? ## Tags [#Postmortem](/incidents/tags/postmortem)[#Service Mesh](/incidents/tags/service-mesh)[#Istio](/incidents/tags/istio)[#MTLS](/incidents/tags/mtls)[#Configuration](/incidents/tags/configuration) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fmesh-config-ordering-outages "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Two%20Outages%20Where%20We%20Told%20Envoy%20Something%20We%20Did%20Not%20Mean&url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fmesh-config-ordering-outages "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fmesh-config-ordering-outages&title=Two%20Outages%20Where%20We%20Told%20Envoy%20Something%20We%20Did%20Not%20Mean&summary=A%20service%20mesh%20caused%20two%20outages%2C%20both%20self-inflicted%20and%20both%20configuration%20rather%20than%20software%3A%20a%20VirtualService%20applied%20before%20its%20DestinationRule%2C%20and%20a%20PeerAuthentication%20scoped%20to%20the%20wrong%20namespace.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Two%20Outages%20Where%20We%20Told%20Envoy%20Something%20We%20Did%20Not%20Mean%20https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fmesh-config-ordering-outages "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fmesh-config-ordering-outages&text=Two%20Outages%20Where%20We%20Told%20Envoy%20Something%20We%20Did%20Not%20Mean "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fmesh-config-ordering-outages&title=Two%20Outages%20Where%20We%20Told%20Envoy%20Something%20We%20Did%20Not%20Mean "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fmesh-config-ordering-outages&t=Two%20Outages%20Where%20We%20Told%20Envoy%20Something%20We%20Did%20Not%20Mean "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fincidents%2Fpost%2Fmesh-config-ordering-outages&media=&description=A%20service%20mesh%20caused%20two%20outages%2C%20both%20self-inflicted%20and%20both%20configuration%20rather%20than%20software%3A%20a%20VirtualService%20applied%20before%20its%20DestinationRule%2C%20and%20a%20PeerAuthentication%20scoped%20to%20the%20wrong%20namespace. "Share on Pinterest")[Email]() ## Comments ## Continue on this topic The same subject, covered a different way from the article above. - [ ![](/_astro/hero.D85wlyeO_Z2h3jXe.webp) Article ### Service Mesh Deep Dive: Istio vs. Linkerd Trying to figure out service mesh? This article compares Istio and Linkerd on Kubernetes, looking at how they handle traffic, security, and more. Find out which one might be the best fit for you. ](/blog/post/service-mesh-istio-vs-linkerd) - [ ![](/_astro/hero.Uvd7kA_G_Z1rXexc.webp) Case study ### A Service Mesh Rollout That Was Worth It Istio across 34 services on EKS: what mTLS, traffic shifting and free golden signals actually bought us, what the sidecars cost in memory and upgrade pain, and the two answers that would have made the whole thing a mistake. ](/case-studies/post/service-mesh-rollout-when-worth-it) - [ Cheatsheet ### kubectl kubectl is the command-line tool for talking to a Kubernetes cluster. Use it to deploy apps, inspect and manage resources, stream logs, and debug running pods. ](/cheatsheets/kubectl) - [ ![](/_astro/hero.CXbTjs-G_1Jhua8.webp) Quiz ### ArgoCD: GitOps Automation with Kubernetes Test your knowledge of ArgoCD fundamentals with this quiz covering GitOps principles, application deployment, sync policies, health status, ApplicationSets, and continuous delivery best practices for Kubernetes. ](/quizzes/post/argocd-gitops-automation-quiz) ## You might also enjoy More posts on similar topics ## [Three Dashboards Reported Zero Revenue, and Every Job Exited 0](/incidents/post/cron-silent-zero-revenue-failure) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Reliability - Data Engineering Summary The failure mode that finally forced the pipeline rewrite was mundane. A vendor changed an SFTP filename. One job silently produced a zero-row file, six downstream jobs consumed that file #Postmortem#Cron#SQS+3 tags [read more](/incidents/post/cron-silent-zero-revenue-failure) ## [A Regional Control-Plane Failure Took a Payments API Offline](/incidents/post/single-region-payments-api-regional-outage) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Reliability - Architecture Summary A payments API that moves real money had run in a single AWS region for years. A regional control-plane incident took the whole service offline for a few hours. There was no second region #Postmortem#AWS#Multi Region+3 tags [read more](/incidents/post/single-region-payments-api-regional-outage) ## [Three Weeks of "Sometimes Doesn't Load", While p50 Stayed Green](/incidents/post/p99-tail-latency-three-week-blind-spot) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Performance - Observability Summary The complaint that started this was not about speed. It was a support ticket saying the dashboard "sometimes doesn't load", from a customer whose account was roughly forty times larger tha #Postmortem#Latency#Caching+3 tags [read more](/incidents/post/p99-tail-latency-three-week-blind-spot) ## [The Admission Gate Denied an Image Built from a Fork](/incidents/post/slsa-admission-gate-fork-built-image) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - Security - CI/CD Summary The enforcement phase of a supply-chain hardening program denied 23 pods in its first six weeks. Most were rollout tail: twenty stale base images, two images built by hand on a developer l #Postmortem#SLSA#Supply Chain+3 tags [read more](/incidents/post/slsa-admission-gate-fork-built-image) 4 related posts