Microservices Security and Zero Trust

Advanced System Design & Software Architecture · Lesson 44 · Microservices and Enterprise Architecture

Lesson Overview

The modular monolith in Layered and Modular Monolith Architecture trusted its own JVM. OrderService called PricingService in-process. The database password sat in one place. A stolen laptop did not let a stranger invoke capture(). Splitting that process into pods on a shared cluster removes the assumption and leaves the credentials behind unless you replace them on purpose.

Zero trust, in the sense of NIST SP 800-207, is the replacement assumption. No network location is trusted. A packet that arrived from the same namespace is not a friend. Every request is authenticated, authorized for that specific operation, and observed. Least privilege means the order pod’s identity can insert into the order schema and call payment’s capture API, and it cannot read the catalog’s cost table or the cluster’s secret store. The perimeter VPN is not the control. The call is the control.

Two identities get conflated in reviews, and the designs fail differently. The user identity is the learner, expressed as an OAuth 2.0 access token (typically a JWT) issued by an identity provider that speaks OpenID Connect for login. The workload identity is the order service itself: a Kubernetes ServiceAccount, a SPIFFE ID such as spiffe://cluster.local/ns/orders/sa/order, or an OAuth client that uses the client-credentials grant. mTLS proves which workload is on the socket. The JWT proves which user delegated which scope, or which client is calling as itself. Neither one substitutes for the other. A mesh that encrypts every hop and then lets any pod with a valid mesh certificate call POST /refunds has confidentiality without authorization.

This lesson uses Spring Security’s resource server (oauth2ResourceServer().jwt) and OAuth2 client (ServletOAuth2AuthorizedClientExchangeFilterFunction on WebClient) because those types validate issuers, audiences, and signatures instead of decoding Base64 in a filter. It uses Kubernetes NetworkPolicy and, where a mesh is present, Istio PeerAuthentication with STRICT mTLS and an AuthorizationPolicy that names the source principal. It does not invent a gateway SDK. The traces from Observability: Logging, Metrics and Tracing are how you see a 401. The contracts from API Design, Versioning and Contracts are what authorization is allowing. Module 5’s payment, multi-tenant SaaS, and e-commerce case studies are where a missing audience check becomes a cross-tenant incident rather than a diagram footnote.

Learning Objectives

By the end of this lesson, you should be able to:

1. State the zero-trust assumption in operational terms: network location, namespace, and “internal” DNS names are not authentication.
2. Separate user identity (authorization-code or session login, then a user access token) from workload identity (client credentials, ServiceAccount, or a SPIFFE ID presented through mTLS).
3. Validate a service JWT with Spring Security’s JwtDecoder: signature via JWKS, iss, exp / nbf, and an aud that names this service, rejecting alg=none and untrusted jku URLs.
4. Explain why forwarding the learner’s bearer token to every downstream service is a confused deputy, and when client credentials or token exchange (RFC 8693) is the hop you want instead.
5. Require mutual TLS either in the application (server.ssl.client-auth=need) or at a mesh (PeerAuthentication mode STRICT), and still enforce application authorization after the handshake.
6. Write a Kubernetes NetworkPolicy that admits only the peers a deployment should have, including the DNS exception that default-deny egress always needs.
7. Give each service a database role limited to its schema, and keep long-lived secrets out of images and out of arguments visible in ps and process listings.
8. Threat-model one checkout call well enough to name the control that stops lateral movement, token replay, and a public client forging traceparent or X-User-Id.

Prerequisites

• Lesson 34, API Gateway and Backend for Frontend: the edge already terminates browser TLS and validates the learner JWT. This lesson starts at the first internal hop.
• Lesson 40, API Design, Versioning and Contracts: scopes attach to operations you have named. Authorization without a contract is a list of paths that drift.
• Lesson 43, Observability: Logging, Metrics and Tracing: you will log sub and a stable error code, not the token, and you will not honor a public traceparent as a free sampling switch.
• Lesson 7, Layered and Modular Monolith Architecture: schema-per-module grants are the database half of least privilege. Extraction does not widen them.
• Familiarity with TLS certificates at the level of “a private key stays on one side of a handshake.” You do not need to operate a mesh.

Detailed Core Concepts

The assumption that replaced the perimeter

A traditional network put services on an RFC 1918 subnet and treated every caller on that subnet as internal. Security groups or a firewall opened ports. The application checked the user at the front door and then trusted X-User-Id from anything that could route. That model fails in a Kubernetes cluster for a mechanical reason: many workloads share the network, any compromised pod can open a TCP connection to a ClusterIP, and a stolen developer kubeconfig is not “outside.” NIST SP 800-207’s useful sentence for this course is that trust is not implied by network location. Authentication and authorization are evaluated per request. The encryption of the transport is necessary and is not the authorization decision.

Zero trust is not a product you buy and wire in front of the old flat network. It is a set of constraints on identity, policy, and visibility. For TechLearningPro those constraints are concrete. Every pod has its own ServiceAccount. Every service has an OAuth client or a mesh identity, not a shared “internal” basic-auth password. Every request that mutates an order carries a token whose audience is the order service. Every packet path you did not name is denied by a NetworkPolicy. Every denial is a metric and a log line without the credential. You will not finish all of that in one quarter. You can refuse to add a new service that depends on the old assumption.

User tokens and workload tokens answer different questions

The learner signs in through an identity provider. The browser receives an access token whose sub is the learner, whose iss is the provider, and whose scopes include what the product allows a learner to do, such as orders.write on their own cart. Spring Security’s resource server turns that into an Authentication whose authorities are the scopes prefixed with SCOPE_ by the default JwtGrantedAuthoritiesConverter. hasAuthority("SCOPE_orders.write") is that default, not a custom invention. The token is a user credential. Its audience should be the gateway or the BFF the learner called, not every internal service in the company.

The order service calls payment because a purchase must capture funds. Payment is not interested in impersonating the learner across the platform. It needs to know that the caller is the order service, and that this call is a capture for an order the order service has already authorized. That is a workload credential. The OAuth 2.0 client-credentials grant (RFC 6749) is the standard shape: the order service authenticates as a client with its own client_id and a secret or a private key (JWT bearer client assertion, RFC 7523, when you are ready to stop shipping a shared secret), and it receives an access token whose sub or client_id is the service and whose aud is payment. There is no learner password in that exchange.

Forwarding the learner’s token to payment, pricing, and the warehouse is the confused deputy. Payment now holds a credential that can do everything the learner can do at every API that accepts that audience. A bug or a server-side request in payment becomes the learner. If the token’s audience was never restricted, it becomes every API. The deputy pattern feels convenient because the downstream service “just works” with the same decoder. The cost shows up the day a low-privilege worker gains a code-execution bug.

When the downstream service truly must enforce the learner’s rights, use token exchange (RFC 8693) or a token the authorization server mints for that audience with a reduced scope. The caller presents the user token to the token endpoint with grant_type=urn:ietf:params:oauth:grant-type:token-exchange and receives a new token whose aud is only the downstream resource. The original token is not replayed. Many internal hops do not need this. Pricing a SKU is a workload call: client credentials, and the user id is a field in the request body that the order service is accountable for, not a credential.

What “validate the JWT” has to include

Decoding the middle segment of a JWT and trusting sub is not validation. A resource server has to check all of the following before it builds an Authentication.

The signature must verify against a key the service obtained from a trusted JWKS URL, not from a jku header the attacker supplied, and not from a key embedded in the token. alg=none must be rejected. Algorithms you do not use must be rejected so a token signed with a weaker algorithm you forgot to disable cannot pass. NimbusJwtDecoder, which Spring Security uses, verifies the signature. A hand-written split("\\.") decoder does not.

iss must equal the issuer you configured. exp must be in the future and nbf, if present, must be in the past, with a small clock skew (Spring’s validator uses a bounded skew). aud must contain this resource. A token minted for the gateway is not a token for payment. Spring’s default issuer validator does not, by itself, express your audience rule. You add a JwtClaimValidator on aud or an equivalent OAuth2TokenValidator and set it on the decoder. Without that line, any token from the same issuer is accepted by every service that shares the issuer. That is the most common “we use JWT” hole in microservice estates.

Scopes or roles in the token authorize the operation, not merely the door. SCOPE_orders.read does not imply POST /refunds. Map authorities from a claim you documented (scope or scp, or a roles claim if you must) with an explicit converter, and write the matcher on the method and path. A single .authenticated() on every route means a valid token for “read catalog” can capture a payment if the client can reach the URL.

Tokens are bearer instruments. Anyone who sees one can replay it until exp. Keep lifetimes short for service tokens (minutes, not days). Prefer proof-of-possession or mTLS-bound access tokens when the threat model includes token theft from logs, and until then do not log the token. Do not put tokens in query strings. TLS is what stops the replay from being trivial on the wire; it does not stop replay of a token copied from a log index.

mTLS proves the peer, then authorization still runs

Mutual TLS means both sides present certificates. The client checks the server name as usual. The server requires a client certificate (server.ssl.client-auth=need in Spring Boot, not want, which allows anonymous clients to proceed). The client certificate’s identity becomes the workload identity. In a mesh such as Istio, the sidecar or ambient proxy terminates that handshake. The application often still sees HTTP from localhost. That is fine only if the mesh identity is what your policy uses and the application does not trust a spoofable header instead.

Istio’s control is two objects. PeerAuthentication with mtls.mode: STRICT in a namespace refuses plaintext from peers that have not presented a mesh certificate. AuthorizationPolicy with action: ALLOW and a source.principals entry such as cluster.local/ns/checkout/sa/checkout is the allow list for an operation. A policy that only sets STRICT mTLS still allows every mesh identity to call every port. STRICT without an allow policy is encryption, not least privilege.

SPIFFE gives the certificate a URI SAN: spiffe://cluster.local/ns/orders/sa/order. That name is stable when pods are replaced. It is a better workload identity than a DNS name that moves with a Service, and it is a better identity than a long-lived client secret stored in a Kubernetes Secret that every developer with kubectl get secret can read. Where you do not run a mesh, client credentials over one-way TLS plus a NetworkPolicy is an acceptable, explicit step. Pretending that “we will add mTLS later” while shipping a shared internal password is how later becomes never. Write down which hop is still a shared secret.

Application-level authorization does not disappear behind the mesh. The mesh knows the source ServiceAccount. It does not know that this capture is for an order this service owns, or that the amount matches the authorization you stored. Those are business rules. The mesh is the transport and the workload allow-list. The service is the business rule. Teams that move all checks into YAML recreate a gateway god-object in the mesh config, and teams that trust the mesh principal and skip the business rule refund arbitrary order ids presented by any allowed caller. You want both layers, aimed at different questions.

Network policy is the coarse blast-radius limit

A NetworkPolicy selects pods and, once it selects them, denies the directions you list as policy types except the rules you add. The useful baseline is: the order pods accept ingress only from the gateway or the checkout namespace, on port 8080, and they egress only to payment, to their database, to DNS, and to the telemetry collector. Everything else is closed. This does not replace JWT checks. It limits which compromised workloads can even attempt the check. A marketing pod in another namespace should not be able to open a socket to payment, even if it somehow obtained a token.

DNS is the rule people forget. CoreDNS typically lives in kube-system on UDP and TCP 53. A default-deny egress that does not allow it makes the application fail at startup with “unknown host,” which gets misdiagnosed as an application bug. The collector and the database have to be explicit too. Policies are code. Review them with the same pull request as the deployment, and test them by running a pod that should be denied and showing the timeout.

Kubernetes RBAC is the control plane version of the same idea. The order ServiceAccount does not need list secrets in the cluster. automountServiceAccountToken: false on workloads that do not call the API server removes a credential attackers love to find mounted at /var/run/secrets/kubernetes.io/serviceaccount. Workloads that do call the API use a projected token with a narrow audience and a short expiration, not the legacy unbounded token.

Data stores are part of the trust boundary

Each service owns its schema, as the modular monolith already required. The database role used by the order pod has USAGE on the order schema and DML on its tables, not SUPERUSER, and not SELECT on payment tables that happen to sit in the same cluster during a migration. A shared database with distinct roles is still a shared failure domain, which Architecture Migration and Modernization will narrow. It is already a better privilege boundary than one application user that can read every table.

Secrets: the image does not contain the password. The process does not take it as a command-line argument (those appear in the node’s process list and often in the API audit log). A Kubernetes Secret mounted as a file is the baseline, with RBAC so only that ServiceAccount’s pods can mount it, and with etcd encryption at rest turned on by the platform. A short-lived database credential issued by the cloud IAM or a vault agent is strictly better when you have it, because a copied Secret remains valid until someone rotates it. Rotation has to be a rehearsed path, not a wiki sentence. Spring will read spring.datasource.password from an environment variable that the platform injected. Do not log the datasource URL if it embeds a password. Prefer a URL without credentials and a separate password property.

Architecture and Design Diagrams

The learner token stops at the edge. Internal hops use a workload credential. mTLS covers the socket. The network policy limits who can open the socket. The database role limits what a stolen JDBC password can read.

mermaidCopy
flowchart LR
  Browser[Learner browser]
  IdP[Identity provider]
  GW[API gateway]
  Order[order-service]
  Pay[payment-service]
  DB[(order schema)]

  Browser -->|authorization code| IdP
  Browser -->|user access token aud=gateway| GW
  GW -->|mTLS + exchanged or service token| Order
  Order -->|client credentials aud=payment| Pay
  Order -->|schema role, no shared superuser| DB

What the order service checks before it captures. Audience and workload identity are different gates. Both fail closed.

mermaidCopy
sequenceDiagram
  participant C as Checkout workload
  participant O as Order service
  participant P as Payment service
  participant I as Identity provider

  C->>O: POST /orders<br/>mTLS + user-scoped token aud=order
  Note over O: verify signature, iss, exp, aud<br/>require SCOPE_orders.write<br/>authorize this learner owns the cart
  O->>I: client_credentials<br/>client_id=order-service
  I-->>O: access token aud=payment
  O->>P: POST /charges<br/>mTLS + workload token
  Note over P: aud is payment<br/>caller is order-service<br/>not the learner token
  P-->>O: 201
  O-->>C: 201

A compromised pod’s reach after policy. The marketing namespace is not in the allow list, so a stolen payment URL is not a usable path even before the JWT check.

mermaidCopy
flowchart TD
  M[marketing pod] -->|denied by NetworkPolicy| P[payment]
  C[checkout pod] -->|allowed :8080| O[order]
  O -->|allowed| P
  O -->|allowed| DB[(order schema)]
  O -->|denied| Cat[(catalog tables)]

Technical Explanation

Spring Security resource server, with an audience check

spring-boot-starter-oauth2-resource-server wires a filter that authenticates bearer tokens. The configuration that matches this lesson is stateless, explicitly authorizes by method and scope, and does not use the learner session.

javaCopy
@Configuration
@EnableWebSecurity
public class OrderSecurityConfig {

    @Bean
    SecurityFilterChain orderChain(HttpSecurity http) throws Exception {
        http
            .csrf(csrf -> csrf.disable())
            .sessionManagement(s -> s.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
            .authorizeHttpRequests(auth -> auth
                .requestMatchers("/actuator/health/liveness", "/actuator/health/readiness").permitAll()
                .requestMatchers(HttpMethod.POST, "/orders").hasAuthority("SCOPE_orders.write")
                .requestMatchers(HttpMethod.GET, "/orders/{id}").hasAuthority("SCOPE_orders.read")
                .anyRequest().denyAll())
            .oauth2ResourceServer(oauth2 -> oauth2.jwt(Customizer.withDefaults()));
        return http.build();
    }
}

CSRF protection is disabled because this endpoint does not use a browser cookie session. A cookie-authenticated BFF does not get this shortcut. denyAll on everything you did not name beats a trailing .authenticated() that grows new routes into “any valid token.”

The decoder below takes the issuer from configuration, uses Spring’s default issuer-and-time validator, and adds the audience. Jwt.getAudience() is a list. A token with no aud, or with an aud for a different service, fails.

javaCopy
@Bean
JwtDecoder jwtDecoder(@Value("${spring.security.oauth2.resourceserver.jwt.issuer-uri}") String issuer) {
    NimbusJwtDecoder decoder = NimbusJwtDecoder.withIssuerLocation(issuer).build();
    OAuth2TokenValidator<Jwt> audience = new JwtClaimValidator<List<String>>(
            "aud",
            aud -> aud != null && aud.contains("order-service"));
    decoder.setJwtValidator(new DelegatingOAuth2TokenValidator<>(
            JwtValidators.createDefaultWithIssuer(issuer),
            audience));
    return decoder;
}

withIssuerLocation loads the provider’s OpenID Connect discovery document and then the JWKS. Do not point it at a host an inbound request can influence. Pin the issuer URI in configuration. The same pattern on payment uses aud containing payment-service, so a token stolen from a verbose order log cannot be replayed there if the audiences differ. If you mistakenly use one audience for the whole mesh, theft of any service token is theft of all of them. That is the audience bug one level up.

Scopes are not ownership. SCOPE_orders.read allows the operation. The method must still check that authentication.getName() (the JWT sub) is allowed to see this order id. A missing object-level check is how one learner reads another’s invoice. Multi-tenant SaaS in Module 5, Design a Multi-Tenant SaaS Platform, is the same hole with a tenant id: the token’s tenant claim and the row’s tenant column have to match, and a client-supplied tenant header is not the source of truth.

Calling the next service as a workload

The OAuth2 client registration uses authorization-grant-type: client_credentials. The WebClient filter attaches the bearer token for that registration and refreshes it when the provider says it expired. You do not cache the token in a static field with no expiry.

javaCopy
@Bean
WebClient paymentClient(ClientRegistrationRepository registrations,
                        OAuth2AuthorizedClientRepository authorizedClients) {
    ServletOAuth2AuthorizedClientExchangeFilterFunction oauth =
            new ServletOAuth2AuthorizedClientExchangeFilterFunction(registrations, authorizedClients);
    oauth.setDefaultClientRegistrationId("payment");
    return WebClient.builder()
            .baseUrl("https://payment.internal.techlearningpro.example")
            .apply(oauth.oauth2Configuration())
            .build();
}

The registration’s token URI is the identity provider, the scope is the narrow charges.capture, and the client secret comes from the environment or a mounted file, not from the git repository. Payment’s resource server expects aud=payment-service. The order service does not copy the inbound Authorization header onto this call. If a future requirement says payment must see the learner, pass learnerId in the JSON body under the order service’s authority, or exchange a downscoped token. Do not do both “just in case.”

mTLS in the pod when you are not on a mesh

yamlCopy
server:
  ssl:
    enabled: true
    key-store: file:/keys/order-tls.p12
    key-store-password: ${TLS_KEYSTORE_PASSWORD}
    trust-store: file:/keys/client-ca.p12
    trust-store-password: ${TLS_TRUSTSTORE_PASSWORD}
    client-auth: need

need rejects the handshake without a client certificate signed by a CA in the trust store. Certificate rotation has to overlap: the trust store accepts the current and the next CA, then the old CA leaves. A rotation that restarts every pod in lockstep is an outage you scheduled. Mesh proxies exist partly so application teams do not each build this rotation. If the platform provides STRICT mTLS, do not also terminate a second, unrelated TLS layer unless you have a reason you can say in one sentence. Double TLS is latency and a second set of broken trust stores.

An Istio allow policy for the same intent names the principal, not the pod IP:

yamlCopy
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: order-allow-checkout
  namespace: orders
spec:
  selector:
    matchLabels:
      app: order-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/checkout/sa/checkout
      to:
        - operation:
            methods: ["POST"]
            paths: ["/orders"]

A missing policy with Istio’s default can be “allow all authenticated peers” depending on the mesh installation. Treat the default as something you read in the installed version’s docs, not as folklore. The safe review question is: what is denied if a new ServiceAccount appears tomorrow?

Headers the application must not trust

X-User-Id, X-Tenant-Id, and X-Internal-User set by a caller are not identities. If the gateway authenticates the user and then calls the order service, the order service still validates a token. A header is acceptable as a hint only when the mesh or the gateway overwrites it after authentication and the network policy prevents any other caller from reaching the pod. The usual way that exception goes wrong is a second ingress, a debug port, or a new route that bypasses the overwrite. Prefer the token. It travels with a signature.

The same rule applies to the support and trace headers from Lesson 43. X-Request-Id is correlation. It is not authentication. A public traceparent is not a reason to trust the caller.

Practical Implementation or Design Walkthrough

Problem

Checkout, order, and payment are separate deployments. Learners authenticate at the gateway. A staff tool must refund an order. A compromised catalog pod must not be able to call payment or read order rows. Service tokens must not be reusable across services. The design has to survive a developer pasting a curl with a copied bearer token.

Requirements

• Gateway validates the learner JWT (issuer, signature, expiry, audience gateway) and exchanges or otherwise obtains an order-scoped token for POST /orders. The inbound browser token is not sent to payment.
• Order accepts only aud=order-service, requires SCOPE_orders.write for create, and checks the learner owns the cart.
• Order calls payment with client credentials, scope charges.capture, aud=payment-service, over TLS. mTLS is STRICT if the mesh is on; otherwise client-auth: need on payment’s ingress from the mesh or the gateway, and one-way TLS plus network policy on the service-to-service hop you have not moved yet. Do not claim mTLS on a hop that is still a shared secret. Label that hop in the ADR as residual risk.
• NetworkPolicy on payment: ingress from order namespace only, egress to the acquirer and DNS. Catalog cannot open the socket.
• Database: role order_app with rights on schema orders only.
• Logs: sub, client_id, path, decision. Never the raw JWT. Lesson 43’s redaction test covers this.
• Deny by default on routes you did not list.

Steps

1. Pin issuer-uri and the expected audience in each service’s configuration. Add the JwtClaimValidator shown above. A test using Spring Security’s test support posts a token with the wrong aud and expects 401.
2. Register the payment client with client_credentials. Confirm with a packet capture or a stub server that the outbound Authorization value is not the inbound one. This is the confused-deputy test, and it is cheaper than the incident.
3. Split scopes: learners do not get refunds.write. A staff client does, through a separate client and a group claim you map with a JwtAuthenticationConverter only if the default scope converter is not the claim you issue. Document the claim name. Do not read a "roles" claim you have not configured a converter for and assume Spring saw it. It did not.
4. Apply the AuthorizationPolicy or the equivalent application check so only the staff workload identity, not the checkout ServiceAccount, can call POST /orders/{id}/refunds. Checkout’s network path to that URL should fail at the policy layer too.
5. Apply NetworkPolicy, including DNS and the collector, in a namespace test before production. Run a pod in the catalog namespace that tries payment:8080 and show the timeout in the pull request.
6. Create order_app and revoke the migration-era user that owns every schema. Flyway runs as a different role that is not the runtime role.
7. Set automountServiceAccountToken: false on catalog and other workloads that do not call the API. Leave a projected token only on the controller that needs it.

Failure behavior

Expired service tokens are refreshed by the OAuth client filter. A down identity provider means new tokens cannot be issued; existing unexpired tokens still work, which is a reason to keep a short but non-zero lifetime and to cache the JWKS (the decoder does) so payment can still validate during a brief IdP blip. An IdP outage that exceeds token lifetime is an availability incident. Treat the IdP as a dependency in the SLI sense of Lesson 43, with a clear failure mode: checkout fails closed, it does not skip authentication.

A clock skew beyond the validator’s allowance causes spurious 401s after a deploy. NTP is a platform requirement, not an application retry.

Wrong audience is a 401, logged as aud_rejected, without the token string. Alerting on a spike of 401s catches both an attack and a bad configuration rollout.

Expected result

A token minted for the gateway is rejected by order and by payment. The checkout ServiceAccount can create an order and cannot refund. A pod in catalog cannot open a TCP connection to payment. The order database role cannot SELECT from catalog tables. A log line for a denied call contains the caller identity and not the bearer token. That is the security half of an operable microservice, and it is the bar for the payment and SaaS case studies.

Enterprise Real-World Scenarios

A payment capture path and a staff refund path

Acquirers and card networks will not accept “the cluster is private” as authentication. Design a Payment Processing System needs a workload identity for capture, a different principal for refunds, and an audit log of client_id plus the human sub when a staff member initiated the refund. The user token that started checkout is the wrong credential to store in the capture worker. The worker may run minutes later, the user token may be expired, and replaying it would widen audience. The durable authorization is the order service’s record that this order was approved, plus the worker’s own client credential to call the acquirer adapter. Staff refunds are a second client, a second scope, and a second NetworkPolicy source. Combining them because “it’s all payment” is how a buggy storefront flow gains the refund scope. The case study should draw two arrows into the payment namespace, with different principals, not one trusted pipe.

A multi-tenant SaaS control plane

Design a Multi-Tenant SaaS Platform adds a tenant boundary on top of the same mechanics. The token carries a tenant id the issuer put there, not a tenant id the browser chose. Every query includes that tenant. A workload that runs background jobs uses a tenant-scoped credential or an explicit job record that names the tenant, not a cluster-wide super token that “passes the tenant in a header.” Cross-tenant access is a break-glass role with a short lifetime, a reason code, and an audit span attribute. mTLS between the control plane and the data plane stops a forged header from another namespace from being enough. The zero-trust statement in that case study is testable: a token for tenant A presented to a URL that contains tenant B’s id returns 403, and a pod outside the data-plane allow list never reaches the socket.

Best Practices

• Validate issuer, signature, time, and audience in every service. Share a small library or a platform filter if you must, but do not share one audience.
• Use client credentials (or a mesh principal) for workload hops. Forward the user token only to the component that is the user’s resource server.
• Authorize the operation and the object. Scope checks and “this learner owns this order” are both required.
• Prefer denyAll plus explicit routes over a terminal .authenticated().
• Run STRICT mTLS where the mesh exists, and an AuthorizationPolicy that names principals. Say so when a hop is still only server TLS.
• Default-deny NetworkPolicy, with DNS, datastore, and telemetry written down. Test a pod that must be rejected.
• Separate runtime database roles from migration roles. No superuser in the application Secret.
• Keep token lifetimes short, rotate client secrets, and never log Authorization.
• Turn off automount of the API token unless the process calls the API server.
• Review a new service with one question: if this pod is compromised, what can it call and what can it read? If the answer is “the subnet,” the design is not done.

Common Mistakes and Anti-Patterns

Trusting the subnet or the namespace. “Internal services have no auth” feels like a migration shortcut. Any pod that can route to the Service can call it. The fix is a workload credential and a network policy on day one, even if the credential is a single client-credentials client you intend to specialize later.

Forwarding the learner JWT. Downstream services become confused deputies, and a log in any of them is a bearer token for the user. Exchange or replace the token at the hop. Pass user identifiers as data, under the caller’s authority.

Validating signature and not audience. Every service that trusts the same issuer accepts every token that issuer signs. Pin aud. The test is a token for service A presented to service B.

client-auth: want or permissive mesh defaults. Optional client certificates train callers to present nothing. STRICT or need, plus an allow list, is the setting that matches the threat model you claimed.

A shared database user during “temporary” coexistence. The extracted service and the monolith both log in as app. A SQL injection in the monolith reads the new service’s tables. Separate roles before you separate deployables, which is the seam Lesson 7 already asked for.

Headers as identity. X-User-Id set by the client, or set by the gateway but still accepted from anywhere that can reach the pod, is an authentication bypass. The signature on the token is the control. The header is correlation.

Long-lived tokens in CI logs and crash dumps. A client secret or a JWT printed by a debug interceptor survives in the log index far past exp when someone also disabled expiry in a test decoder and then copied that decoder. Production decoders do not have an expiry switch.

Troubleshooting and Design Review

Symptom: random 401s after deploy, worse on one node. Cause: clock skew, or a JWKS cache still holding a rotated key while the signer has moved, or the opposite, a signer publishing a new key the decoder has not fetched. Check: pod NTP, the kid on the failing token, and whether withIssuerLocation can reach discovery. Fix: fix time, confirm JWKS refresh, and keep overlapping keys during rotation. Do not “fix” this by disabling expiry.

Symptom: payment accepts a token captured from the gateway logs. Cause: shared audience or payment not checking aud. Fix: distinct audiences and the validator above. Rotate the leaked token’s client and shorten lifetime.

Symptom: catalog pod can curl payment. Cause: no NetworkPolicy selects the payment pods, so the default remains allow. Fix: a policy that selects them. Confirm with a test pod, not with kubectl describe alone.

Symptom: order pods crash-loop with UnknownHostException after a policy change. Cause: egress deny without DNS. Fix: allow UDP/TCP 53 to the DNS namespace, then re-test the deny you still want.

Symptom: refunds succeed for the checkout client. Cause: one scope orders.write used for create and refund, or an AuthorizationPolicy that allows the whole namespace. Fix: a distinct scope and a distinct source principal. Add a test that authenticates as checkout and receives 403 on the refund route.

Design review. Where does the user token stop? What is the aud of each hop? Which hops are mTLS and which are residual shared secrets? What can a compromised pod reach at L3 and at the database role? Which header would you believe if the signature check were removed, and why is that header still in the code? Is the IdP failure mode closed?

Interview Questions and Answers

Q1. What does zero trust change about a call from checkout to order inside the cluster?

Network location is not a credential. The call still has to authenticate a workload or a user, authorize the specific operation, and travel over a transport that authenticates the peer. “Same namespace” and “ClusterIP” are not on that list. NIST SP 800-207 is the usual citation. The practical test is a new pod, with no special firewall exception, attempting the call and failing.

Q2. Why are a learner JWT and a client-credentials token different?

The learner token represents a person and the scopes that person delegated, and its audience should be the front door that person called. The client-credentials token represents the order service as itself, with a scope such as charges.capture and an audience of payment. One answers “who is the human?” The other answers “which workload is calling?” Using the human token for the workload hop makes every downstream service a deputy that can act as the human.

Q3. What do you check before you trust a JWT, beyond the signature?

Issuer, expiry, not-before, and audience, against values pinned in the service, using keys from a JWKS URL you configured. Reject alg=none and any jku the token tries to choose. Then authorize the scope for the route and the object (this learner, this order). Spring Security’s NimbusJwtDecoder and JwtValidators.createDefaultWithIssuer cover signature, issuer, and time. Audience is a JwtClaimValidator you add. Skipping audience is the bug that lets one issuer’s token work everywhere.

Q4. What is the confused deputy in microservice hops?

A service that holds a user’s broad credential and uses it to call somewhere the user did not directly call. Checkout forwards the browser token to order, which forwards it to payment and to a reporting tool. A bug in reporting is now able to act as the learner at every API that accepts that token. The fix is a workload credential on the hop, or token exchange (RFC 8693) that mints a new token with the downstream audience and a smaller scope.

Q5. What does mTLS give you that one-way TLS plus a bearer token does not, and what does it not give you?

mTLS authenticates the calling workload at the handshake, so a stolen bearer token used from a different client certificate fails, and a process that never received a certificate cannot connect. It does not authorize the business operation, it does not bind the call to a particular order id, and STRICT mode alone does not restrict which mesh identity may connect. You still want an allow policy and an application check. If you are not actually terminating client certificates, do not claim this property.

Q6. How do you stop a compromised catalog pod from calling payment?

A NetworkPolicy that selects payment pods and allows ingress only from the order workload, plus payment’s own authentication so a forged source IP inside the allowed path is still not enough. RBAC so catalog’s ServiceAccount cannot read payment’s Secret. A database role that cannot see payment tables. Defense in depth is several of these, not a debate about which single one is elegant. Test by running the call from a catalog pod.

Q7. Why is X-User-Id from the gateway a weak identity?

It is unsigned. Any caller who can reach the pod can set it, unless every possible path overwrites the header and every other path is unreachable. A second ingress, a sidecar debug port, or a mis-routed service mesh policy bypasses the overwrite. A JWT signed by the identity provider travels with its own check. Correlation headers such as X-Request-Id stay unsigned on purpose and are never treated as sub.

Q8. What is wrong with one Kubernetes Secret and one database user shared by the monolith and the new service?

Anyone who can compromise or merely log the older, larger codebase can use that user. The privilege boundary you meant to create by extracting a service does not exist at the data layer. Runtime roles should be per service and per schema. Migration tools use a different role. Secrets should be mountable only by the ServiceAccount that needs them. A shared secret also means a rotation restarts every workload at once.

Q9. The identity provider is down. What should checkout do?

Fail closed. Do not skip JWT validation because discovery times out. Decoders should keep a cached JWKS so tokens that are already issued can still be checked through a short IdP blip. New logins and new client-credentials grants will fail once caches and token lifetimes run out, and that is an availability incident you page, not a reason to accept unsigned traffic. The failure belongs on the checkout SLI.

Q10. How should Module 5’s payment and multi-tenant designs show this lesson?

The payment design names two principals into the payment namespace, capture and refund, with different scopes, and it does not store the shopper’s access token for a later worker. The SaaS design takes the tenant from the token the issuer signed, enforces it on every query, and shows a 403 when the path’s tenant and the token’s tenant differ. Both name which hops are mTLS, which NetworkPolicy selects the pods, and what a compromised neighboring workload can still reach. A box labeled “internal auth” is not an answer.

Hands-On Exercises

Exercise 1: Lock down order and payment

Problem. Specify identities, token rules, and network rules for gateway, order, and payment as described in the walkthrough. Include a staff refund that checkout must not be able to call.

Tasks.

1. For each hop, name the credential type, the aud, and the scope. State where the learner token stops.
2. List the JWT checks the order decoder performs. Include one negative test (wrong audience, expired, missing scope).
3. Say whether each hop is STRICT mTLS, server.ssl.client-auth=need, or one-way TLS, and label any residual shared secret.
4. Draft the payment NetworkPolicy peers, including DNS. Identify one namespace that must time out.
5. Define the database role for order_app and what it must not be able to read.
6. State what is logged on a 401 and what is forbidden in that log line.
7. Describe IdP-down behavior.

Deliverables. A trust-boundary diagram, the audience table, the policy intent, and the negative tests.

Acceptance. A gateway token is useless at payment. Checkout receives 403 on refund. A catalog pod cannot open payment’s port. No design step relies on “the cluster is private.”

Exercise 2: Review an “internal” migration proposal

The proposal keeps payment unauthenticated from any pod in the cluster, forwards the learner Authorization header from the gateway to payment, uses one database user called app, and identifies the caller with X-User-Id. Write the review. For each choice, name the incident it enables and the replacement control from this lesson. Do not add unrelated product features.

Rubric.

Lesson Summary

Zero trust means a call is authenticated and authorized every time, regardless of namespace and private IP. The learner’s JWT and the service’s workload credential are different instruments: the first stops at the user’s resource server, the second is client credentials or a mesh principal aimed at one audience. Spring Security’s JwtDecoder is the check for signature, issuer, time, and an explicit audience validator. mTLS, whether client-auth: need or Istio STRICT, authenticates the peer and still needs an allow policy and a business rule. NetworkPolicy and per-schema database roles limit what a compromised pod can touch, and they must be tested with a caller that is supposed to fail. Headers are not identity. Tokens do not go in logs. Module 5’s payment and multi-tenant case studies are where these controls become the difference between a diagram and a boundary you can defend.

Further Reading

• NIST SP 800-207, Zero Trust Architecture — the assumption this lesson uses, without adopting a vendor diagram.
• RFC 6749 (OAuth 2.0) and RFC 8693 (token exchange) — client credentials versus exchanging a user token for a narrower audience.
• OpenID Connect Core — how the learner’s login relates to the tokens the gateway validates.
• Spring Security, OAuth 2.0 Resource Server JWT — NimbusJwtDecoder, JwtValidators, oauth2ResourceServer.
• Spring Security, OAuth 2.0 Client — ServletOAuth2AuthorizedClientExchangeFilterFunction and client registrations.
• Kubernetes, Network Policies — default deny once a policy selects a pod, and why DNS must be explicit.
• Istio, PeerAuthentication and AuthorizationPolicy — STRICT mTLS and principal-based allows.
• SPIFFE specification — the URI SAN form spiffe://trust-domain/path as a workload identity.
• Next: Microservices Testing Strategies, where the audience test and the “catalog pod is denied” test become repeatable.
• Module 5: Design a Payment Processing System and Design a Multi-Tenant SaaS Platform.