NOTE: This was tested on OCP 4.11 with OpenShift Service Mesh 2.2 (Istio 1.12)

Just a braindump. I recently came across a scenario where ingress traffic for the service mesh was not first routed to an ingress gateway. Instead a regular route in OpenShift was used which directly routed the traffic to the pod.

In the response of a HTTP request to that application, you could see a few headers security-aware people might not be too fond of, in particular these were:

  • x-powered-by -> was set by application
  • server
  • x-envoy-upstream-service-time
  • x-envoy-decorator-operation

The response in question had headers included similar to that:

< x-envoy-upstream-service-time: 0
< server: istio-envoy
< x-envoy-decorator-operation: api-endpoint.bookinfo.svc.cluster.local:8080/*

The x-powered-by headers was set by the application itself.

How would you normally get rid of these with an ingress-gateway?

When using an ingress gateway, the decorator header is not included in the response, so, nothing to do here \o/ As far as the rest is concered, you’d still get them in the response:

< x-envoy-upstream-service-time: 1
< server: istio-envoy

There’s a good overview and a couple of suggestions in this issue on github

So, let’s try the virtualservice first:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-endpoint
spec:
  gateways:
  - my-gateway
  hosts:
  - api-endpoint-via-ingressgw.apps.example.com
  http:
  - headers:
      response:
        remove:
        - x-envoy-upstream-service-time
        - server
    match:
      # omitted
    route:
      # omitted

If we try that one out, we’ll see that server is still included in the response. Scalability in terms of every development team might need to include it might be an issue with this solution as well, as pointed out in the above mentioned Github issue.

Anyhow, let’s continue and try to remove the server header. Since we can’t do it with the virtualservice, my next try is with an envoyfilter. The istio community has already solved a lot of problems. That one too: https://github.com/istio/istio/issues/13861#issuecomment-854501441 is one instance.

So, let’s try it out:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-response-remove-headers
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          server_header_transformation: PASS_THROUGH
  - applyTo: ROUTE_CONFIGURATION
    match:
      context: GATEWAY
    patch:
      operation: MERGE
      value:
        response_headers_to_remove:
        - "server"

The server_header_transformation property set to PASS_THROUGH basically says don’t touch the server header according to the envoy docs, and we remove it afterwards via the route config’s option to remove response headers.

Let’s give it a go:

* Connected to api-endpoint-via-ingressgw.apps.ocp4.example.com (192.168.70.3) port 80 (#0)
> GET /headers HTTP/1.1
> Host: api-endpoint-via-ingressgw.apps.ocp4.example.com
> User-Agent: curl/7.82.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 878
< date: Wed, 12 Oct 2022 13:59:11 GMT
< set-cookie: d532e04ae658c119b00ba289b6065a41=4e8f8864884eddf5ecd45b7d0a31fb1c; path=/; HttpOnly
< cache-control: private
< 
* Connection #0 to host api-endpoint-via-ingressgw.apps.ocp4.example.com left intact

That looks good.

So, now that we’ve removed the server header where I’m using an envoyfilter anyway, let’s remove the x-envoy-upstream-service-time there as well and remove the respective config from the virtualservice and give it a go -> the result is as we expect it to be, both headers are removed.

What if we don’t use an ingress gateway? We need to strip the decorator header then as well. Our current envoyfilter selects the ingress gateway and uses GATEWAY as context. From the istio docs this is probably not how we can address this according to the PatchContext:

The specific config generation context to match on. Istio Pilot generates envoy configuration in the context of a gateway, inbound traffic to sidecar and outbound traffic from sidecar.

We’re in the area of inbound traffic to the sidecar with the scenario of traffic from the router to the pod.

So, here’s the filter that worked for me:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: response-headers-filter
spec:
  workloadSelector:
    labels:
      app: myapp
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          server_header_transformation: PASS_THROUGH
  - applyTo: HTTP_ROUTE
    match:
      context: SIDECAR_INBOUND
    patch:
      operation: MERGE
      value:
        decorator:
          propagate: false # removes the decorator header
        response_headers_to_remove:
        - x-envoy-upstream-service-time
        - x-powered-by
        - server