Istio - EnvoyFilter to strip headers from response when no Ingress Gateway is used
NOTE: This was tested on OCP 4.11 with OpenShift Service Mesh 2.2 (Istio 1.12)
Just a braindump. I recently came across a scenario where ingress traffic for the service mesh was not first routed to an ingress gateway. Instead a regular route in OpenShift was used which directly routed the traffic to the pod.
In the response of a HTTP request to that application, you could see a few headers security-aware people might not be too fond of, in particular these were:
- x-powered-by -> was set by application
The response in question had headers included similar to that:
< x-envoy-upstream-service-time: 0 < server: istio-envoy < x-envoy-decorator-operation: api-endpoint.bookinfo.svc.cluster.local:8080/*
x-powered-by headers was set by the application itself.
How would you normally get rid of these with an ingress-gateway?
When using an ingress gateway, the decorator header is not included in the response, so, nothing to do here \o/ As far as the rest is concered, you’d still get them in the response:
< x-envoy-upstream-service-time: 1 < server: istio-envoy
There’s a good overview and a couple of suggestions in this issue on github
So, let’s try the virtualservice first:
apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: name: api-endpoint spec: gateways: - my-gateway hosts: - api-endpoint-via-ingressgw.apps.example.com http: - headers: response: remove: - x-envoy-upstream-service-time - server match: # omitted route: # omitted
If we try that one out, we’ll see that
server is still included in the response. Scalability in terms of every development team might need to include it might be an issue with this solution as well, as pointed out in the above mentioned Github issue.
Anyhow, let’s continue and try to remove the
server header. Since we can’t do it with the virtualservice, my next try is with an envoyfilter.
The istio community has already solved a lot of problems. That one too: https://github.com/istio/istio/issues/13861#issuecomment-854501441 is one instance.
So, let’s try it out:
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: gateway-response-remove-headers namespace: istio-system spec: workloadSelector: labels: istio: ingressgateway configPatches: - applyTo: NETWORK_FILTER match: context: GATEWAY listener: filterChain: filter: name: "envoy.filters.network.http_connection_manager" patch: operation: MERGE value: typed_config: "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager" server_header_transformation: PASS_THROUGH - applyTo: ROUTE_CONFIGURATION match: context: GATEWAY patch: operation: MERGE value: response_headers_to_remove: - "server"
server_header_transformation property set to
PASS_THROUGH basically says don’t touch the
server header according to the envoy docs, and we remove it afterwards via the route config’s option to remove response headers.
Let’s give it a go:
* Connected to api-endpoint-via-ingressgw.apps.ocp4.example.com (192.168.70.3) port 80 (#0) > GET /headers HTTP/1.1 > Host: api-endpoint-via-ingressgw.apps.ocp4.example.com > User-Agent: curl/7.82.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < content-type: application/json < content-length: 878 < date: Wed, 12 Oct 2022 13:59:11 GMT < set-cookie: d532e04ae658c119b00ba289b6065a41=4e8f8864884eddf5ecd45b7d0a31fb1c; path=/; HttpOnly < cache-control: private < * Connection #0 to host api-endpoint-via-ingressgw.apps.ocp4.example.com left intact
That looks good.
So, now that we’ve removed the
server header where I’m using an envoyfilter anyway, let’s remove the
x-envoy-upstream-service-time there as well and remove the respective config from the virtualservice and give it a go -> the result is as we expect it to be, both headers are removed.
What if we don’t use an ingress gateway? We need to strip the decorator header then as well. Our current envoyfilter selects the ingress gateway and uses
GATEWAY as context. From the istio docs this is probably not how we can address this according to the
The specific config generation context to match on. Istio Pilot generates envoy configuration in the context of a gateway, inbound traffic to sidecar and outbound traffic from sidecar.
We’re in the area of inbound traffic to the sidecar with the scenario of traffic from the router to the pod.
So, here’s the filter that worked for me:
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: response-headers-filter spec: workloadSelector: labels: app: myapp configPatches: - applyTo: NETWORK_FILTER match: context: SIDECAR_INBOUND listener: filterChain: filter: name: "envoy.filters.network.http_connection_manager" patch: operation: MERGE value: typed_config: "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager" server_header_transformation: PASS_THROUGH - applyTo: HTTP_ROUTE match: context: SIDECAR_INBOUND patch: operation: MERGE value: decorator: propagate: false # removes the decorator header response_headers_to_remove: - x-envoy-upstream-service-time - x-powered-by - server