If you are trying out the dev preview of the multi-primary (in different network) multi-cluster deployment model in Red Hat OpenShift Service Mesh (nice blog post with a basic walk through): great, have fun!

Assuming that’s up and running in your 2 or more clusters. Once you check out Kiali, you might wonder how to visualize your shiny new traffic flow across multiple clusters. Some work needs to be done.

First of all, the Kiali docs are here (I leave it up the reader to switch to the required Kiali version in the docs, of you are, e.g. using the v1.73 in case you use the supported version part of the OpenShift Service Mesh): https://kiali.io/docs/configuration/multi-cluster/

Do not skip the prerequisites. You’ll circle back to them eventually.

  • Aggregated metrics: A simple solution for the lazy homelabbers, that does not scale very well is using prometheus’s remore_write option. Configure prometheus in cluster2 to remote-write to cluster1. Cluster1 (in my case, the cluster is called south) serves as one and only Kiali instance.

  • Authentication: If you are coming from a single mesh / single cluster installation, you likeky have, like me, the default authentication strategy configured in Kiali, which is openshift. It’s confy and also provides out of the RBAC. But, if you check the Kiali docs, it’s not an option. And yes, of course I’ve tried, because I only skimmed through the docs at the first go :)

So, for now, ignoring RBAC (that’s something for later to look at), keeping it simple: anonmyous it is. But no authentication and no authorization whatsoever might not be the best idea. We could at least put an oauth-proxy in front and maybe put Kiali into the mesh to be able to e.g. apply some additional Authorization Policies.

So, let’s do it.

In my setup, Kiali is not managed by the SMCP and I took a few shortcuts. Here’s the Kiali CR, using Kiali v1.73, where you’ll see that I’m skipping e.g. authentication for proemtheus and grafana, and disable tracing in Kiali entirely. Lab and all …

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: kiali
spec:
  auth:
    strategy: anonymous
  deployment:
    logger:
      log_level: info
    namespace: kiali
    pod_annotations:
      proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
    pod_labels:
      sidecar.istio.io/inject: 'true'
    view_only_mode: false
  external_services:
    grafana:
      in_cluster_url: 'https://grafana-http.istio-system.svc:3000'
      url: 'https://grafana-istio-system.apps.north.ocp4.lab-janine.de'
    istio:
      config_map_name: istio-basic
      istio_sidecar_injector_config_map_name: istio-sidecar-injector-basic
      istiod_deployment_name: istiod-basic
      url_service_version: 'http://istiod-basic.istio-system:15014/version'
    tracing:
      enabled: false
    prometheus:
      url: 'http://prometheus-http.istio-system.svc:9090'
  istio_namespace: istio-system
  version: default

This puts Kiali inside the mesh, as per Kiali docs.

Now, the oauth-proxy. Configs, docs etc can be found here: https://github.com/openshift/oauth-proxy I’ll spare you the nitty gritty details for each and every secret that is mounted in the deployment. The gist of the deployment boils down to, at least for now, until I might find the time for revisiting:

  • separate deployment instead of trying to fiddle it into Kiali somehow. there was no obvious way to do it, so for now I rather deploy it separately.
  • upstream is the kiali service. it’s https, signed by the service-ca, so we need to ensure the oauth-proxy trusts the service-ca.
  • add it to the mesh
  • don’t forget to configure the redirect uri

a few yaml snippets covering these bits:

Deployment of oauth-proxy:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: kiali-oauth
  namespace: kiali
  labels:
    app: kiali-oauth
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kiali-oauth
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kiali-oauth
      annotations:
        sidecar.istio.io/inject: 'true'
    spec:
      ...
      serviceAccountName: kiali-oauth
      securityContext: {}
      containers:
        - name: kiali-proxy
          ...
          env:
            - name: SSL_CERT_DIR
              value: /kialibundle
          ports:
            - name: https
              containerPort: 3001
              protocol: TCP
            - name: http
              containerPort: 3080
              protocol: TCP
          volumeMounts:
            - name: secret-kiali-oauth-tls
              mountPath: /etc/tls/private
            - name: secret-htpasswd
              mountPath: /etc/proxy/htpasswd
            - name: secret-kiali-proxy
              mountPath: /etc/proxy/secrets
            - name: kiali-cabundle
              readOnly: true
              mountPath: /kialibundle
          args:
            - '-provider=openshift'
            - '-https-address=:3001'
            - '-http-address=:3080'
            - '-upstream=https://kiali.kiali.svc.cluster.local:20001'
            - '-htpasswd-file=/etc/proxy/htpasswd/auth'
            - '-display-htpasswd-form=false'
            - '-openshift-sar={"namespace": "istio-system", "resource": "pods", "verb": "get"}'
            - '-client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token'
            - '-openshift-service-account=kiali-oauth'
            - '-cookie-secret-file=/etc/proxy/secrets/session_secret'
            - '-tls-cert=/etc/tls/private/tls.crt'
            - '-tls-key=/etc/tls/private/tls.key'
            - '-openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
      serviceAccount: kiali-oauth
      volumes:
        - name: secret-kiali-oauth-tls
          secret:
            secretName: kiali-oauth-tls
            defaultMode: 420
        - name: secret-htpasswd
          secret:
            secretName: htpasswd
            items:
              - key: auth
                path: auth
            defaultMode: 420
        - name: secret-kiali-proxy
          secret:
            secretName: kiali-proxy
            defaultMode: 420
        - name: kiali-cabundle
          configMap:
            name: kiali-cabundle
            defaultMode: 420
      dnsPolicy: ClusterFirst

Some things to note:

  • I’m reusing the kaili-cabundle configmap which just contains the cluster’s service-ca (check for the annotation service.beta.openshift.io/inject-cabundle: 'true').
  • htpasswd and the kiali-proxy secret contain, well, the htpasswd, and the session secret for the oauth-proxy.
  • the env SSL_CERT_DIR is me probably just doing things rather quickly - that’s a clear item for revisitation at some point. I’m using the fact that you can point go, which Kiali is written in, to look in directories of your choosing for trust bundles. For now, using it to point it to the path where it’ll find the service CA certificate.
  • kiali-oauth-tls and '-https-address=:3001'. Technically I do not need TLS for my oauth-proxy. It’s part of the mesh. Neither am I using it, however, I couldn’t quickly find how to turn it off. And since, if https is on, it’s looking for a tls.key and crt, I inject a secret, created by the servce signer, that contains it. I simply annotated the kiali-oauth service with the respective annotation to get it created.

The easy bit, the service

kind: Service
apiVersion: v1
metadata:
  name: kiali-oauth
  namespace: kiali
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: kiali-oauth-tls
spec:
  ports:
    - name: https-kiali-oauth
      protocol: TCP
      port: 3001
      targetPort: 3001
    - name: http-kiali-oauth
      protocol: TCP
      port: 3080
      targetPort: 3080
  internalTrafficPolicy: Cluster
  type: ClusterIP
  sessionAffinity: None
  selector:
    app: kiali-oauth

As mentioned just before, the annotation is there. For the sake of completeness, I also included both in the service, http as well as https. We’re just using the http port though.

Now, it’s an oauth-proxy, we also need a redirect URI. We can set that by annotation the service account that’s running the oauth-proxy:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kiali-oauth
  namespace: kiali
  annotations:
    serviceaccounts.openshift.io/oauth-redirecturi.primary: 'https://kiali.mesh.apps.north.ocp4.lab-janine.de'

With the oauth-proxy now running, we still need to route our ingress traffic to the proxy. I’m running in a mesh with a wildcard subdomain for the mesh ingress as well as an Istio Gateway. Check my older blog post on how this is set up.

The final piece, the VirtualService:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kiali-mesh
  namespace: kiali
spec:
  gateways:
    - istio-system/mesh-apps-gateway
  hosts:
    - kiali.mesh.apps.north.ocp4.lab-janine.de
  http:
    - route:
        - destination:
            host: kiali-oauth.kiali.svc.cluster.local
            port:
              number: 3080
          weight: 100
  tls:
    - match:
        - port: 443
          sniHosts:
            - kiali.mesh.apps.north.ocp4.lab-janine.de
      route:
        - destination:
            host: kiali-oauth.kiali.svc.cluster.local
            port:
              number: 3080
          weight: 100

That’s it. Now Kiali is in the mesh, simple authentication via oauth-proxy and even simpler RBAC via sar.

How it actually looks in Kiali then:

How it looks in Kiali