Kiali – Documentation

News: KIALI-SECURITY-003 - Installation into ad-hoc namespaces

Tue, 11 May 2021 11:00:00 -0600

Description

Disclosure date: May 11, 2021
Affected Releases: prior to 1.33.0
Impact Score: 6.6 - AV:N/AC:L/PR:H/UI:N/S:C/C:L/I:L/A:L

A vulnerability was found in the Kiali Operator allowing installation of a specified image into any namespace.

Kiali users are exposed to this vulnerability if all the following conditions are met:

Kiali operator is used for installation.
Kiali CR was edited to install an image into an unapproved namespace.

This vulnerability is filed as CVE-2021-3495

Mitigation

If you can update:

Update to Kiali Operator v1.33.0 or later.

If you can not update:

Ensure only trusted individuals can create or edit a Kiali CRs (resources of kind “kiali”).

News: KIALI-SECURITY-002 - Authentication bypass when using the OpenID login strategy

Mon, 01 Jan 0001 00:00:00 +0000

Description

Disclosure date: March 5, 2021
Affected Releases: 1.26.0, 1.26.1, 1.26.2, 1.27.0, 1.28.0, 1.28.1, 1.29.0, 1.29.1, 1.30.0
Impact Score: 7.0 - AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:N/E:F/RL:X/RC:C

A vulnerability was found in Kiali allowing an attacker to bypass the authentication mechanism. The vulnerability lets an attacker build forged credentials and use them to gain unauthorized access to Kiali.

Kiali users are exposed to this vulnerability if all the following conditions are met:

Kiali is setup with the openid authentication strategy.
As a result of configurations in both Kiali and your OpenID server, Kiali uses the implicit flow of the OpenID specification to negotiate authentication.
Kiali is setup with RBAC turned off.

This vulnerability is filed as CVE-2021-20278

Mitigation

If you can update:

Update to Kiali v1.31.0 or later.
If you need an earlier version, only Kiali 1.26.3 and 1.29.2 are fixed.

If you are locked with an older version of Kiali, you have three options:

Configure Kiali to use the authorization code flow of the OpenID specification; or
Configure Kiali to use the implicit flow of the OpenID specification and enable RBAC; or
Configure Kiali to use any of the other available authentication mechanisms.

News: KIALI-SECURITY-001 - Authentication bypass using forged credentials

Tue, 24 Mar 2020 11:00:00 -0600

Description

Disclosure date: March 25, 2020
Affected Releases: 0.4.0 to 1.15.0
Impact Score: 9.4 - AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:H/A:H

A vulnerability was found in Kiali allowing an attacker to bypass the authentication mechanism. Currently, Kiali has four authentication mechanisms: login, token, openshift and ldap. All are vulnerable.

The vulnerability lets an attacker build forged credentials and use them to gain unauthorized access to Kiali.

Additionally, it was found that Kiali credentials were not being validated properly. Depending on the authentication mechanism configured in Kiali, this could facilitate unauthorized access into Kiali with forged and/or invalid credentials.

These vulnerabilities are filed as CVE-2020-1762 and CVE-2020-1764

Detection

Use the following bash script to check if you are vulnerable:

KIALI_VERSION=$(kubectl get pods -n istio-system -l app=kiali -o yaml | sed -n 's/^.*image: .*:v\(.*\)$/\1/p' | sort -u)
kubectl get deploy kiali -n istio-system -o yaml | grep -q LOGIN_TOKEN_SIGNING_KEY
TEST_KEY_ENV=$?
kubectl get cm kiali -n istio-system -o yaml | grep signing_key | grep -vq kiali
TEST_KEY_CFG=$?
VERSION_ENTRIES=(${KIALI_VERSION//./ })
echo "Your Kiali version found: ${KIALI_VERSION}"
[ ${VERSION_ENTRIES[0]} -lt "1" ] || ([ ${VERSION_ENTRIES[0]} -eq "1" ] && (\
  [ ${VERSION_ENTRIES[1]} -lt "15" ] || ([ ${VERSION_ENTRIES[1]} -eq "15" ] && ( \
  [ ${VERSION_ENTRIES[2]} -le "0" ])))) && echo "Your Kiali version is vulnerable"
[ $TEST_KEY_ENV -eq 1 ] && [ $TEST_KEY_CFG -eq 1 ] && echo "Your Kiali configuration looks vulnerable"

The script output will be similar to this:

Your Kiali version found: 1.14.0
Your Kiali version is vulnerable
Your Kiali configuration looks vulnerable

Mitigation

Update to Kiali 1.15.1 or later.

Alternatively, if you cannot update to version 1.15.1, mitigation is possible by setting a secure signing key when deploying Kiali. If you installed via Kiali operator, you could use the following bash script:

SIGN_KEY=$(chars=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890; for i in {1..20}; do echo -n "${chars:RANDOM%${#chars}:1}"; done; echo)
kubectl get kiali -n $(kubectl get kiali --all-namespaces --no-headers -o custom-columns=NS:.metadata.namespace) -o yaml | sed "s/spec:/spec:\n    login_token:\n      signing_key: $SIGN_KEY/" | kubectl apply -f -

If you installed via Istio helm charts or istioctl command, you could use the following bash script:

KIALI_INSTALL_NAMESPACE=istio-system
SIGN_KEY=$(chars=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890; for i in {1..20}; do echo -n "${chars:RANDOM%${#chars}:1}"; done; echo)
kubectl get cm kiali -n $KIALI_INSTALL_NAMESPACE -o yaml | sed "s/server:/login_token:\\n      signing_key: $SIGN_KEY\\n    server:/" | kubectl apply -f -
kubectl delete pod -l app=kiali -n $KIALI_INSTALL_NAMESPACE

Docs: Introduction

Mon, 01 Jan 0001 00:00:00 +0000

So far, we know how good Kiali can be to understand applications, their relationships with each other and with external applications.

In the previous tutorial, Kiali was setup to observe just a single cluster. Now, we will expand its capabilities to observe more than one cluster. The extra clusters are remotes, meaning that there is not a control plane on them, they only have user applications.

This topology is called primary-remote and it is very useful to spread applications into different clusters having just one primary cluster, which is where Istio and Kiali are installed.

This scenario is a good choice when as an application administrator or architect, you want to give a different set of clusters to different sets of developers and you also want all these applications to belong to the same mesh. This scenario is also very helpful to give applications high availability capabilities while keeping the observability together (we are referring to just applications in terms of high availability, for Istio, we might want to install a multi-primary deployment model, which is on the roadmap for the multicluster journey for Kiali).

In this tutorial we will be deploying Istio in a primary-remote deployment. At first, we will install the “east” cluster with Istio, then we will add the “west” remote cluster and join it to the mesh. Then we will see how Kiali allows us to observe and manage both clusters and their applications. Metrics will be aggregated into the “east” cluster using Prometheus federation and a single Kiali will be deployed on the “east” cluster.

If you already have a primary-remote deployment, you can skip to instaliing Kiali.

Docs: Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Kiali uses Jaeger as a default distributed tracing backend. In this tutorial, we will replace it for Grafana Tempo.

We will setup a local environment in minikube, and install Kiali with Tempo as a distributed backend. This is a simplified architecture diagram:

We will install Tempo with the Tempo Operator and enable Jaeger query frontend to be compatible with Kiali in order to query traces.
We will setup Istio to send traces to the Tempo collector using the zipkin protocol. It is enabled by default from version 3.0 or higher of the Tempo Operator.
We will install MinIO and setup it up as object store, S3 compatible.

Environment

We use the following environment:

Istio 1.18.1
Kiali 1.72
Minikube 1.30
Tempo operator TempoStack v3.0

There are different installation methods for Grafana Tempo, but in this tutorial we will use the Tempo operator.

Docs: Jaeger

Mon, 01 Jan 0001 00:00:00 +0000

Jaeger configuration

Jaeger is a highly recommended service because Kiali uses distributed tracing data for several features, providing an enhanced experience.

By default, Kiali will try to reach Jaeger at the GRPC-enabled URL of the form http://tracing.<istio_namespace_name>:16685/jaeger, which is the usual case if you are using the Jaeger Istio add-on. If this endpoint is unreachable, Kiali will disable features that use distributed tracing data.

If your Jaeger instance has a different service name or is installed to a different namespace, you must manually provide the endpoint where it is available, like in the following example:

spec:
  external_services:
    tracing:
      # Enabled by default. Kiali will anyway fallback to disabled if
      # Jaeger is unreachable.
      enabled: true
      # Jaeger service name is "tracing" and is in the "telemetry" namespace.
      # Make sure the URL you provide corresponds to the non-GRPC enabled endpoint
      # if you set "use_grpc" to false.
      internal_url: "http://tracing.telemetry:16685/jaeger"
      use_grpc: true
      # Public facing URL of Jaeger
      external_url: "http://my-jaeger-host/jaeger"

Minimally, you must provide spec.external_services.tracing.internal_url to enable Kiali features that use distributed tracing data. However, Kiali can provide contextual links that users can use to jump to the Jaeger console to inspect tracing data more in depth. For these links to be available you need to set the spec.external_services.tracing.external_url to the URL where you expose Jaeger outside the cluster.

Default values for connecting to Jaeger are based on the Istio’s provided sample add-on manifests. If your Jaeger setup differs significantly from the sample add-ons, make sure that Istio is also properly configured to push traces to the right URL.

Jaeger authentication configuration

The Kiali CR provides authentication configuration that will be used also for querying the version check to provide information in the Mesh graph.

spec:
  external_services:
    tracing:
      enabled: true
      auth:
        insecure_skip_verify: false
        password: "pwd"
        token: ""
        type: "basic"
        use_kiali_token: false
        username: "user"
      health_check_url: ""

To configure a secret to be used as a password, see this FAQ entry.

TLS Certificate Configuration

If your Jaeger server uses HTTPS with a certificate issued by a private CA, see the TLS Configuration page to learn how to configure Kiali to trust your CA.

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Platform Setup

This tutorial assumes you will have access to a Kubernetes cluster with Istio installed.

This tutorial has been tested using:

a Minikube installation.
an OpenShift installation.

Tip

Platform dependent tasks will be indicated with a special note like this.

This tutorial has been tested using:

minikube v1.16.0, istio 1.8.1 and kiali v1.28.0
openshift v4.8.3, istio 1.11.0 and kiali v1.39.0

Install Istio

Once you have your Kubernetes cluster ready, follow the Istio Getting Started to install and setup a demo profile that will be used in this tutorial.

Determining ingress IP and ports and creating DNS entries will be necessary in the following steps.

DNS entries can be added in a basic way to the /etc/hosts file but you can use any other DNS service that allows to resolve a domain with the external Ingress IP.

Minikube

This tutorial uses Minikube tunnel feature for external Ingress IP.

OpenShift

This tutorial uses a route for external Ingress IP.

Update Kiali

Istio defines a specific Kiali version as an addon.

In this tutorial we are going to update Kiali to the latest release version.

Assuming you have installed the addons following the Istio Getting Started guide, you can uninstall Kiali with the command:

kubectl delete -f ${ISTIO_HOME}/samples/addons/kiali.yaml --ignore-not-found

There are multiple ways to install a recent version of Kiali, this tutorial follows the Quick Start using Helm Chart.

helm install \
  --namespace istio-system \
  --set auth.strategy="anonymous" \
  --repo https://kiali.org/helm-charts \
  kiali-server \
  kiali-server

Access the Kiali UI

The Istio istioctl client has an easy method to expose and access Kiali:

${ISTIO_HOME}/bin/istioctl dashboard kiali

There are other alternatives to expose Kiali or other Addons in Istio. Check Remotely Accessing Telemetry Addons for more information.

After the Prerequisites you should be able to access Kiali. Verify its version by clicking the “?” icon and selecting “About”:

Docs: Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

Run Kiali locally

Kiali can be run directly on your machine without being installed into a Kubernetes cluster. It uses your kubeconfig to connect to your cluster(s). If needed, it can port-forward into the cluster to connect to your external services (prometheus, tracing, istio, grafana).

Running Kiali locally is currently experimental. Functionality may change between releases.

Download the Kiali binary from the Kiali GitHub releases page for your OS and Arch.

Start Kiali which runs the backend server on localhost and opens your default browser to the Kiali UI.

kiali run

To see the full list of options

kiali run --help

If the cluster name in your kubeconfig does not match the cluster name in Istio you can override this with --cluster-name-overrides kubeconfig-name=istio-cluster-name. The flag is a comma separated list so you can override as many names as you need.

Install Kiali

You can quickly install Kiali into your cluster via one of the following two methods.

These instructions are not recommended for production environments. Find more detailed information on installing Kiali, see the installation guide.

Before you install Kiali you must have already installed Istio along with its telemetry storage addon (i.e. Prometheus). You might also consider installing Istio’s optional tracing addon (i.e. Jaeger) and optional Grafana addon but those are not required by Kiali. Refer to the Istio documentation for details.

Install via Istio Addons

If you downloaded Istio, the easiest way to install and try Kiali is by running:

kubectl apply -f ${ISTIO_HOME}/samples/addons/kiali.yaml

To uninstall:

kubectl delete -f ${ISTIO_HOME}/samples/addons/kiali.yaml --ignore-not-found

Install via Helm

Only Helm v3 has been tested. Previous Helm versions may or may not work.

To install the latest version of Kiali Server using Helm, run the following command:

helm install \
  --namespace istio-system \
  --set auth.strategy="anonymous" \
  --repo https://kiali.org/helm-charts \
  kiali-server \
  kiali-server

If you get a validation error, you may have to pass the option --disable-openapi-validation (this is needed on some versions of OpenShift, for example).

To uninstall:

helm uninstall --namespace istio-system kiali-server

Access to the UI

Run the following command:

kubectl port-forward svc/kiali 20001:20001 -n istio-system

Then, access Kiali by visiting https://localhost:20001/ in your preferred web browser.

News: Release Notes

Mon, 01 Jan 0001 00:00:00 +0000

For additional information check our sprint demo videos and blogs.

2.24.0

Sprint Release: March 30, 2026

Features:

Fixes:

Upgrade Change Notes:

Kiali v2.21 introduces traffic graph caching. Enabled by default. When a user navigates to the Traffic Graph and renders the initial graph, Kiali will start a background job to regenerate the graph periodically, based on the refresh interval set in the UI. The background job will cache the resulting graph and return it on subsequent UI requests. This can greatly improve re-render times, especially for larger graphs. Note that the initial graph render time will be unchanged. It is still recommended to use “Manual” refresh when working with large meshes, in order to fully define the desired graph before performing the initial request. Any fundamental change to the graph definition will invalidate the cache and restart a new refresh job. Users can navigate away and then back to the traffic graph, and resume with the latest cached graph, within the timeout period (10m by default).

Backend resource utilization may be affected, although is not anticipated to change significantly. The caching can be disabled in the Kiali config via:

spec.kiali_internal.graph_cache.enabled: false

Any questions, comments or feedback appreciated. Visit #kiali on Istio Slack or start a Discussion in Github at https://github.com/kiali/kiali.

2.20.0

Sprint Release: December 22, 2025

Features:

Fixes:

2.19.0

Sprint Release: November 24, 2025

Features:

Fixes:

2.18.0

Sprint Release: November 03, 2025

Features:

Fixes:

2.17.0

Sprint Release: October 13, 2025

Features:

Fixes:

Upgrade Change Notes:

The following fields are no longer used by the Kiali CR and MUST be removed, if currently set.

spec.external_services.istio.registry

2.16.0

Sprint Release: September 22, 2025

Features:

Fixes:

Upgrade Change Notes:

Discovery Selectors

Kiali now properly supports Istio control planes deployed into different namespaces. As part of this support both the spec.istio_namespace and spec.external_services.istio.root_namespace configuration fields have been removed. As such, Kiali Discovery Selectors, when defined, must include Istio’s control plane namespace(s). If you are using Kiali Discovery Selectors, please ensure that this new requirement is met. Note that Kiali’s deployment namespace is always included, and so co-located Istio control planes will be discovered.

The following fields are no longer used by the Kiali CR and MUST be removed, if currently set.

spec.external_services.istio.root_namespace

2.15.0

Sprint Release: September 02, 2025

Features:

Fixes:

Upgrade Change Notes:

Version 2.15.0 introduces a CRD schema for Kiali. The CRD version has not changed. But, validation will now occur on the cluster when the Kiali CRs are created or modified.

2.14.0

Sprint Release: August 08, 2025

Features:

Fixes:

Upgrade Change Notes:

The following fields are no longer used by the Kiali CR and MUST be removed, if currently set.

no longer used
- spec.istio_namespace
- spec.in_cluster
- spec.deployment.remote_secret_path
now auto-discovered
- spec.external_services.istio.config_map_name
- spec.external_services.istio.istiod_pod_monitoring_port
- spec.external_services.istio.envoy_admin_local_port
- spec.external_services.istio.istio_canary_version
- spec.external_services.istio.istio_injection_annotation
- spec.external_services.istio.istio_sidecar_annotation
- spec.external_services.istio.istiod_deployment_name
- spec.external_services.istio.istiod_pod_monitoring_port
- spec.external_services.istio.url_service_version

2.13.0

Sprint Release: July 21, 2025

Features:

Fixes:

MeshPage: Fix dataplane namespace count

2.12.0

Sprint Release: Jun 30, 2025

Features:

Fixes:

2.11.0

Sprint Release: Jun 09, 2025

Features:

Fixes:

2.10.0

Sprint Release: May 18, 2025

Features:

Fixes:

2.9.0

Sprint Release: Apr 25, 2025

Features:

Fixes:

2.8.0

Sprint Release: Apr 07, 2025

Features:

Fixes:

Deprecations:

Remove Cytoscape graph implementation

After a 4 month deprecation period the support for Kiali’s original, Cytoscape-based, graph implementation has ended. The ‘spec.kiali_feature_flags.ui_defaults.graph.impl’ configuration setting is no longer supported, and the sole implementation going forward uses PatternFly Topology. This has allowed for a significant cleanup of the Kiali code base, and removal of several dated dependencies. We’d like to thank the Cytoscape project, without which Kiali would not have existed. It is an excellent library, and our migration to PatternFly was motivated by a need to settle on a uniform component library.

2.7.0

Sprint Release: Mar 17, 2025

Features:

Fixes:

2.6.0

Sprint Release: Feb 21, 2025

Features:

Fixes:

Upgrade Notes:

The default values for the following Kiali CR fields have changed:

spec.istio_labels.app_label_name
- previous default: “app”
- new default: unset
spec.istio_labels.version_label_name
- previous default: “version”
- new default: unset

The change is related to the work done for Kiali issue 7603, included with this release. By default Kiali now allows for a mixing app labeling schemes, using the same set of app and version label pairings recognized by Istio:

service.istio.io/canonical-name, service.istio.io/canonical-revision
app.kubernetes.io/name, app.kubernetes.io/version
app, version

Users can configure a single labeling scheme by setting the existing CR fields, or leaving them set when upgrading.

2.5.0

Sprint Release: Feb 03, 2025

Features:

Fixes:

2.4.0

Sprint Release: Jan 13, 2025

Features:

Fixes:

2.3.0

Sprint Release: Dec 23, 2024

Features:

Fixes:

2.2.0

Sprint Release: Dec 02, 2024

Features:

Fixes:

2.1.0

Sprint Release: Nov 11, 2024

Features:

Fixes:

Deprecations:

RedHat Community Operator
- The community operator created confusion as to which operator to use on OpenShift. It will no longer be updated and will eventually be removed. OpenShift users are encouraged to use the productized operator, which is included with licensed copies of OpenShift.

2.0.0

Sprint Release: Oct 21, 2024

The first major Kiali release in over 5 years! There are two main reasons for the major version update:

There is a breaking change in Kiali’s namespace management configuration. To limit the namespaces accessible to Kiali, or made visible to users, Kiali v2.0 users will configure Discovery Selectors.

There is no longer support for the following deprecated configuration settings:

spec.deployment.accessible_namespaces
api.namespaces.exclude
api.namespaces.include
api.namespaces.label_selector_exclude
api.namespaces.label_selector_include

Kiali has a new traffic graph implementation.

The Cytoscape implementation has been deprecated and is no longer the default. Kiali has moved to PatternFly Topology to align with the rest of the Kiali interface, which is already implemented using PatternFly components. The old graph implementation will be removed as soon as the Kiali maintainers believe the new implementation has proven itself in the field. Until that time, it can still be accessed by setting:

spec:
  kiali_feature_flags:
    ui_defaults:
      graph:
        impl: "cy"

Features:

Fixes:

1.89.4

Sprint Release: Sep 30, 2024

Features:

The next feature release will be Kiali v 2.0.0

Fixes:

1.89.3

Sprint Release: Sep 09, 2024

Features:

The next feature release will be Kiali v 2.0.0

Fixes:

1.89.0

Sprint Release: Aug 19, 2024

Features:

Fixes:

1.88.0

Sprint Release: Jul 29, 2024

Features:

Fixes:

1.87.0

Sprint Release: Jul 08, 2024

Features:

Fixes:

Deprecations:

Kiali is deprecating its current namespace selection approach. For a description of the new mechanism see https://github.com/kiali/kiali/blob/master/design/KEPS/namespace-discovery/proposal.md. The following configuration is deprecated:
- spec.deployment.accessible_namespaces
Note that the following settings have already been deprecated and will soon be removed:
- api.namespaces.exclude
- api.namespaces.include
- api.namespaces.label_selector_exclude
- api.namespaces.label_selector_include

Fixes:

1.80.0

Sprint Release: Feb 09, 2024

Features:

Fixes:

1.79.0

Sprint Release: Jan 19, 2024

Features:

Fixes:

1.78.0

Sprint Release: Dec 08, 2023

Features:

Fixes:

Deprecations:

Kiali is deprecating use of the Jaeger exporter for Kiali’s own traces. Kiali will move to supporting only the OTel exporter.

1.67.0

Sprint Release: Apr 21, 2023

Features:

Fixes:

Notes:

There have been changes to Namespace Management. The changes are backward compatible but it is recommended to understand the changes regarding Discovery Selector support, cluster-wide access, and accessible-namespaces. For more see here.

Deprecations:

The following settings are deprecated and may be removed in a future release:
- api.namespaces.exclude
- api.namespaces.include
- api.namespaces.label_selector_exclude
- api.namespaces.label_selector_include

It is recommended to instead use Istio Discovery Selectors to limit the namespaces in the mesh.

1.66.1

Sprint Release: Mar 31, 2023

Features:

Fixes:

Notes:

To avoid a known performance degradation, update to v1.66.1 (or later) from v1.66.0.

Deprecations:

Given that the Kiali cache is now mandatory and that there have been several changes to the cache implementation, the .spec.kubernetes_config.cache_* settings have all deprecated and will be removed from the CRD. It is recommended to remove these settings from your CR, if defined.

1.65.0

Sprint Release: Mar 10, 2023

Features:

Fixes:

1.64.0

Sprint Release: Feb 17, 2023

Features:

Fixes:

1.63.2

Sprint Release: Feb 02, 2023

Features:

Fixes:

Notes:

Helm 3.10 is now required to run the Helm Charts.
To avoid a known performance degradation, update to v1.63.2 (or later) from earlier revisions of v1.63.

Deprecations:

The deprecated support for OpenID’s implicit flow has now been removed. If necessary, you must switch to using the more secure authorization code flow.

1.62.1

Sprint Release: Feb 02, 2023

Features:

Fixes:

Notes:

To avoid a known performance degradation, update to v1.62.1 (or later) from v1.62.0.

Deprecations:

The experimental support for multicluster deployment models is now deprecated and may be removed in a future release. New multicluster features are currently being developed within the Kiali community as a replacement.

1.61.0

Sprint Release: Dec 16

Features:

Fixes:

1.60.0

Sprint Release: Nov 25

Features:

Fixes:

Deprecations:

Support for OpenID’s implicit flow is deprecated and will be removed soon. Please, switch to using the authorization code flow which is more secure.

Known Issues:

With the move to a distroless container in v1.59, root CA certificates went missing. This affects the Kiali integration with OpenID. The problem will be fixed in the next release (v1.61). The workaround is to use one of the “-distro” image tags found on the Quay.io repo by specifying it in the deployment.image_version setting of the Kiali CR or server helm chart value. If using the operator, in order to be able to set the deployment.image_version within the Kiali CR, you must enable the allowAdHocKialiImage setting when installing the operator.

1.59.0

Sprint Release: Nov 4, 2022

Features:

Fixes:

1.58.0

Sprint Release: Oct 14, 2022

Features:

Fixes:

1.57.0

Sprint Release: Sep 23

Features:

Fixes:

Upgrade Notes:

The improved control plane card on the Overview page makes use of previously unused metrics. If these metrics have been removed from your environment you will need to add them back for the feature to work. As a result,

Metric	Notes
process_cpu_seconds_total	used to graph cpu usage in the control plane overview card
container_memory_working_set_bytes	used to graph memory usage in the control plane overview card
pilot_proxy_convergence_time_sum	used in control plane overview card to show the average proxy push time
pilot_proxy_convergence_time_count	used in control plane overview card to show the average proxy push time

If these metrics have been removed from your environment you will need to add them back for the feature to work. As a result, we have updated our recommended Prometheus metric thinning configuration. See kiali.io for the updated configuration. The metrics used are not typically very heavy and adding them back should likely not be an issue.

See this FAQ entry for a list of all metrics required by Kiali.

1.56.0

Sprint Release: Sep 2

Features:

Fixes:

k8s api token not auto refreshing (aws eks warning)

Mon, 01 Jan 0001 00:00:00 +0000

Grafana Tempo Configuration

There are two possibilities to integrate Kiali with Grafana Tempo:

Using the Grafana Tempo API: This option returns the traces from the Tempo API in OpenTelemetry format.
Using the Jaeger frontend with the Grafana Tempo backend.
Appendix: Configuration table

Using the Grafana Tempo API

There are two steps to set up Kiali and Grafana Tempo:

Set up the Kiali CR updating the Tracing and Grafana sections.
Set up a Tempo data source in Grafana.

Set up the Kiali CR

This is a configuration example to set up Kiali tracing with Grafana Tempo:

spec:
  external_services:
    tracing:
      # Enabled by default. Kiali will anyway fallback to disabled if
      # Tempo is unreachable.
      enabled: true
      health_check_url: "https://tempo-instance.grafana.net"
      # Tempo service name is "query-frontend" and is in the "tempo" namespace.
      # Make sure the URL you provide corresponds to the non-GRPC enabled endpoint
      # It does not support grpc yet, so make sure "use_grpc" is set to false.
      internal_url: "http://tempo-tempo-query-frontend.tempo.svc.cluster.local:3200/"
      provider: "tempo"
      tempo_config:
        org_id: "1"
        datasource_uid: "a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00"
        url_format: "grafana"
      use_grpc: false
      # Public facing URL of Tempo 
      external_url: "https://grafana-istio-system.apps-crc.testing/"

Kiali uses the external_url to construct “View in tracing” links in the UI. For the Tempo provider the default url_format is grafana. So, by default the URL will have the Grafana UI format when linking to specific services and traces.

It is also possible to set url_format to openshift. In this case the URL will redirect to the UI Plugin in the OpenShift console. When it is set to openshift, there are other settings as well:

spec:
  external_services:
    tracing:
      tempo_config:
        name: "sample"
        namespace: "tempo"
        tenant: "default"
        url_format: "openshift"

When the tenant is specified, if internal_url doesn’t have a path, it will be autocompleted with the Tempo path. For this example:

internal_url: https://tempo-sample-gateway.tempo.svc.cluster.local:8080/

Will be autocompleted to: https://tempo-sample-gateway.tempo.svc.cluster.local:8080/api/traces/v1/{tenant}/tempo

The other valid option for url_format is jaeger, used when the Jaeger UI is available in Tempo.

Set up a Tempo Datasource in Grafana

We can optionally set up a default Tempo datasource in Grafana so that you can view the Tempo tracing data within the Grafana UI, as you see here:

To set up the Tempo datasource, go to the Home menu in the Grafana UI, click Data sources, then click the Add new data source button and select the Tempo data source. You will then be asked to enter some data to configure the new Tempo data source:

The most important values to set up are the following:

Mark the data source as default, so the URL that Kiali uses will redirect properly to the Tempo data source.
Update the HTTP URL. This is the internal URL of the HTTP tempo frontend service. e.g. http://tempo-tempo-query-frontend.tempo.svc.cluster.local:3200/

Additional configuration

The Traces tab in the Kiali UI will show your traces in a bubble chart:

Increasing performance is achievable by enabling gRPC access, specifically for query searches. However, accessing the HTTP API will still be necessary to gather information about individual traces. This is an example to configure the gRPC access:

spec:
  external_services:
    tracing:
      enabled: true
      # grpc port defaults to 9095
      grpc_port: 9095 
      internal_url: "http://query-frontend.tempo:3200"
      provider: "tempo"
      use_grpc: true
      external_url: "http://my-tempo-host:3200"

Service check URL

By default, Kiali will check the service health in the endpoint /status/services, but sometimes, this is exposed in a different url, which can lead to a component unreachable message:

This can be changed with the health_check_url configuration option.

spec:
  external_services:
    tracing:
      health_check_url: "http://query-frontend.tempo:3200"

Configuration for the Grafana Tempo Datasource

In order to correctly redirect Kiali to the right Grafana Tempo Datasource, there are a couple of configuration options to update:

spec:
  external_services:
    tracing:
      tempo_config:
        org_id: "1"
        datasource_uid: "a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00"

org_id is usually not needed since “1” is the default value which is also Tempo’s default org id. The datasource_uid needs to be updated in order to redirect to the right datasource in Grafana versions 10 or higher.

Using the Jaeger frontend with Grafana Tempo tracing backend

It is possible to use the Grafana Tempo tracing backend exposing the Jaeger API. tempo-query is a Jaeger storage plugin. It accepts the full Jaeger query API and translates these requests into Tempo queries.

Since Tempo is not yet part of the built-in addons that are part of Istio, you need to manage your Tempo instance.

Tanka

The official Grafana Tempo documentation explains how to deploy a Tempo instance using Tanka. You will need to tweak the settings from the default Tanka configuration to:

Expose the Zipkin collector
Expose the GRPC Jaeger Query port

When the Tempo instance is deployed with the needed configurations, you have to set meshConfig.defaultConfig.tracing.zipkin.address from Istio to the Tempo Distributor service and the Zipkin port. Tanka will deploy the service in distributor.tempo.svc.cluster.local:9411.

The external_services.tracing.internal_url Kiali option needs to be set to: http://query-frontend.tempo.svc.cluster.local:16685.

Tempo Operator

The Tempo Operator for Kubernetes provides a native Kubernetes solution to deploy Tempo easily in your system.

After installing the Tempo Operator in your cluster, you can create a new Tempo instance with the following CR:

kubectl create namespace tempo
kubectl apply -n tempo -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: smm
spec:
  storageSize: 1Gi
  storage:
    secret:
      type: s3
      name: object-storage
  template:
    queryFrontend:
      component:
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
      jaegerQuery:
        enabled: true
        ingress:
          type: ingress
EOF

Note the name of the bucket where the traces will be stored in our example is called object-storage. Check the Tempo Operator documentation to know more about what storages are supported and how to create the secret properly to provide it to your Tempo instance.

Now, you are ready to configure the meshConfig.defaultConfig.tracing.zipkin.address field in your Istio installation. It needs to be set to the 9411 port of the Tempo Distributor service. For the previous example, this value will be tempo-smm-distributor.tempo.svc.cluster.local:9411.

Now, you need to configure the internal_url setting from Kiali to access the Jaeger API. You can point to the 16685 port to use GRPC or 16686 if not. For the given example, the value would be http://tempo-ssm-query-frontend.tempo.svc.cluster.local:16685.

There is a related tutorial with detailed instructions to setup Kiali and Grafana Tempo with the Operator.

Configuration table

Supported versions

Kiali Version	Jaeger	Tempo	Tempo with JaegerQuery
<= 1.79 (OSSM 2.5)	✅	❌	✅
> 1.79	✅	✅	✅

Minimal configuration for Kiali <= 1.79

In external_services.tracing

	http	grpc
Jaeger	`.internal_url = 'http://jaeger_service_url:16686/jaeger'` `.use_grpc = false`	`.internal_url = 'http://jaeger_service_url:16685/jaeger'` `.use_grpc = true (Not required: by default)`
Tempo	`.internal_url = 'http://query_frontend_url:16686'` `.use_grpc = false`	`.internal_url = 'http://query_frontend_url:16685'` `.use_grpc = true (Not required: by default)`

Minimal configuration for Kiali > 1.79

	http	grpc
Jaeger	`.internal_url = 'http://jaeger_service_url:16686/jaeger'` `.use_grpc = false`	`.internal_url = 'http://jaeger_service_url:16685/jaeger'` `.use_grpc = true (Not required: by default)`
Tempo	`internal_url = 'http://query_frontend_url:3200'` `.use_grpc = false` `.provider = 'tempo'`	`.internal_url = 'http://query_frontend_url:3200'` `.grpc_port: 9095` `.provider: 'tempo'` `.use_grpc = true (Not required: by default)`

Tempo tuning

Resources consumption

Grafana Tempo is a powerful tool, but it can lead to performance issues when not configured correctly. For example, the following configuration is not recommended and may lead to OOM issues for simple queries in the query-frontend component:

spec:
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m

These resources are shared between all the Tempo components. When needed, apply resources to each specific component, instead of applying the resources globally:

spec:
  template:
    queryFrontend:
      component:
        resources:
          limits:
            cpu: "2"
            memory: 2Gi

This Grafana Dashboard is available to measure the resources used in the tempo namespace.

Caching

Tempo offers multi-level caching that is used by default with Tanka and Helm deployment examples. It uses external cache, supporting Memcached and Redis. The lower level cache has a higher hit rate, and caches bloom filters and parquet data. The higher level caches frontend-search data.

Optimizing the cache depends on the application usage, and can be done modifying different parameters:

Connection limit for MemCached: Should be increased in large deployments, as MemCached is set to 1024 by default.
Cache size control: Should be increased when the working set is larger than the size of cache.

Tune search pipeline

There are many parameters to tune the search pipeline, some of these:

max_concurrent_queries: If it is too high it can cause OOM.
concurrent_jobs: How many jobs are done concurrently.
max_retries: When it is too high it can result in a lot of load.

Dedicated attribute columns

When using the vParquet3 storage format , defining dedicated attribute columns can improve the query performance. In order to best choose those columns (Up to 10), a good criteria is to choose attributes that contribute growing the block size (And not those commonly used).

Tempo authentication configuration

The Kiali CR provides authentication configuration that will be used also for querying the version check to provide information in the Mesh graph.

spec:
  external_services:
    tracing:
      enabled: true
      auth:
        insecure_skip_verify: false
        password: "pwd"
        token: ""
        type: "basic"
        use_kiali_token: false
        username: "user"
      health_check_url: ""

To configure a secret to be used as a password, see this FAQ entry.

TLS Certificate Configuration

If your Tempo server uses HTTPS with a certificate issued by a private CA, see the TLS Configuration page to learn how to configure Kiali to trust your CA.

Tempo cache

Kiali 2.2 includes a simple tracing cache for Tempo that stores the last N traces. By default, it is enabled and it keeps the last 200 traces. It can be modified in the Kiali CR with:

spec:
  external_services:
    tracing:
      enabled: true
      tempo_config:
        cache_enabled: true
        cache_capacity: 200

Kiali emits some cache metrics. The following query obtains the cache hit rate:

(sum(kiali_cache_hits_total{name="tempo"})/sum(kiali_cache_requests_total{name="tempo"})) * 100

Docs: Install Travel Demo

Mon, 01 Jan 0001 00:00:00 +0000

Deploy the Travel Demo

This demo application will deploy several services grouped into three namespaces.

Note that at this step we are going to deploy the application without any reference to Istio.

We will join services to the ServiceMesh in a following step.

To create and deploy the namespaces perform the following commands:

OpenShift

OpenShift users can substitute oc for kubectl. OpenShift users will need to add the necessary NetworkAttachmentDefinition to each namespace. Also, the necessary SecurityContextConstraints for the service accounts defined in the namespace (minimally, default).

kubectl create namespace travel-agency
kubectl create namespace travel-portal
kubectl create namespace travel-control

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_agency.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_portal.yaml) -n travel-portal
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_control.yaml) -n travel-control

Check that all deployments rolled out as expected:

$ kubectl get deployments -n travel-control
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
control   1/1     1            1           85s

$ kubectl get deployments -n travel-portal
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
travels   1/1     1            1           91s
viaggi    1/1     1            1           91s
voyages   1/1     1            1           91s

$ kubectl get deployments -n travel-agency
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
cars-v1         1/1     1            1           96s
discounts-v1    1/1     1            1           96s
flights-v1      1/1     1            1           96s
hotels-v1       1/1     1            1           96s
insurances-v1   1/1     1            1           96s
mysqldb-v1      1/1     1            1           96s
travels-v1      1/1     1            1           96s

Understanding the demo application

Travel Portal namespace

The Travel Demo application simulates two business domains organized in different namespaces.

In a first namespace called travel-portal there will be deployed several travel shops, where users can search for and book flights, hotels, cars or insurance.

The shop applications can behave differently based on request characteristics like channel (web or mobile) or user (new or existing).

These workloads may generate different types of traffic to imitate different real scenarios.

All the portals consume a service called travels deployed in the travel-agency namespace.

Travel Agency namespace

A second namespace called travel-agency will host a set of services created to provide quotes for travel.

A main travels service will be the business entry point for the travel agency. It receives a destination city and a user as parameters and it calculates all elements that compose a travel budget: airfare, lodging, car reservation and travel insurance.

Each service can provide an independent quote and the travels service must then aggregate them into a single response.

Additionally, some users, like registered users, can have access to special discounts, managed as well by an external service.

Service relations between namespaces can be described in the following diagram:

Travel Portal and Travel Agency flow

A typical flow consists of the following steps:

. A portal queries the travels service for available destinations. . Travels service queries the available hotels and returns to the portal shop. . A user selects a destination and a type of travel, which may include a flight and/or a car, hotel and insurance. . Cars, Hotels and Flights may have available discounts depending on user type.

Travel Control namespace

The travel-control namespace runs a business dashboard with two key features:

Allow setting changes for every travel shop simulator (traffic ratio, device, user and type of travel).
Provide a business view of the total requests generated from the travel-portal namespace to the travel-agency services, organized by business criteria as grouped per shop, per type of traffic and per city.

Docs: Kiali and Tempo setup

Mon, 01 Jan 0001 00:00:00 +0000

Steps to install Kiali and Grafana Tempo

We will start minikube:

minikube start

It is a requirement to have cert-manager installed:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml

Install the operator. It is important to download a version 3.0 or higher. In previous versions, the zipkin collector was not exposed, there was no way to change it as it was not defined in the CRD.

kubectl apply -f https://github.com/grafana/tempo-operator/releases/download/v0.3.0/tempo-operator.yaml

We will create the tempo namespace:

kubectl create namespace tempo

We will deploy minio, this is a sample minio.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. Will be used in deployment below.
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    - ReadWriteOnce
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the PVC created earlier
      volumes:
        - name: storage
          persistentVolumeClaim:
            # Name of the PVC created earlier
            claimName: minio-pv-claim
      initContainers:
        - name: create-buckets
          image: busybox:1.28
          command:
            - "sh"
            - "-c"
            - "mkdir -p /storage/tempo-data"
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
      containers:
        - name: minio
          # Pulls the default Minio image from Docker Hub
          image: minio/minio:latest
          args:
            - server
            - /storage
            - --console-address
            - ":9001"
          env:
            # Minio access key and secret key
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
            - containerPort: 9001
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: api
    - port: 9001
      targetPort: 9001
      protocol: TCP
      name: console
  selector:
    app: minio

And apply the yaml:

kubectl apply -n tempo -f minio.yaml

We will create a secret to access minio:

kubectl create secret generic -n tempo tempostack-dev-minio \
--from-literal=bucket="tempo-data" \
--from-literal=endpoint="http://minio:9000" \
--from-literal=access_key_id="minio" \
--from-literal=access_key_secret="minio123"

Install Grafana tempo with the operator. We will use the secret created in the previous step:

kubectl apply -n tempo -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: smm
spec:
  storageSize: 1Gi
  storage:
    secret:
      type: s3
      name: tempostack-dev-minio
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
        ingress:
          type: ingress
EOF

As an optional step, we can check if all the deployments have started correctly, and the services distributor has the port 9411 and the query frontend 16686:

kubectl get all -n tempo

(Optional) We can test if minio is working with a batch job to send some traces, in this case, to the open telemetry collector:

kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: tracegen
spec:
  template:
    spec:
      containers:
        - name: tracegen
          image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:latest
          command:
            - "./tracegen"
          args:
            - -otlp-endpoint=tempo-smm-distributor.tempo.svc.cluster.local:4317
            - -otlp-insecure
            - -duration=30s
            - -workers=1
      restartPolicy: Never
  backoffLimit: 4
EOF

And access the minio console:

kubectl port-forward --namespace istio-system service/minio 9001:9001

Install Istio with helm (Option I)

Istio can be installed with Helm following the instructions. The zipkin address needs to be set:

--set values.meshConfig.defaultConfig.tracing.zipkin.address="tempo-smm-distributor.tempo:9411"

And then, install Jaeger as Istio addon.

Install Istio using Kiali source code (Option II)

For development purposes, if we have Kiali source code, we can use the kiali hack scripts:

hack/istio/install-istio-via-istioctl.sh -c kubectl -a "prometheus grafana" -s values.meshConfig.defaultConfig.tracing.zipkin.address="tempo-smm-distributor.tempo:9411"

Install Kiali and bookinfo demo with some traffic generation

Install kiali:

helm install \
    --namespace istio-system \
    --set external_services.tracing.internal_url=http://tempo-smm-query-frontend.tempo:16685 \
    --set external_services.tracing.external_url=http://localhost:16686 \
    --set auth.strategy=anonymous \
    kiali-server \
    kiali/kiali-server

Install bookinfo with traffic generator

curl -L -o install-bookinfo.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/install-bookinfo-demo.sh
chmod +x install-bookinfo.sh
./install-bookinfo.sh -c kubectl -tg -id ${ISTIO_DIR}

And access Kiali:

kubectl port-forward svc/kiali 20001:20001 -n istio-system

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial is a walkthrough guide to install everything. For this reason, we will need:

minikube
istioctl
helm

This tutorial was tested on:

Minikube v1.30.1
Istio v1.18.1
Kiali v1.70

Clusters are provided by minikube instances, but this tutorial should work on on any Kubernetes environment.

We will set up some environment variables for the following commands:

CLUSTER_EAST="east"
CLUSTER_WEST="west"
ISTIO_DIR="absolute-path-to-istio-folder"

As Istio will be installed on more than one cluster and needs to communicate between clusters, we need to create certificates for the Istio installation. We will follow the Istio documentation related to certificates to achieve this:

mkdir -p certs
pushd certs

make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk root-ca

make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk $CLUSTER_EAST-cacerts
make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk $CLUSTER_WEST-cacerts

popd

The result is two certificates for then use when installing Istio in the future.

Docs: Deploy East cluster

Mon, 01 Jan 0001 00:00:00 +0000

Run the following commands to deploy the first cluster:

minikube start -p $CLUSTER_EAST --network istio --memory 8g --cpus 4

For both clusters, we need to configure MetalLB, which is a load balancer. This is because we need to assign an external IP to the required ingress gateways to enable cross cluster communication between Istio and the applications installed.

minikube addons enable metallb -p $CLUSTER_EAST

We set up some environment variables with IP ranges that MetalLB will then assign to the services:

MINIKUBE_IP=$(minikube ip -p $CLUSTER_EAST)
MINIKUBE_IP_NETWORK=$(echo $MINIKUBE_IP | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1/')
MINIKUBE_LB_RANGE="${MINIKUBE_IP_NETWORK}.20-${MINIKUBE_IP_NETWORK}.29"

cat <<EOF | kubectl --context $CLUSTER_EAST apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: [${MINIKUBE_LB_RANGE}]
EOF

We should have the first cluster deployed and ready to use.

Docs: First Steps

Mon, 01 Jan 0001 00:00:00 +0000

Missing Sidecars

The Travel Demo has been deployed in the previous step but without installing any Istio sidecar proxy.

In that case, the application won’t connect to the control plane and won’t take advantage of Istio’s features.

In Kiali, we will see the new namespaces in the overview page:

But we won’t see any traffic in the graph page for any of these new namespaces:

If we examine the Applications, Workloads or Services page, it will confirm that there are missing sidecars:

Enable Sidecars

In this tutorial, we will add namespaces and workloads into the ServiceMesh individually step by step.

This will help you to understand how Istio sidecar proxies work and how applications can use Istio’s features.

We are going to start with the control workload deployed into the travel-control namespace:

Step 1

Enable Auto Injection on the travel-control namespace

Step 2

Enable Auto Injection for control workload

Understanding what happened:

(i) Sidecar Injection

(ii) Automatic Sidecar Injection

Open Travel Demo to Outside Traffic

The control workload now has an Istio sidecar proxy injected but this application is not accessible from the outside.

In this step we are going to expose the control service using an Istio Ingress Gateway which will map a path to a route at the edge of the mesh.

Step 1

Create a DNS entry for the control service associated with the External IP of the Istio Ingress

There are multiple ways to create a DNS entry depending of the platform, servers or services that you are using. This step depends on the platform you have chosen, please review Determining the Ingress IP and Ports for more details.

Minikube

Kubernetes Service EXTERNAL-IP for “LoadBalancer” TYPE is provided in minikube plaform using the minikube tunnel tool.

For minikube we will check the External IP of the Ingress gateway:

$ kubectl get services/istio-ingressgateway -n istio-system
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   10.101.6.144   10.101.6.144   15021:30757/TCP,80:32647/TCP,443:30900/TCP,31400:30427/TCP,15443:31072/TCP   19h

And we will add a simple entry to the /etc/hosts of the tutorial machine with the desired DNS entry:

...
10.101.6.144 control.travel-control.istio-cluster.org
...

Then, from this machine, the url control.travel-control.istio-cluster.org will resolve to the External IP of the Ingress Gateway of Istio.

OpenShift

OpenShift does not provide the Kubernetes Service EXTERNAL-IP for “LoadBalancer” TYPE. Instead, you can expose the istio-ingressgateway service.

For OpenShift we will expose the Ingress gateway as a service:

$ oc expose service istio-ingressgateway -n istio-system
$ oc get routes -n istio-system
NAME                   HOST/PORT                                  PATH   SERVICES               PORT    TERMINATION          WILDCARD
istio-ingressgateway   <YOUR_ROUTE_HOST>                                 istio-ingressgateway   http2                        None

Then, from this machine, the host <YOUR_ROUTE_HOST> will resolve to the External IP of the Ingress Gateway of Istio. For OpenShift we will not define a DNS entry, instead, where you see control.travel-control.istio-cluster.org in the steps below, subsitute the value of <YOUR_ROUTE_HOST>.

Step 2

Use the Request Routing Wizard on the control service to generate a traffic rule

Use “Add Route Rule” button to add a default rule where any request will be routed to the control workload.

Use the Advanced Options and add a gateway with host control.travel-control.istio-cluster.org and create the Istio config.

Verify the Istio configuration generated.

Step 3

Test the control service by pointing your browser to \http://control.travel-control.istio-cluster.org

Step 4

Review travel-control namespace in Kiali

Understanding what happened:

External traffic enters into the cluster through a Gateway
Traffic is routed to the control service through a VirtualService
Kiali Graph visualizes the traffic telemetry reported from the control sidecar proxy
- Only the travel-control namespace is part of the mesh

(i) Istio Gateway

(ii) Istio Virtual Service

Docs: Install Istio on East cluster

Mon, 01 Jan 0001 00:00:00 +0000

The east cluster is the primary one, consequently is where the istiod process will be installed alongside other applications like Kiali.

Run the following commands to install Istio:

kubectl create namespace istio-system --context $CLUSTER_EAST

kubectl create secret generic cacerts -n istio-system --context $CLUSTER_EAST \
      --from-file=certs/$CLUSTER_EAST/ca-cert.pem \
      --from-file=certs/$CLUSTER_EAST/ca-key.pem \
      --from-file=certs/$CLUSTER_EAST/root-cert.pem \
      --from-file=certs/$CLUSTER_EAST/cert-chain.pem

kubectl --context=$CLUSTER_EAST label namespace istio-system topology.istio.io/network=network1

cat <<EOF > $CLUSTER_EAST.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: $CLUSTER_EAST
      network: network1
EOF

istioctl install -y --set values.pilot.env.EXTERNAL_ISTIOD=true --context=$CLUSTER_EAST -f $CLUSTER_EAST.yaml

After the installation, we need to create what we called an “east-west” gateway. It’s an ingress gateway just for the cross cluster configuration as we are opting to use the installation for different networks (this will be the case in the majority of the production scenarios).

$ISTIO_DIR/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster $CLUSTER_EAST --network network1 | \
    istioctl --context=$CLUSTER_EAST install -y -f -

Then, we need to expose the istiod service as well as the applications for the cross cluster communication:

kubectl apply --context=$CLUSTER_EAST -n istio-system -f \
    $ISTIO_DIR/samples/multicluster/expose-istiod.yaml

kubectl --context=$CLUSTER_EAST apply -n istio-system -f \
    $ISTIO_DIR/samples/multicluster/expose-services.yaml

export DISCOVERY_ADDRESS=$(kubectl \
    --context=$CLUSTER_EAST \
    -n istio-system get svc istio-eastwestgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Finally, we need to install Prometheus, which is important and required for Kiali to operate:

kubectl --context $CLUSTER_EAST -n istio-system apply -f $ISTIO_DIR/samples/addons/prometheus.yaml

Docs: Observe

Mon, 01 Jan 0001 00:00:00 +0000

Enable Sidecars in all workloads

An Istio sidecar proxy adds a workload into the mesh.

Proxies connect with the control plane and provide Service Mesh functionality.

Automatically providing metrics, logs and traces is a major feature of the sidecar.

In the previous steps we have added a sidecar only in the travel-control namespace’s control workload.

We have added new powerful features but the application is still missing visibility from other workloads.

Step 1

Switch to the Workload graph and select multiple namespaces to identify missing sidecars in the Travel Demo application

That control workload provides good visibility of its traffic, but telemetry is partially enabled, as travel-portal and travel-agency workloads don’t have sidecar proxies.

Step 2

Enable proxy injection in travel-portal and travel-agency namespaces

In the First Steps of this tutorial we didn’t inject the sidecar proxies on purpose to show a scenario where only some workloads may have sidecars.

Typically, Istio users annotate namespaces before the deployment to allow Istio to automatically add the sidecar when the application is rolled out into the cluster. Perform the following commands:

kubectl label namespace travel-agency istio-injection=enabled
kubectl label namespace travel-portal istio-injection=enabled

kubectl rollout restart deploy -n travel-portal
kubectl rollout restart deploy -n travel-agency

Verify that travel-control, travel-portal and travel-agency workloads have sidecars deployed:

Step 3

Verify updated telemetry for travel-portal and travel-agency namespaces

Graph walkthrough

The graph provides a powerful set of Graph Features to visualize the traffic topology of the service mesh.

In this step, we will show how to use the Graph to show relevant information in the context of the Travel Demo application.

Our goal will be to identify the most critical service of the demo application.

Step 1

Select all travel- namespaces in the graph and enable Traffic Distribution edge labels in the Display Options:

Review the status of the mesh, everything seems healthy, but also note that hotels service has more load compared to other services inlcuded in the travel-agency namespace.

Step 2

Select the hotels service, use the graph side-panel to select a trace from the Traces tab:

Combining telemetry and tracing information will show that there are traces started from a portal that involve multiple services but also other traces that only consume the hotels service.

Step 3

Select the main travels application and double-click to zoom in

The graph can focus on an element to study a particular context in detail. Note that a contextual menu is available using right-click, to easily shortcut the navigation to other sections.

Application details

Kiali provides Detail Views to navigate into applications, workloads and services.

These views provide information about the structure, health, metrics, logs, traces and Istio configuration for any application component.

In this tutorial we are going to learn how to use them to examine the main travels application of our example.

Step 1

Navigate to the travels application

An application is an abstract group of workloads and services labeled with the same “application” name.

From Service Mesh perspective this concept is significant as telemetry and tracing signals are mainly grouped by “application” even if multiple workloads are involved.

At this point of the tutorial, the travels application is quite simple, just a travels-v1 workload exposed through the travels service. Navigate to the travels-v1 workload detail by clicking the link in the travels application overview.

Step 2

Examine Outbound Metrics of travels-v1 workload

The Metrics tab provides a powerful visualization of telemetry collected by the Istio proxy sidecar. It presents a dashboard of charts, each of which can be expanded for closer inspection. Expand the Request volume chart:

Metrics Settings provides multiple predefined criteria out-of-the-box. Additionally, enable the spans checkbox to correlate metrics and tracing spans in a single chart.

We can see in the context of the Travels application, the hotels service request volume differs from that of the other travel-agency services.

By examining the Request Duration chart also shows that there is no suspicious delay, so probably this asymmetric volume is part of the application business’ logic.

Step 3

Review Logs of travels-v1 workload

The Logs tab provides a unified view of application container logs with the Istio sidecar proxy logs. It also offers a spans checkbox, providing a correlated view of both logs and tracing, helping identify spans of interest.

From the application container log we can spot that there are two main business methods: GetDestinations and GetTravelQuote.

In the Istio sidecar proxy log we see that GetDestinations invokes a GET /hotels request without parameters.

However, GetTravelQuote invokes multiple requests to other services using a specific city as a parameter.

Then, as discussed in the Travel Demo design, an initial query returns all available hotels before letting the user choose one and then get specific quotes for other destination services.

That scenario is shown in the increase of the hotels service utilization.

Step 4

Review Traces of workload-v1

Now we have identified that the hotels service has more use than other travel-agency services.

The next step is to get more context to answer if some particular service is acting slower than expected.

The Traces tab allows comparison between traces and metrics histograms, letting the user determine if a particular spike is expected in the context of average values.

In the same context, individual spans can be compared in more detail, helping to identify a problematic step in the broader scenario.

Docs: Connect

Mon, 01 Jan 0001 00:00:00 +0000

Request Routing

The Travel Demo application has several portals deployed on the travel-portal namespace consuming the travels service deployed on the travel-agency namespace.

The travels service is backed by a single workload called travels-v1 that receives requests from all portal workloads.

At a moment of the lifecycle the business needs of the portals may differ and new versions of the travels service may be necessary.

This step will show how to route requests dynamically to multiple versions of the travels service.

Step 1

Deploy travels-v2 and travels-v3 workloads

To deploy the new versions of the travels service execute the following commands:

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v2.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v3.yaml) -n travel-agency

As there is no specific routing defined, when there are multiple workloads for travels service the requests are uniformly distributed.

Step 2

Investigate the http headers used by the Travel Demo application

The Traffic Management features of Istio allow you to define Matching Conditions for dynamic request routing.

In our scenario we would like to perform the following routing logic:

All traffic from travels.uk routed to travels-v1
All traffic from viaggi.it routed to travels-v2
All traffic from voyages.fr routed to travels-v3

Portal workloads use HTTP/1.1 protocols to call the travels service, so one strategy could be to use the HTTP headers to define the matching condition.

But, where to find the HTTP headers ? That information typically belongs to the application domain and we should examine the code, documentation or dynamically trace a request to understand which headers are being used in this context.

There are multiple possibilities. The Travel Demo application uses an Istio Annotation feature to add an annotation into the Deployment descriptor, which adds additional Istio configuration into the proxy.

In our example the HTTP Headers are added as part of the trace context.

Then tracing will populate custom tags with the portal, device, user and travel used.

Step 3

Use the Request Routing Wizard on travels service to generate a traffic rule

We will define three “Request Matching” rules as part of this request routing. Define all three rules before clicking the Create button.

In the first rule, we will add a request match for when the portal header has the value of travels.uk.

Define the exact match, like below, and click the “Add Match” button to update the “Matching selected” for this rule.

Move to “Route To” tab and update the destination for this “Request Matching” rule. Then use the “Add Route Rule” to create the first rule.

Add similar rules to route traffic from viaggi.it to travels-v2 workload and from voyages.fr to travels-v3 workload.

When the three rules are defined you can use “Create” button to generate all Istio configurations needed for this scenario. Note that the rule ordering does not matter in this scenario.

The Istio config for a given service is found on the “Istio Config” card, on the Service Details page.

Step 4

Verify that the Request Routing is working from the travels-portal Graph

Once the Request Routing is working we can verify that outbound traffic from every portal goes to the single travels workload. To see this clearly use a “Workload Graph” for the “travel-portal” namespace, enable “Traffic Distribution” edge labels and disable the “Service Nodes” Display option:

Note that no distribution label on an edge implies 100% of traffic.

Examining the “Inbound Traffic” for any of the travels workloads will show a similar pattern in the telemetry.

Using a custom time range to select a large interval, we can see how the workload initially received traffic from all portals but then only a single portal after the Request Routing scenarios were defined.

Step 5

Update or delete Istio Configuration

Kiali Wizards allow you to define high level Service Mesh scenarios and will generate the Istio Configuration needed for its implementation (VirtualServices, DestinationRules, Gateways and PeerRequests). These scenarios can be updated or deleted from the “Actions” menu of a given service.

To experiment further you can navigate to the travels service and update your configuration by selecting “Request Routing”, as shown below. When you have finished experimenting with Routing Request scenarios then use the “Actions” menu to delete the generated Istio config.

Fault Injection

The Observe step has spotted that the hotels service has additional traffic compared with other services deployed in the travel-agency namespace.

Also, this service becomes critical in the main business logic. It is responsible for querying all available destinations, presenting them to the user, and getting a quote for the selected destination.

This also means that the hotels service may be one of the weakest points of the Travel Demo application.

This step will show how to test the resilience of the Travel Demo application by injecting faults into the hotels service and then observing how the application reacts to this scenario.

Step 1

Use the Fault Injection Wizard on hotels service to inject a delay

Select an HTTP Delay and specify the “Delay percentage” and “Fixed Delay” values. The default values will introduce a 5 seconds delay into 100% of received requests.

Step 2

Understanding source and destination metrics

Telemetry is collected from proxies and it is labeled with information about the source and destination workloads.

In our example, let’s say that travels service (“Service A” in the Istio diagram below) invokes the hotels service (“Service B” in the diagram). Travels is the “source” workload and hotels is the “destination” workload. The travels proxy will report telemetry from the source perspective and hotels proxy will report telemetry from the destination perspective. Let’s look at the latency reporting from both perspectives.

The travels workload proxy has the Fault Injection configuration so it will perform the call to the hotels service and will apply the delay on the travels workload side (this is reported as source telemetry).

We can see in the hotels telemetry reported by the source (the travels proxy) that there is a visible gap showing 5 second delay in the request duration.

But as the Fault Injection delay is applied on the source proxy (travels), the destination proxy (hotels) is unaffected and its destination telemetry show no delay.

Step 3

Study the impact of the travels service delay

The injected delay is propagated from the travels service to the downstream services deployed on travel-portal namespace, degrading the overall response time. But the downstream services are unaware, operate normally, and show a green status.

Step 4

Update or delete Istio Configuration

As part of this step you can update the Fault Injection scenario to test different delays. When finished, you can delete the generated Istio config for the hotels service.

Traffic Shifting

In the previous Request Routing step we have deployed two new versions of the travels service using the travels-v2 and travels-v3 workloads.

That scenario showed how Istio can route specific requests to specific workloads. It was configured such that each portal deployed in the travel-portal namespace (travels.uk, viaggi.it and voyages.fr) were routed to a specific travels workload (travels-v1, travels-v2 and travels-v3).

This Traffic Shifting step will simulate a new scenario: the new travels-v2 and travels-v3 workloads will represent new improvements for the travels service that will be used by all requests.

These new improvements implemented in travels-v2 and travels-v3 represent two alternative ways to address a specific problem. Our goal is to test them before deciding which one to use as a next version.

At the beginning we will send 80% of the traffic into the original travels-v1 workload, and will split 10% of the traffic each on travels-v2 and travels-v3.

Step 1

Use the Traffic Shifting Wizard on travels service

Create a scenario with 80% of the traffic distributed to travels-v1 workload and 10% of the traffic distributed each to travels-v2 and travels-v3.

Step 2

Examine Traffic Shifting distribution from the travels-agency Graph

Step 3

Compare travels workload and assess new changes proposed in travels-v2 and travels-v3

Istio Telemetry is grouped per logical application. That has the advantage of easily comparing different but related workloads, for one or more services.

In our example, we can use the “Inbound Metrics” and “Outbound Metrics” tabs in the travels application details, group by “Local version” and compare how travels-v2 and travels-v3 are working.

The charts show that the Traffic distribution is working accordingly and 80% is being distributed to travels-v1 workload and they also show no big differences between travels-v2 and travels-v3 in terms of request duration.

Step 4

Update or delete Istio Configuration

As part of this step you can update the Traffic Shifting scenario to test different distributions. When finished, you can delete the generated Istio config for the travels service.

TCP Traffic Shifting

The Travel Demo application has a database service used by several services deployed in the travel-agency namespace.

At some point in the lifecycle of the application the telemetry shows that the database service degrades and starts to increase the average response time.

This is a common situation. In this case, a database specialist suggests an update of the original indexes due to the data growth.

Our database specialist is suggesting two approaches and proposes to prepare two versions of the database service to test which may work better.

This step will show how the “Traffic Shifting” strategy can be applied to TCP services to test which new database indexing strategy works better.

Step 1

Deploy mysqldb-v2 and mysqldb-v3 workloads

To deploy the new versions of the mysqldb service execute the commands:

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/mysql-v2.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/mysql-v3.yaml) -n travel-agency

Step 2

Use the TCP Traffic Shifting Wizard on mysqldb service

Create a scenario with 80% of the traffic distributed to mysqldb-v1 workload and 10% of the traffic distributed each to mysqldb-v2 and mysqldb-v3.

Step 3

Examine Traffic Shifting distribution from the travels-agency Graph

Note that TCP telemetry has different types of metrics, as “Traffic Distribution” is only available for HTTP/gRPC services, for this service we need to use “Traffic Rate” to evaluate the distribution of data (bytes-per-second) between mysqldb workloads.

Step 4

Compare mysqldb workload and study new indexes proposed in mysqldb-v2 and mysqldb-v3

TCP services have different telemetry but it’s still grouped by versions, allowing the user to compare and study pattern differences for mysqldb-v2 and mysqldb-v3.

The charts show more peaks in mysqldb-v2 compared to mysqldb-v3 but overall a similar behavior, so it’s probably safe to choose either strategy to shift all traffic.

Step 5

Update or delete Istio Configuration

As part of this step you can update the TCP Traffic Shifting scenario to test a different distribution. When finished, you can delete the generated Istio config for the mysqldb service.

Request Timeouts

In the Fault Injection step we showed how we could introduce a delay in the critical hotels service and test the resilience of the application.

The delay was propagated across services and Kiali showed how services accepted the delay without creating errors on the system.

But in real scenarios delays may have important consequences. Services may prefer to fail sooner, and recover, rather than propagating a delay across services.

This step will show how to add a request timeout for one of the portals deployed in travel-portal namespace. The travel.uk and viaggi.it portals will accept delays but voyages.fr will timeout and fail.

Step 1

Use the Fault Injection Wizard on hotels service to inject a delay

Repeat the Fault Injection step to add delay on hotels service.

Step 2

Use the Request Routing Wizard on travels service to add a route rule with delay for voyages.fr

Add a rule to add a request timeout only on requests coming from voyages.fr portal:

Use the Request Matching tab to add a matching condition for the portal header with voyages.fr value.
Use the Request Timeouts tab to add an HTTP Timeout for this rule.
Add the rule to the scenario.

A first rule should be added to the list like:

Add a second rule to match any request and create the scenario. With this configuration, requests coming from voyages.fr will match the first rule and all others will match the second rule.

Step 3

Review the impact of the request timeout in the travels service

Create the rule. The Graph will show how requests coming from voyages.fr start to fail, due to the request timeout introduced.

Requests coming from other portals work without failures but are degraded by the hotels delay.

This scenario can be visualized in detail if we examine the “Inbound Metrics” and we group by “Remote app” and “Response code”.

As expected, the requests coming from voyages.fr don’t propagate the delay and they fail in the 2 seconds range, meanwhile requests from other portals don’t fail but they propagate the delay introduced in the hotels service.

Step 4

Update or delete Istio Configuration

As part of this step you can update the scenarios defined around hotels and travels services to experiment with more conditions, or you can delete the generated Istio config in both services.

Circuit Breaking

Distributed systems will benefit from failing quickly and applying back pressure, as opposed to propagating delays and errors through the system.

Circuit breaking is an important technique used to limit the impact of failures, latency spikes, and other types of network problems.

This step will show how to apply a Circuit Breaker into the travels service in order to limit the number of concurrent requests and connections.

Step 1

Deploy a new loadtester portal in the travel-portal namespace

In this example we are going to deploy a new workload that will simulate an important increase in the load of the system.

OpenShift

OpenShift users may need to also add the associated loadtester serviceaccount to the necessary securitycontextcontraints.

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

The loadtester workload will try to create 50 concurrent connections to the travels service, adding considerable pressure to the travels-agency namespace.

The Travel Demo application is capable of handling this load and in a first look it doesn’t show unhealthy status.

But in a real scenario an unexpected increase in the load of a service like this may have a significant impact in the overall system status.

Step 2

Use the Traffic Shifting Wizard on travels service to generate a traffic rule

Use the “Traffic Shifting” Wizard to distribute traffic (evenly) to the travels workloads and use the “Advanced Options” to add a “Circuit Breaker” to the scenario.

The “Connection Pool” settings will indicate that the proxy sidecar will reject requests when the number of concurrent connections and requests exceeds more than one.

The “Outlier Detection” will eject a host from the connection pool if there is more than one consecutive error.

Step 3

Study the behavior of the Circuit Breaker in the travels service

In the loadtester versioned-app Graph we can see that the travels service’s Circuit Breaker accepts some, but fails most, connections.

Remember, that these connections are stopped by the proxy on the loadtester side. That “fail sooner” pattern prevents overloading the network.

Using the Graph we can select the failed edge, check the Flags tab, and see that those requests are closed by the Circuit breaker.

If we examine the “Request volume” metric from the “Outbound Metrics” tab we can see the evolution of the requests, and how the introduction of the Circuit Breaker made the proxy reduce the request volume.

Step 4

Update or delete Istio Configuration

As part of this step you can update the scenarios defined around the travels service to experiment with more Circuit Breaker settings, or you can delete the generated Istio config in the service.

Understanding what happened:

(i) Circuit Breaking

(ii) Outlier Detection

(iii) Connection Pool Settings

(iv) Envoy’s Circuit breaking Architecture

Mirroring

This tutorial has shown several scenarios where Istio can route traffic to different versions in order to compare versions and evaluate which one works best.

The Traffic Shifting step was focused on travels service adding a new travels-v2 and travels-v3 workloads and the TCP Traffic Shifting showed how this scenario can be used on TCP services like mysqldb service.

Mirroring (or shadowing) is a particular case of the Traffic Shifting scenario where the proxy sends a copy of live traffic to a mirrored service.

The mirrored traffic happens out of band of the primary request path. It allows for testing of alternate services, in production environments, with minimal risk.

Istio mirrored traffic is only supported for HTTP/gRPC protocols.

This step will show how to apply mirrored traffic into the travels service.

Step 1

Use the Traffic Shifting Wizard on travels service

We will simulate the following:

travels-v1 is the original traffic and it will keep 80% of the traffic
travels-v2 is the new version to deploy, it’s being evaluated and it will get 20% of the traffic to compare against travels-v1
But travels-v3 will be considered as a new, experimental version for testing outside of the regular request path. It will be defined as a mirrored workload on 50% of the original requests.

Step 2

Examine Traffic Shifting distribution from the travels-agency Graph

Note that Istio does not report mirrored traffic telemetry from the source proxy. It is reported from the destination proxy, although it is not flagged as mirrored, and therefore an edge from travels to the travels-v3 workload will appear in the graph. Note the traffic rates reflect the expected ratio of 80/20 between travels-v1 and travels-v2, with travels-v3 at about half of that total.

This can be examined better using the “Source” and “Destination” metrics from the “Inbound Metrics” tab.

The “Source” proxy, in this case the proxies injected into the workloads of travel-portal namespace, won’t report telemetry for travels-v3 mirrored workload.

But the “Destination” proxy, in this case the proxy injected in the travels-v3 workload, will collect the telemetry from the mirrored traffic.

Step 3

Update or delete Istio Configuration

As part of this step you can update the Mirroring scenario to test different mirrored distributions.

When finished you can delete the generated Istio config for the travels service.

Docs: Install Kiali

Mon, 01 Jan 0001 00:00:00 +0000

Run the following command to install Kiali using the Kiali operator:

kubectl config use-context $CLUSTER_EAST

helm upgrade --install --namespace istio-system --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server

Verify that Kiali is running with the following command:

istioctl dashboard kiali

There are other alternatives to expose Kiali or other Addons in Istio. Check Remotely Accessing Telemetry Addons for more information.

Docs: Install Travels on East cluster

Mon, 01 Jan 0001 00:00:00 +0000

Run the following commands to install Travels application on east cluster:

kubectl create namespace travel-agency --context $CLUSTER_EAST
kubectl create namespace travel-portal --context $CLUSTER_EAST
kubectl create namespace travel-control --context $CLUSTER_EAST

kubectl label namespace travel-agency istio-injection=enabled --context $CLUSTER_EAST
kubectl label namespace travel-portal istio-injection=enabled --context $CLUSTER_EAST
kubectl label namespace travel-control istio-injection=enabled --context $CLUSTER_EAST

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_agency.yaml) -n travel-agency --context $CLUSTER_EAST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_portal.yaml) -n travel-portal --context $CLUSTER_EAST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_control.yaml) -n travel-control --context $CLUSTER_EAST

After the installation, we can see that the Travels application is running on the east cluster:

It is important to note that Kiali only observes one istio-system namespace as we did not configure it for multicluster yet.

Go to the Graph page and select the three namespaces related to the Travels demo in the namespace dropdown menu. This shows you the in-cluster traffic:

So far, we installed everything on one cluster, similarly to the Travels tutorial for a single cluster.

Now we will expand this topology to include a remote cluster. As we commented this situation can be very common in a production scenario, either because we might want to split some applications into different clusters, generally because they are maintained by different developers or for high availability or just making applications available in other zones to reduce latencies.

Docs: Secure

Mon, 01 Jan 0001 00:00:00 +0000

Authorization Policies and Sidecars

Security is one of the main pillars of Istio features.

The Istio Security High Level Architecture provides a comprehensive solution to design and implement multiple security scenarios.

In this tutorial we will show how Kiali can use telemetry information to create security policies for the workloads deployed in a given namespace.

Istio telemetry aggregates the ServiceAccount information used in the workloads communication. This information can be used to define authorization policies that deny and allow actions on future live traffic communication status.

Additionally, Istio sidecars can be created to limit the hosts with which a given workload can communicate. This improves traffic control, and also reduces the memory footprint of the proxies.

This step will show how we can define authorization policies for the travel-agency namespace, in the Travel Demo application, for all existing traffic in a given time period.

Once authorization policies are defined, a new workload will be rejected if it doesn’t match the security rules defined.

Step 1

Undeploy the loadtester workload from travel-portal namespace

In this example we will use the loadtester workload as the “intruder” in our security rules.

If we have followed the previous tutorial steps, we need to undeploy it from the system.

kubectl delete -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

We should validate that telemetry has updated the travel-portal namespace and “Security” can be enabled in the Graph Display options.

Step 2

Create Authorization Policies, and Istio Sidecars, for current traffic for travel-agency namespace

Every workload in the cluster uses a Service Account.

travels.uk, viaggi.it and voyages.fr workloads use the default cluster.local/ns/travel-portal/sa/default ServiceAccount defined automatically per namespace.

This information is propagated into the Istio Telemetry and Kiali can use it to define a set of AuthorizationPolicy rules, and Istio Sidecars.

The Sidecars restrict the list of hosts with which each workload can communicate, based on the current traffic.

The “Create Traffic Policies” action, located in the Overview page, will create these definitions.

This will generate a main DENY ALL rule to protect the whole namespace, and an individual ALLOW rule per workload identified in the telemetry.

It will create also an individual Sidecar per workload, each of them containing the set of hosts.

As an example, we can see that for the travels-v1 workload, the following list of hosts are added to the sidecar.

Step 3

Deploy the loadtester portal in the travel-portal namespace

If the loadtester workload uses a different ServiceAccount then, when it’s deployed, it won’t comply with the AuthorizationPolicy rules defined in the previous step.

OpenShift

OpenShift users may need to also add the associated loadtester serviceaccount to the necessary securitycontextcontraints.

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

Now, travels workload will reject requests made by loadtester workload and that situation will be reflected in Graph:

This can also be verified in the details page using the Outbound Metrics tab grouped by response code (only the 403 line is present).

Inspecting the Logs tab confirms that loadtester workload is getting a HTTP 403 Forbidden response from travels workloads, as expected.

Step 4

Update travels-v1 AuthorizationPolicy to allow loadtester ServiceAccount

AuthorizationPolicy resources are defined per workload using matching selectors.

As part of the example, we can show how a ServiceAccount can be added into an existing rule to allow traffic from loadtester workload into the travels-v1 workload only.

As expected, now we can see that travels-v1 workload accepts requests from all travel-portal namespace workloads, but travels-v2 and travels-v3 continue rejecting requests from loadtester source.

Using “Outbound Metrics” tab from the loadtester workload we can group per “Remote version” and “Response code” to get a detailed view of this AuthorizationPolicy change.

Step 5

Verify the proxies clusters list is limited by the Sidecars

According to Istio Sidecar documentation, Istio configures all mesh sidecar proxies to reach every mesh workload. After the sidecars are created, the list of hosts is reduced according to the current traffic. To verify this, we can look for the clusters configured in each proxy.

As an example, looking into the cars-v1 workload, we can see that there is a reduced number of clusters with which the proxy can communicate.

Step 6

Update or delete Istio Configuration

As part of this step, you can update the AuthorizationPolicies and Istio Sidecars generated for the travel-agency namespace, and experiment with more security rules. Or, you can delete the generated Istio config for the namespace.

Docs: Deploy West cluster

Mon, 01 Jan 0001 00:00:00 +0000

Run the following commands to deploy the second cluster:

minikube start -p $CLUSTER_WEST --network istio --memory 8g --cpus 4

Similar to the east cluster, we configure MetalLB:

minikube addons enable metallb -p $CLUSTER_WEST

MINIKUBE_IP=$(minikube ip -p $CLUSTER_WEST)
MINIKUBE_IP_NETWORK=$(echo $MINIKUBE_IP | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1/')
MINIKUBE_LB_RANGE="${MINIKUBE_IP_NETWORK}.30-${MINIKUBE_IP_NETWORK}.39"

cat <<EOF | kubectl --context $CLUSTER_WEST apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: [${MINIKUBE_LB_RANGE}]
EOF

Docs: Install Istio on West cluster

Mon, 01 Jan 0001 00:00:00 +0000

This installation will be different as this cluster will be a remote. In a remote cluster, it won’t be an Istio control plane. Istio will install some resources that allows the primary control plane to configure the workloads in the remote cluster like injecting the sidecars and configuring the low level routing.

kubectl create namespace istio-system --context $CLUSTER_WEST

kubectl create secret generic cacerts -n istio-system --context $CLUSTER_WEST \
      --from-file=certs/$CLUSTER_WEST/ca-cert.pem \
      --from-file=certs/$CLUSTER_WEST/ca-key.pem \
      --from-file=certs/$CLUSTER_WEST/root-cert.pem \
      --from-file=certs/$CLUSTER_WEST/cert-chain.pem

kubectl --context=$CLUSTER_WEST annotate namespace istio-system topology.istio.io/controlPlaneClusters=$CLUSTER_EAST
kubectl --context=$CLUSTER_WEST label namespace istio-system topology.istio.io/network=network2

cat <<EOF > $CLUSTER_WEST.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: remote
  values:
    istiodRemote:
      injectionPath: /inject/cluster/$CLUSTER_WEST/net/network2
    global:
      remotePilotAddress: ${DISCOVERY_ADDRESS}
EOF

istioctl install -y --context=$CLUSTER_WEST -f $CLUSTER_WEST.yaml

We will also install a Prometheus instance on the remote. We will federate both Prometheus, with the east’s one being the place where all metrics will be gathered together:

kubectl apply -f $ISTIO_DIR/samples/addons/prometheus.yaml --context $CLUSTER_WEST

An important step is to create a secret on east cluster allowing it to fetch information of the remote cluster:

istioctl x create-remote-secret \
    --context=$CLUSTER_WEST \
    --name=$CLUSTER_WEST | \
    kubectl apply -f - --context=$CLUSTER_EAST

Finally, we create the east-west gateway

$ISTIO_DIR/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster $CLUSTER_WEST --network network2 | \
    istioctl --context=$CLUSTER_WEST install -y -f -

Prometheus federation

Kiali requires unified metrics from a single Prometheus endpoint for all clusters, even in a multi-cluster environment. In this tutorial, we will federate the two Prometheus instances, meaning that all the remote’s metrics should be fetched by the main Prometheus.

We will configure east’s Prometheus to fetch west’s metrics:

kubectl patch svc prometheus -n istio-system --context $CLUSTER_WEST -p "{\"spec\": {\"type\": \"LoadBalancer\"}}"

WEST_PROMETHEUS_ADDRESS=$(kubectl --context=$CLUSTER_WEST -n istio-system get svc prometheus -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl -L -o prometheus.yaml https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/prometheus.yaml

sed -i "s/WEST_PROMETHEUS_ADDRESS/$WEST_PROMETHEUS_ADDRESS/g" prometheus.yaml

kubectl --context=$CLUSTER_EAST apply -f prometheus.yaml -n istio-system

Docs: Configure Kiali for multicluster

Mon, 01 Jan 0001 00:00:00 +0000

We will configure Kiali to access the remote cluster. This will require a secret (similar to the Istio secret) containing the credentials for Kiali to fetch information from the remote cluster:

curl -L -o kiali-prepare-remote-cluster.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/kiali-prepare-remote-cluster.sh

chmod +x kiali-prepare-remote-cluster.sh

./kiali-prepare-remote-cluster.sh --kiali-cluster-context $CLUSTER_EAST --remote-cluster-context $CLUSTER_WEST

Finally, upgrade the installation for Kiali to pick up the secret:

kubectl config use-context $CLUSTER_EAST

helm upgrade --install --namespace istio-system --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server

As result, we can quickly see that a new namespace appear in the Overview, the istio-system namespace from west cluster:

Docs: Anonymous strategy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The anonymous strategy removes any authentication requirement. Users will have access to Kiali without providing any credentials.

Although the anonymous strategy doesn’t provide any access protection, it’s valid for some use-cases. Some examples known from the community:

Exposing Kiali through a reverse proxy, where the reverse proxy is providing a custom authentication mechanism.
Exposing Kiali on an already limited network of trusted users.
When Kiali is accessed through kubectl port-forward or alike commands that allow usage of the cluster’s RBAC capabilities to limit access.
When developing Kiali, where a developer has a private instance on his own machine.

It’s worth to emphasize that the anonymous strategy will leave Kiali unsecured. If you are using this option, make sure that Kiali is available only to trusted users, or access is protected by other means.

Set-up

To use the anonymous strategy, use the following configuration in the Kiali CR:

spec:
  auth:
    strategy: anonymous

The anonymous strategy doesn’t have any additional configuration.

Access control

When using the anonymous strategy, the content displayed in Kiali is based on the permissions of the Kiali service account. By default, the Kiali service account has cluster wide access and will be able to display everything in the cluster.

OpenShift

If you are running Kiali in OpenShift, access can be customized by changing privileges to the Kiali ServiceAccount. For example, to reduce permissions to individual namespaces, first, remove the cluster-wide permissions granted by default:

  oc delete clusterrolebindings kiali

Then grant the kiali role only in needed namespaces. For example:

  oc adm policy add-role-to-user kiali system:serviceaccount:istio-system:kiali-service-account -n ${NAMESPACE}

View only

You can tell the Kiali Operator to install Kiali in “view only” mode (this does work for either OpenShift or Kubernetes). You do this by setting the view_only_mode to true in the Kiali CR, which allows Kiali to read service mesh resources found in the cluster, but it does not allow any change:

spec:
  deployment:
    view_only_mode: true

Docs: How to Contribute

Mon, 01 Jan 0001 00:00:00 +0000

Contributing to the Docs

To contribute to the kiali.io docs see kiali.io on github.

In short you will:

Fork the kiali.io repo on GitHub.
Make your changes and send a pull request (PR).
If you’re not yet ready for a review, add “WIP” to the PR name to indicate it’s a work in progress.
1. Don’t add the Hugo property “draft = true” to the page front matter, it prevents auto-deployment of the content preview.
Wait for the automated PR workflow to do some checks. When it’s ready, you should see a comment like this: deploy/netlify — Deploy preview ready!
Click Details to the right of “Deploy preview ready” to see a preview of your updates.
Continue updating your doc and pushing your changes until you’re happy with the content.

Updating a single page

If you’ve just spotted something you’d like to change while using the docs, there is a shortcut for you:

Click Edit this page in the top right hand corner of the page.
If you don’t already have an up to date fork of the project repo, you are prompted to get one:
1. Click Fork this repository and propose changes or Update your Fork to get an up to date version of the project to edit.
2. The appropriate page in your fork is displayed in edit mode.
Follow the steps above to make, preview, and propose your changes.

Creating an issue

If you’ve found a problem in the docs, but you’re not sure how to fix it yourself, please create an issue in the kiali.io repo. You can also create an issue about a specific page by clicking the Create Documentation Issue button in the top right hand corner of the page.

Contributing to the Code

For code contribution see the kiali project’s CONTRIBUTING page.

Docs: Install Travels on West cluster

Mon, 01 Jan 0001 00:00:00 +0000

We are going to deploy two new services just to distribute traffic on the new cluster. These services are travels v2 and v3:

kubectl create ns travel-agency --context $CLUSTER_WEST

kubectl label namespace travel-agency istio-injection=enabled --context $CLUSTER_WEST

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v2.yaml) -n travel-agency --context $CLUSTER_WEST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v3.yaml) -n travel-agency --context $CLUSTER_WEST

cat <<EOF | kubectl -n travel-agency --context $CLUSTER_WEST apply -f -
apiVersion: v1
kind: Service
metadata:
  name: travels
  labels:
    app: travels
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: travels
---
apiVersion: v1
kind: Service
metadata:
  name: insurances
  labels:
    app: insurances
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: insurances
---
apiVersion: v1
kind: Service
metadata:
  name: hotels
  labels:
    app: hotels
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: hotels
---
apiVersion: v1
kind: Service
metadata:
  name: flights
  labels:
    app: flights
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: flights
---
apiVersion: v1
kind: Service
metadata:
  name: discounts
  labels:
    app: discounts
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: discounts
---
apiVersion: v1
kind: Service
metadata:
  name: cars
  labels:
    app: cars
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: cars
EOF

After the installation, we can see that traffic is flowing to the remote cluster too:

This is happening automatically, Istio balances the traffic to both services. The key thing to notice here is that there is a concept called namespace sameness in Istio that is very important when planning our multicluster setup.

In both clusters, we can see that we have the same namespaces. They are called the same in both. Also, we can see that the services in both clusters need to exist and be called the same.

When we created the west’s namespaces, they are called the same, and also notice that even if we do not have instances of insurances or cars, we created the services. This is because travel services from the cluster will try to communicate with these services, not caring at all if the applications are in the west or east cluster. Istio will handle the routing in the back.

From this moment, we can start playing with Kiali to introduce some scenarios previously seen in the Travels tutorial.

Docs: Kiali Backstage Plugin

Mon, 01 Jan 0001 00:00:00 +0000

The Kiali Backstage Plugin provides information about each Service Mesh object related with an entity in Backstage, a framework for building developer portals.

The plugin has different views to be included in Backstage:

Added as cards to see resource lists
Added as a tab which includes all the predefined Kiali cards
Added as a page, which views are not going to be filtered by entities and offer a full Kiali view

The Kiali Backstage plugin is released as a technology preview in Red Hat Developer Hub.

Documentation

Get Involved

Development guide

Docs: Kiali Chatbot

Mon, 01 Jan 0001 00:00:00 +0000

Kiali Chatbot is Kiali’s built-in AI assistant in the Kiali UI. It lets you ask questions about your service mesh and get answers backed by live data from Kiali and its configured backends (Prometheus, tracing, Kubernetes, etc.).

It does not require an external MCP server. Kiali includes its own set of MCP-style tools internally, so the AI can call them without depending on a separate MCP deployment.

Status

The Kiali chatbot was first released in Kiali version 2.22 and it is in Dev preview.

How does it work

At a high level:

The Kiali UI sends your chat request (prompt + context + selected model) to the Kiali backend.
Kiali selects the configured provider/model from chat_ai.
The provider calls the LLM with a set of internal MCP tools (defined in Kiali under kiali/ai/mcp).
The LLM may request tool calls (e.g. mesh graph, traces, resource details, workload logs, Istio config operations).
Kiali executes those tool calls against Kiali/Kubernetes/Prometheus/tracing backends and returns the final answer, including optional UI navigation actions and documentation citations.

For configuration keys (enable/disable, providers/models, store), see the chat_ai section in the Kiali CR spec.

Tool schemas (inputs/outputs)

Kiali Chatbot uses internal tools with defined input schemas and structured outputs.

Configuring the Kiali Chatbot

The Kiali Chatbot is disabled by default. To enable it, set chat_ai.enabled: true. When enabled, you will see the chatbot icon in the Kiali UI:

You must also configure at least one provider and model (including an API key), and pick a default provider/model.

Switching model providers

Kiali Chatbot providers and models are configured in chat_ai:

Providers: OpenAI-compatible (type: openai) and Google (type: google).
Models are selected by name (per-provider) and can be enabled/disabled.
API keys can be set inline (not recommended) or via secret:<secret-name>:<key-in-secret>.

Example configuration (showing an OpenAI-compatible provider using Gemini via OpenAI endpoint):

chat_ai:
  enabled: true
  default_provider: "openai"
  providers:
    - name: "openai"
      enabled: true
      description: "OpenAI API Provider"
      type: "openai"
      config: "default"
      default_model: "gemini"
      models:
        - name: "gemini"
          enabled: true
          model: "gemini-2.5-pro"
          description: "Model provided by Google with OpenAI API Support"
          endpoint: "https://generativelanguage.googleapis.com/v1beta/openai"
          key: "secret:my-key-secret:openai-gemini"

You can also select the configured models and providers in the chatbot window:

What you can ask

Examples of tasks that work well:

Mesh/namespace topology and summaries (graph, status)
Basic observability questions (metrics, traces)
Troubleshooting workflows (get logs for a workload, identify failing namespaces)

Example prompts

“Show me the mesh graph for namespace bookinfo.”
“Which workloads in istio-system look unhealthy and why?”
“Get traces for service productpage in bookinfo for the last 30m.”

Next step

If you want to use an AI assistant outside the Kiali UI (for example, in an IDE), see Kiali MCP.

Docs: OpenShift

Mon, 01 Jan 0001 00:00:00 +0000

These are specific notes for the External Kiali deployment model on OpenShift.

Installation

It is highly recommended that the Kiali Operator be deployed on all clusters, even if the Kiali Server itself is not deployed on some clusters. This will ensure that the proper namespace and remote cluster resources can be created. Clusters without a Kiali Server will require only the remote cluster resources necessary for remote Kiali Server authentication. To install these resources, configure the Kiali CR with:

spec.deployment.remote_cluster_resources_only: true

This Kiali CR will result in an installation requiring very limited resources.

Authorization Strategy

When using the openshift authentication strategy on OpenShift, make sure to read and apply any guidance found in the notes for multi-cluster.

Docs: OSSM Console

Mon, 01 Jan 0001 00:00:00 +0000

OpenShift Service Mesh Console (OSSMC) is a Kiali integration for OpenShift Console based on OpenShift dynamic plugins technology. It integrates part of the Kiali UI functionality into the OpenShift Console, providing visibility of the Service Mesh.

OSSMC was first released in September 2022 as a developer preview. It has since be released GA in October 2023.

Documentation

Get Involved

Releases

Release list

Docs: OSSMC User Guide

Mon, 01 Jan 0001 00:00:00 +0000

The OpenShift Service Mesh Console (OSSMC) is an extension to the OpenShift Console which provides visibility into your Service Mesh. With the OSSMC plugin installed, a new Service Mesh menu category is available in the navigation menu on the left side of the web console, as well as new Service Mesh tabs that enhance the existing Workloads and Services OpenShift console detail pages.

The features of the OSSMC plugin are the same as those of the standalone Kiali Console, but the pages are organized differently to better integrate with the OpenShift console. The OSSMC plugin does not replace the Kiali Console, and after installing the OSSMC plugin, you can still access the standalone Kiali Console. This User Guide, however, will discuss the extensions you see from within the OpenShift Console itself.

The OSSMC only supports a single tenant today. Whether that tenant is configured to access only a subset of OpenShift projects or has access cluster-wide to all projects does not matter, however, only a single tenant can be accessed.

Overview

The Overview page provides a summary of your mesh by showing cards representing the namespaces participating in the mesh. Each namespace card has summary metric graphs and additional health details. There are links in the cards that take you to other pages within OSSMC.

Traffic Graph

The Traffic Graph page provides the full topology view of your mesh. The mesh is represented by nodes and edges - each node representing a component of the mesh and each edge representing traffic flowing through the mesh between components.

Istio Config

The Istio Config page provides a list of all Istio configuration files in your mesh with a column that provides a quick way to know if the configuration for each resource is valid.

Mesh

The Mesh page provides detailed information about the Istio infrastructure status. It shows an infrastructure topology view with core and add-on components, their health, and how they are connected to each other.

Workload

The Workloads view has a tab Service Mesh that provides a lot of mesh-related detail for the selected workload. The details are grouped into several sub-tabs: Overview, Traffic, Logs, Inbound Metrics, Outbound Metrics, Traces, and Envoy.

Workload: Overview

The Workload: Overview sub-tab provides a summary of the selected workload including a localized topology graph showing the workload with all inbound and outbound edges and nodes.

Workload: Traffic

The Workload: Traffic sub-tab provides information about all inbound and outbound traffic to the workload.

Workload: Logs

The Workload: Logs sub-tab provides the logs for the workload’s containers. You can view container logs individually or in a unified fashion, ordered by log time. This is especially helpful to see how the Envoy sidecar proxy logs relate to your workload’s application logs. You can enable the tracing span integration which then allows you to see which logs correspond to trace spans.

Workload: Metrics

You can see both inbound and outbound metric graphs in the corresponding sub-tabs. All the workload metrics can be displayed here, providing you with a detail view of the performance of your workload. You can enable the tracing span integration which allows you to see which spans occurred at the same time as the metrics. You can then click on a span marker in the graph to view the specific spans associated with that timeframe.

Workload: Traces

The Traces sub-tab provides a chart showing the trace spans collected over the given timeframe. Click on a bubble to drill down into those trace spans; the trace spans can provide you the most low-level detail within your workload application, down to the individual request level.

The trace details view will give further details, including heatmaps that provide you with a comparison of one span in relation to other requests and spans in the same timeframe.

If you hover over a cell in a heatmap, a tooltip will give some details on the cell data:

When the OpenShift tracing UI plugin is enabled, Kiali will try to auto discover the plugin settings and the View in Tracing Kiali link will redirect to the plugin (for Kiali 2.8.0+). If the plugin config needs to be adjusted, the following settings should be included in the plugin-conf ConfigMap:

{
  ...
  "observability": {
    "instance": "sample",
    "namespace": "tempo",
    "tenant": "default"
  }
}

Workload: Envoy

The Envoy sub-tab provides information about the Envoy sidecar configuration. This is useful when you need to dig down deep into the sidecar configuration when debugging things such as connectivity issues.

Services

The Services view has a tab Service Mesh that provides mesh-related detail for the selected service. The details are grouped into several sub-tabs: Overview, Traffic, Inbound Metrics, Traces. These sub-tabs are similar in nature as the Workload sub-tabs with the same names and serve the same functions.

Projects

The Projects view has a tab Service Mesh that provides traffic graph information about that project. It is the same information shown in the Traffic Graph page but specific to that project.

Docs: TLS Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Overview

When Kiali connects to external services (Prometheus, Grafana, Jaeger/Tempo, Perses) over HTTPS, it needs to verify the TLS certificates presented by those services. By default, Kiali trusts the system certificate authorities (CAs) that are built into the container image.

If your external services use certificates issued by a private CA (such as an internal corporate CA, a service mesh CA, or self-signed certificates), you need to configure Kiali to trust those additional CAs.

Adding Custom Certificate Authorities

Kiali uses a global CA bundle mechanism to trust additional certificate authorities. All custom CAs are added to a single certificate pool that applies to all HTTPS connections Kiali makes to external services.

On Kubernetes

To add custom CAs, create a ConfigMap named <kiali-instance-name>-cabundle in the Kiali namespace. The default instance name is kiali, so the ConfigMap would be named kiali-cabundle:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system  # Or your Kiali namespace
data:
  additional-ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgIQAqxcJmoLQ...
    ... (your CA certificate) ...
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIDyTCCArGgAwIBAgIRAJ4K...
    ... (additional CA certificates if needed) ...
    -----END CERTIFICATE-----

Key name: The key must be additional-ca-bundle.pem. You can include multiple CA certificates in PEM format in the same file.

Alternative keys: You can also use openid-server-ca.crt or (on OpenShift) oauth-server-ca.crt as key names. While these names suggest specific purposes, all CAs are loaded into Kiali’s global certificate pool and trusted for all TLS connections. Using additional-ca-bundle.pem is recommended for clarity.

For OpenShift OAuth authentication: On OpenShift, you can alternatively create a separate ConfigMap named <instance-name>-oauth-cabundle with the key oauth-server-ca.crt. See the OpenShift authentication documentation for details. However, adding your CA to kiali-cabundle under additional-ca-bundle.pem achieves the same result.

On OpenShift

On OpenShift, the Kiali Operator automatically creates a ConfigMap named <kiali-instance-name>-cabundle-openshift (e.g., kiali-cabundle-openshift) with the annotation service.beta.openshift.io/inject-cabundle: "true". This tells OpenShift to automatically inject the cluster’s service CA into the ConfigMap.

This means that by default, Kiali on OpenShift already trusts:

The system CAs
The OpenShift service CA (used by services with serving certificates)

If you need to add additional CAs beyond the OpenShift service CA, create a separate ConfigMap named <kiali-instance-name>-cabundle (e.g., kiali-cabundle):

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system  # Or your Kiali namespace
data:
  additional-ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgIQAqxcJmoLQ...
    ... (your CA certificate) ...
    -----END CERTIFICATE-----

The operator uses a projected volume that automatically combines both ConfigMaps, so your custom CAs work alongside the OpenShift service CA.

How It Works

When Kiali starts, it loads certificates from:

System certificate pool: The default trusted CAs from the container’s operating system
Additional CA bundle: Certificates from /kiali-cabundle/additional-ca-bundle.pem (if present)
OpenShift service CA (OpenShift only): Certificates from /kiali-cabundle/service-ca.crt (automatically injected from the <instance-name>-cabundle-openshift ConfigMap)
OpenID server CA (OpenID auth only): Certificates from /kiali-cabundle/openid-server-ca.crt (if present)
OAuth CA bundle (OpenShift with OAuth auth): Certificates from /kiali-cabundle/oauth-server-ca.crt (if the <instance-name>-oauth-cabundle ConfigMap exists)

All these certificates are combined into a single certificate pool used for all HTTPS connections to external services.

On OpenShift: The operator uses a projected volume that automatically combines multiple ConfigMap sources (<instance-name>-cabundle-openshift, <instance-name>-cabundle, and <instance-name>-oauth-cabundle) into the /kiali-cabundle mount path. This means you don’t need to manually merge ConfigMaps - each ConfigMap can be managed independently.

Automatic refresh: Kiali watches CA bundle files for changes using filesystem notifications (fsnotify) and automatically refreshes the certificate pool without requiring a pod restart. When you update the ConfigMap, Kubernetes propagates the changes to the mounted volume based on the kubelet’s sync interval (default: 60 seconds). Once the files are updated on disk, Kiali detects and applies them immediately. Total propagation time is typically 0-90 seconds after the ConfigMap update.

Skipping Certificate Verification

If you need to temporarily skip certificate verification (for testing purposes only), you can set insecure_skip_verify: true in the authentication configuration for each external service:

spec:
  external_services:
    prometheus:
      auth:
        insecure_skip_verify: true
    grafana:
      auth:
        insecure_skip_verify: true
    tracing:
      auth:
        insecure_skip_verify: true

Security warning: Disabling certificate verification makes Kiali vulnerable to man-in-the-middle attacks. Only use this option for testing purposes, never in production.

Common Scenarios

Internal Corporate CA

If your organization has an internal CA that issues certificates for internal services:

Obtain the root CA certificate (public part only) from your security team
Create the ConfigMap with the CA certificate as shown above

Self-Signed Certificates

For development or testing environments using self-signed certificates:

Export the certificate from your service (usually the same certificate that was generated)
Create the ConfigMap with that certificate

Istio Service Mesh mTLS

If your external services are part of the Istio service mesh and use Istio’s mTLS:

Kiali typically accesses these services through their Kubernetes service names, which may bypass the sidecar
If you need to go through the mesh, you may need to add Istio’s root CA to the bundle

cert-manager Issued Certificates

If you use cert-manager with a private CA:

The CA certificate is typically stored in a Secret (e.g., my-ca-secret with key ca.crt)
Extract the CA and add it to the ConfigMap:

kubectl get secret my-ca-secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl create configmap kiali-cabundle -n istio-system --from-file=additional-ca-bundle.pem=ca.crt

Troubleshooting

Certificate Errors in Logs

If you see errors like x509: certificate signed by unknown authority in Kiali logs:

Verify the ConfigMap exists and has the correct name
Check that the key is exactly additional-ca-bundle.pem
Ensure the certificate is in valid PEM format
Verify the CA certificate is the correct one (the root or intermediate CA that signed the service’s certificate)

Verifying the ConfigMap is Mounted

Check that the ConfigMap is properly mounted in the Kiali pod:

kubectl exec -n istio-system deploy/kiali -- ls -la /kiali-cabundle/

You should see your CA bundle file listed.

Testing Certificate Chain

To verify your CA certificate is correct, you can test it outside of Kiali:

# Get the server's certificate chain
openssl s_client -connect prometheus.istio-system:9090 -showcerts

# Verify against your CA
openssl verify -CAfile your-ca.pem server-cert.pem

Docs: Kiali Chatbot tools (schemas)

Mon, 01 Jan 0001 00:00:00 +0000

Kiali Chatbot uses internal MCP-style tools (implemented inside Kiali) to fetch live data and perform safe actions. These are not external MCP server tools.

The tool input schemas are defined in Kiali under kiali/ai/mcp/tools/*.yaml. The tool outputs are JSON structures returned by the Kiali backend and consumed by the model and/or UI.

Tool list

get_action_ui: returns UI navigation actions (buttons/links).
get_citations: returns documentation links relevant to the user query.
get_mesh_graph: returns mesh health/topology summaries (and supporting raw payloads).
get_resource_detail: returns service/workload details or lists (same payload shapes as existing Kiali APIs).
get_pod_performance: returns usage vs requests/limits summary (CPU/memory).
get_traces: returns a compact trace summary (bottlenecks/errors).
get_logs: returns workload/pod logs with optional filtering.
manage_istio_config: list/get/create/patch/delete Istio objects (with a confirmation gate for sensitive actions).

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Istio

Before you install Kiali you must have already installed Istio along with its telemetry storage addon (e.g. Prometheus). You might also consider installing Istio’s optional tracing addon (e.g. Tempo) and optional Grafana addon but those are not required by Kiali. Refer to the Istio documentation for details.

Optionally Enable the Debug Interface

Like istioctl, Kiali can make use of Istio’s port 8080 “Debug Interface” API. Despite the naming, this is required for accessing the status of the proxies.

The ENABLE_DEBUG_ON_HTTP setting controls the relevant API access. Istio suggests to disable this for security, but Kiali requires ENABLE_DEBUG_ON_HTTP=true, which is the default.

If you prefer not to enable the Istio API then certain Kiali features will be unavailable. If disabled, set spec.external_services.istio.istio_api_enabled: false in the Kiali CR.

For more information, see the Istio documentation.

Version Compatibility

It is always recommended that users run a supported version of Istio. The Istio news page posts end-of-support (EOL) dates. Supported Kiali versions include only the Kiali versions associated with supported Istio versions.

Starting with Kiali v2.4, each Kiali release is tested against the currently supported Istio releases. Unless otherwise noted, a Kiali release will be compatible with those releases. Older, untested Istio versions may also be compatible. Known incompatibilities will be noted in the table below. Prior to Kiali v2.4, compatibility is guaranteed only against the latest Istio release at the time. Although compatibility may be fine with other versions.

Istio	Tested Kiali Versions	Notes
1.28	2.17 and higher
1.27	2.12 and higher
1.26	2.9 and higher
1.25	2.5-2.16	Istio 1.25 is out of support.
1.24	2.0-2.13	Istio 1.24 is out of support.
1.23	1.87, 2.4-2.8	Istio 1.23 is out of support. Kiali v2 requires migration from Kiali v1 non-default namespace management (i.e. accessible_namespaces) to Discovery Selectors.
1.22	1.87, 2.4-2.5	Istio 1.22 is out of support. Kiali v1.86 is the recommended minimum for Istio Ambient users. Starting with Kiali v1.86,.1 Istio v1.22 is required.
1.21	1.81	Istio 1.21 is out of support.
1.20	1.78	Istio 1.20 is out of support.
1.19	1.75	Istio 1.19 is out of support.
1.18	1.73	Istio 1.18 is out of support.
1.17	1.66	Istio 1.17 is out of support. Avoid 1.63.0,1.63.1 due to a regression.
1.16	1.63	Istio 1.16 is out of support. Avoid 1.62.0,1.63.0,1.63.1 due to a regression.
1.15	1.59	Istio 1.15 is out of support.
1.14	1.54	Istio 1.14 is out of support.
1.13	1.49	Istio 1.13 is out of support.
1.12	1.44	Istio 1.12 is out of support.
1.11	1.41	Istio 1.11 is out of support.
1.10	1.37	Istio 1.10 is out of support.
1.9	1.33	Istio 1.9 is out of support.
1.8	1.28	Istio 1.8 is out of support. It removes all support for mixer/telemetry V1, as does Kiali 1.26.0. Use earlier versions of Kiali for mixer.
1.7	1.25	Istio 1.7 is out of support. Istioctl no longer installs Kiali. Use the Istio samples/addons for quick demo installs.
1.6	1.21	Istio 1.6 is out of support. Kiali 1.17 is recommended for Istio < 1.6.

OpenShift Service Mesh Version Compatibility

OpenShift

If you are running Red Hat OpenShift Service Mesh (OSSM), use only the bundled, supported version of Kiali.

OSSM	Kiali	Notes
3.2	2.17
3.1	2.11
3.0	2.4
2.6	1.73
2.5	1.73	OSSM 2.5 is out of support
2.4	1.65	OSSM 2.4 is out of support
2.3	1.57	OSSM 2.3 is out of support
2.2	1.48	OSSM 2.2 is out of support

OpenShift Console Plugin (OSSMC) Version Compatibility

Kiali server with the same version of OSSMC plugin must be installed previously in your OpenShift cluster.

OpenShift	OSSMC Min	OSSMC Max	Notes
4.19+	2.20
4.15+	1.84	2.19
4.12-4.18	1.73	1.83	All OSSMC versions from v1.73 to v1.83 are only compatible with Kiali server v1.73

Maistra Version Compatibility

Maistra	SMCP CR	Kiali	Notes
2.6	2.6	1.73	Using Maistra 2.6 to install service mesh control plane 2.6 requires Kiali Operator v1.73. Other versions are not compatible.
2.6	2.5	1.73	Using Maistra 2.6 to install service mesh control plane 2.5 requires Kiali Operator v1.73. Other versions are not compatible.
2.6	2.4	1.65	Using Maistra 2.6 to install service mesh control plane 2.4 requires Kiali Operator v1.73. Other versions are not compatible.
2.5	2.5	1.73	Using Maistra 2.5 to install service mesh control plane 2.5 requires Kiali Operator v1.73. Other versions are not compatible.
2.5	2.4	1.65	Using Maistra 2.5 to install service mesh control plane 2.4 requires Kiali Operator v1.73. Other versions are not compatible.
2.4	2.4	1.65	Using Maistra 2.4 to install service mesh control plane 2.4 requires Kiali Operator v1.65. Other versions are not compatible.
n/a	2.3	n/a	Service mesh control plane 2.3 is out of support.
n/a	2.2	n/a	Service mesh control plane 2.2 is out of support.
n/a	2.1	n/a	Service mesh control plane 2.1 is out of support.
n/a	2.0	n/a	Service mesh control plane 2.0 is out of support.
n/a	1.1	n/a	Service mesh control plane 1.1 is out of support.
n/a	1.0	n/a	Service mesh control plane 1.0 is out of support.

Browser Compatibility

Kiali requires a modern web browser and supports the last two versions of Chrome, Firefox, Safari or Edge.

Hardware Requirements

Any machine capable of running a Kubernetes based cluster should also be able to run Kiali.

However, Kiali tends to grow in resource usage as your cluster grows. Usually the more namespaces and workloads you have in your cluster, the more memory you will need to allocate to Kiali.

Platform-specific requirements

OpenShift

If you are installing on OpenShift, you must grant the cluster-admin role to the user that is installing Kiali. If OpenShift is installed locally on the machine you are using, the following command should log you in as user system:admin which has this cluster-admin role:

$ oc login -u system:admin

For most commands listed on this documentation, the Kubernetes CLI command kubectl is used to interact with the cluster environment. On OpenShift you can simply replace kubectl with oc, unless otherwise noted.

Google Cloud Private Cluster

Private clusters on Google Cloud have network restrictions. Kiali needs your cluster’s firewall to allow access from the Kubernetes API to the Istio Control Plane namespace, for both the 8080 and 15000 ports.

To review the master access firewall rule:

gcloud compute firewall-rules list --filter="name~gke-${CLUSTER_NAME}-[0-9a-z]*-master"

To replace the existing rule and allow master access:

gcloud compute firewall-rules update <firewall-rule-name> --allow <previous-ports>,tcp:8080,tcp:15000

Istio deployments on private clusters also need extra ports to be opened. Check the Istio installation page for GKE to see all the extra installation steps for this platform.

Docs: ACM Observability

Mon, 01 Jan 0001 00:00:00 +0000

OpenShift Only: This guide is specifically for Red Hat OpenShift environments using Red Hat Advanced Cluster Management (ACM) for Kubernetes. ACM is an OpenShift-specific product.

Overview

Red Hat Advanced Cluster Management (ACM) provides centralized observability for multi-cluster OpenShift environments through its Observability Service. When ACM Observability is enabled, metrics from all managed clusters (including the hub cluster itself) are collected and aggregated into a central Thanos-based storage system.

Kiali can query these aggregated metrics either through ACM’s external Observatorium API (using mTLS authentication) or directly through internal Thanos services. This guide explains both options, with detailed steps for the Observatorium API approach.

Architecture

Components

On the Hub Cluster:

ACM Observability Service: Centralized observability platform
- Observatorium API: External HTTPS endpoint with mTLS authentication
- Thanos: Metrics storage and query engine (Query, Query Frontend, Receive, Store)

On Managed Clusters (Hub + Spokes):

User Workload Monitoring (UWM): OpenShift’s Prometheus for user workloads
PodMonitor/ServiceMonitor: Scrape Istio metrics from:
- Sidecar proxies (in application namespaces)
- Control plane (istiod in istio-system)
- Ztunnel (in ztunnel namespace, for L4 metrics in Ambient mode)
- Waypoint proxies (in application namespaces, for L7 metrics in Ambient mode)
Metrics Allowlist ConfigMaps: Define which metrics ACM should collect
Metrics Collector: Runs on each managed cluster and pushes its Prometheus metrics to the hub cluster’s Thanos every 5 minutes (default)

Kiali Deployment Location:

Kiali can be deployed on any cluster with network access to:

The hub cluster’s metrics backend (Observatorium API or internal Thanos services)
Each managed cluster’s Kubernetes API (for workload and configuration data)

Common deployment locations:

Hub cluster (recommended): Co-located with ACM for lower latency metric queries and simplified networking. Can use internal Thanos services (HTTP) or external Observatorium API (HTTPS). Typically requires external deployment mode (ignore_home_cluster: true) since the hub usually doesn’t run mesh workloads or an Istio control plane.
Spoke/managed cluster: Kiali deployed alongside the mesh workloads or the Istio control plane. Must use external Observatorium API route.
Separate management cluster: Kiali deployed externally in dedicated “external deployment” mode (see External Kiali). Must use external Observatorium API route.

This guide assumes Kiali is deployed on the hub cluster in external deployment mode, but the configuration applies to any deployment location.

Metrics Flow

There are two independent flows:

Ingestion (managed cluster → hub):

Istio data plane components (sidecars, ztunnel, or waypoint proxies) expose metrics at :15020/stats/prometheus.
User Workload Monitoring Prometheus scrapes those metrics (typically every 30s).
The ACM observability collector/agent on the managed cluster reads from Prometheus and ships metrics to the hub (typically every 5 minutes).
The hub stores them in Thanos Receive/Store and serves them through Thanos Query Frontend.

Query (Kiali → hub):

Kiali can query metrics through either of these paths:

Via Observatorium API Route (HTTPS with mTLS):

Kiali queries the external Observatorium API route.
Observatorium forwards the request to Thanos Query Frontend.
Thanos Query Frontend reads from Thanos Store/Receive and returns the result back through Observatorium to Kiali.

Via Internal Thanos Service (HTTP):

Kiali queries the internal Thanos Query Frontend service directly within the cluster, bypassing Observatorium.

Expected Latency: 5-6 minutes from traffic generation to visibility in Kiali due to the 5-minute (default) push interval.

Prerequisites

1. ACM Observability Service

ACM MultiClusterObservability must be installed on the hub cluster:

# Verify ACM Observability is running
oc get mco observability

# Check Observatorium API route
oc get route observatorium-api -n open-cluster-management-observability

2. User Workload Monitoring

User Workload Monitoring must be enabled on all clusters (hub and spokes):

# Enable UWM by editing cluster-monitoring-config
oc -n openshift-monitoring edit configmap cluster-monitoring-config

# Add:
# data:
#   config.yaml: |
#     enableUserWorkload: true

# Verify UWM pods are running
oc get pods -n openshift-user-workload-monitoring

See: Enabling monitoring for user-defined projects

3. Istio Metrics Collection

Create ServiceMonitor and PodMonitor resources to collect Istio metrics. The PodMonitor for sidecars must be created in each namespace with Istio sidecars because OpenShift monitoring ignores namespaceSelector in these resources. The ServiceMonitor for istiod is created once in istio-system.

ServiceMonitor for istiod (in istio-system):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s

PodMonitor for Istio proxies (must be applied in every mesh namespace):

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor
  namespace: <your-mesh-namespace>
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_container_name"]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
    - action: replace
      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[$2]:$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - action: replace
      regex: (\d+);((([0-9]+?)(\.|$)){4})
      replacement: '$2:$1'
      sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
      targetLabel: "__address__"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "${1}${2}"
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "${1}${2}"
    - sourceLabels: ["__meta_kubernetes_namespace"]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "<your-mesh-identification-string>"
      targetLabel: mesh_id

See: Configuring OpenShift Monitoring with Service Mesh

Ambient Mode Metrics

If you are using Istio’s Ambient mode instead of (or in addition to) sidecar mode, you need additional PodMonitors to collect metrics from the Ambient data plane components.

Understanding Ambient Mode Metrics

Ambient mode uses a layered architecture with two metric sources:

Ztunnel (L4 metrics only)

Runs as a DaemonSet (namespace varies by installation)
Handles all L4 traffic for pods enrolled in ambient mode
Produces TCP-level metrics:
- istio_tcp_sent_bytes_total
- istio_tcp_received_bytes_total
- istio_tcp_connections_opened_total
- istio_tcp_connections_closed_total
Does not produce HTTP metrics

Waypoint proxies (L7 metrics)

Run as Deployments in application namespaces
Optional L7 proxies deployed per-namespace or per-service
Produce full HTTP metrics (same as sidecars):
- istio_requests_total
- istio_request_duration_milliseconds_*
- istio_request_bytes_*
- istio_response_bytes_*
- Plus all TCP metrics listed above

If you only use ztunnel (no waypoints), Kiali will show TCP traffic but not HTTP-level details like response codes or latency histograms.

PodMonitor for Ztunnel

Create a PodMonitor in the namespace where ztunnel runs. Ztunnel pods expose metrics using the same interface as sidecars:

Container name: istio-proxy
Annotation: prometheus.io/scrape: "true"
Metrics path: /stats/prometheus on port 15020

Because ztunnel uses the same metrics interface, you can use the same PodMonitor configuration shown in the Istio Metrics Collection section above, changing only the namespace field to match your ztunnel namespace.

Note: The ztunnel namespace location depends on your Istio installation method. Verify your ztunnel namespace with: oc get pods -l app=ztunnel -A

PodMonitor for Waypoint Proxies

Create a PodMonitor in each namespace with a waypoint. Waypoint pods also expose metrics using the same interface as sidecars:

Container name: istio-proxy
Annotation: prometheus.io/scrape: "true"
Metrics path: /stats/prometheus on port 15020

Because waypoints use the same metrics interface, you can use the same PodMonitor configuration shown in the Istio Metrics Collection section above.

4. Metrics Allowlist Configuration

ACM only collects metrics that are explicitly allowlisted. For Istio metrics to be collected, create a ConfigMap named observability-metrics-custom-allowlist in the source namespace (see note below) with key uwl_metrics_list.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: observability-metrics-custom-allowlist
  namespace: <your-mesh-namespace>
data:
  uwl_metrics_list.yaml: |
    names:
    # Core Istio metrics below. For additional metrics that Kiali uses,
    # see: https://kiali.io/docs/faq/general/#requiredmetrics
    #
    # L7 (HTTP) metrics - from sidecars and waypoint proxies
    - istio_requests_total
    - istio_request_duration_milliseconds_bucket
    - istio_request_duration_milliseconds_sum
    - istio_request_duration_milliseconds_count
    - istio_request_bytes_bucket
    - istio_request_bytes_sum
    - istio_request_bytes_count
    - istio_response_bytes_bucket
    - istio_response_bytes_sum
    - istio_response_bytes_count
    # L4 (TCP) metrics - from sidecars, waypoint proxies, AND ztunnel
    - istio_tcp_sent_bytes_total
    - istio_tcp_received_bytes_total
    - istio_tcp_connections_opened_total
    - istio_tcp_connections_closed_total

Critical: The ConfigMap must be in the source namespace where metrics originate (e.g., istio-system, application namespaces), NOT in open-cluster-management-observability.

Ambient Mode: The same allowlist works for all Istio data plane components. However, ztunnel only produces TCP metrics (istio_tcp_*), so HTTP metrics in the allowlist will have no data from ztunnel. Waypoints produce both TCP and HTTP metrics, same as sidecars. Create the allowlist ConfigMap in each namespace where you have a PodMonitor, including the namespace where ztunnel runs and any namespaces with waypoint proxies.

See: Adding user workload metrics

Configuring Kiali for ACM Observability

Choosing Between Observatorium API and Internal Thanos Services

You have two options for connecting Kiali to ACM metrics:

Option 1: Observatorium API Route (HTTPS with mTLS)

external_services:
  prometheus:
    url: "https://observatorium-api-<namespace>.<apps-domain>/api/metrics/v1/default"
    auth:
      type: none
      cert_file: "secret:acm-observability-certs:tls.crt"
      key_file: "secret:acm-observability-certs:tls.key"

Provides:

HTTPS with mTLS authentication and encryption
External access (can be accessed from outside the cluster if needed)
RBAC enforcement via Observatorium
Multi-tenant isolation
Requires certificate setup

Option 2: Internal Thanos Service (HTTP)

external_services:
  prometheus:
    url: "http://observability-thanos-query-frontend.open-cluster-management-observability.svc:9090"
    auth:
      type: none

Provides:

Simpler setup (no certificates required)
Direct access to Thanos (potentially lower latency)
Internal cluster networking only
HTTP only (no encryption between Kiali and Thanos)

Recommendation: Use the Observatorium API for production environments where you want encrypted connections and proper authentication. Use internal services for development/testing environments where simplicity is preferred or where network security is already provided by the cluster infrastructure.

The rest of this guide focuses on the Observatorium API approach with mTLS authentication.

Step 1: Obtain mTLS Certificates from ACM

ACM automatically creates long-lived client certificates (1 year validity) for accessing the Observatorium API. Extract these from the hub cluster:

# Extract client certificate (for authentication)
oc get secret observability-grafana-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt

# Extract client key (for authentication)
oc get secret observability-grafana-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key

Note: These certificates are created automatically when ACM MultiClusterObservability is deployed and are already trusted by the Observatorium API.

ACM Version Note: Secret names may vary depending on your ACM version. Before proceeding, verify the secret exists:

oc get secrets -n open-cluster-management-observability | grep -i cert

If observability-grafana-certs doesn’t exist, look for similar secrets containing client certificates.

Step 2: Extract Server CA Certificate

Extract the CA certificate that signed the Observatorium API server certificate. This is used by Kiali to validate the server’s TLS certificate.

First, identify which CA issued the server certificate:

# Get the Observatorium API route hostname
HOST=$(oc get route observatorium-api -n open-cluster-management-observability -o jsonpath='{.spec.host}')

# Check who issued the server certificate
echo | openssl s_client -connect "${HOST}:443" -servername "${HOST}" -showcerts 2>/dev/null | openssl x509 -noout -issuer

Example output:

issuer=C=US, O=Red Hat, Inc., CN=observability-server-ca-certificate

Then, extract the matching CA certificate based on the issuer CN:

If the issuer CN is observability-server-ca-certificate:

oc get secret observability-server-ca-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > server-ca.crt

If the issuer CN is observability-client-ca-certificate:

oc get secret observability-client-ca-certs \
  -n open-cluster-management-observability \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > server-ca.crt

Note: Both secrets are in the open-cluster-management-observability namespace. The exact CA used may vary depending on your ACM version and configuration.

Step 3: Create Kubernetes Resources

Note: <kiali-namespace> and ${KIALI_NAMESPACE} are used as a placeholder for the namespace where Kiali is deployed. This is commonly istio-system but is not required to be - replace with your actual Kiali namespace.

Create the mTLS certificate secret in Kiali’s namespace:

KIALI_NAMESPACE="istio-system"  # Replace with your Kiali namespace

oc create secret generic acm-observability-certs \
  -n ${KIALI_NAMESPACE} \
  --from-file=tls.crt=tls.crt \
  --from-file=tls.key=tls.key

Create the CA bundle ConfigMap in Kiali’s namespace:

oc create configmap kiali-cabundle \
  -n ${KIALI_NAMESPACE} \
  --from-file=additional-ca-bundle.pem=server-ca.crt

On OpenShift: The Kiali Operator (or Helm chart) automatically creates a separate ConfigMap named kiali-cabundle-openshift for the OpenShift service CA, then uses a projected volume to combine it with your custom kiali-cabundle ConfigMap. You only need to create/manage kiali-cabundle with your ACM CA - the system handles merging.

For more details about CA bundle configuration, see TLS Configuration.

Step 4: Get Observatorium API URL

Find the external Observatorium API route URL:

oc get route observatorium-api \
  -n open-cluster-management-observability \
  -o jsonpath='https://{.spec.host}/api/metrics/v1/default'

The URL format is: https://observatorium-api-<namespace>.<apps-domain>/api/metrics/v1/default

Step 5: Configure Kiali

Using Kiali Operator (Kiali CR):

spec:
  external_services:
    prometheus:
      # Use Observatorium API route
      url: "<observatorium-api-url>"

      auth:
        type: none  # mTLS authentication at TLS layer, no Authorization header
        cert_file: "secret:acm-observability-certs:tls.crt"
        key_file: "secret:acm-observability-certs:tls.key"

      # Enable Thanos proxy mode
      thanos_proxy:
        enabled: true
        retention_period: "14d"
        scrape_interval: "5m"

Using Server Helm Chart:

OBSERVATORIUM_API_URL="$(oc get route observatorium-api -n open-cluster-management-observability -o jsonpath='https://{.spec.host}/api/metrics/v1/default')"

helm install kiali kiali-server \
  --namespace ${KIALI_NAMESPACE} \
  --set external_services.prometheus.url="${OBSERVATORIUM_API_URL}" \
  --set external_services.prometheus.auth.type="none" \
  --set external_services.prometheus.auth.cert_file="secret:acm-observability-certs:tls.crt" \
  --set external_services.prometheus.auth.key_file="secret:acm-observability-certs:tls.key" \
  --set external_services.prometheus.thanos_proxy.enabled="true" \
  --set external_services.prometheus.thanos_proxy.retention_period="14d" \
  --set external_services.prometheus.thanos_proxy.scrape_interval="5m"

Important Configuration Notes

Metrics Latency

ACM collects metrics from each cluster’s Prometheus and pushes to Thanos every 5 minutes (default). This means, by default, there is a 5-6 minute delay before new metrics appear in Kiali. This latency is inherent to ACM’s architecture and applies to all managed clusters.

Note: This interval is configurable via the spec.observabilityAddonSpec.interval field (in seconds) in the MultiClusterObservability CR on the hub cluster.

Initial warm-up period: After deploying a new application, it takes approximately twice the collection interval before data appears in Kiali’s graph and metrics tab. This is because Kiali uses PromQL rate() functions which require at least two data points to compute a result, and with ACM’s collection interval, two data points take at least two collection cycles to accumulate. For example, with the default 5-minute interval, expect a ~10-minute warm-up period. After this initial warm-up, all time ranges in Kiali should display data normally. However, keep in mind that the most recent data visible in Kiali will always be at least one collection interval old, since metrics must complete a full collection cycle before they appear in Thanos.

Thanos Proxy Mode

Enable thanos_proxy when using ACM/Thanos:

external_services:
  prometheus:
    thanos_proxy:
      enabled: true
      retention_period: "14d"  # Should match your ACM Thanos retention
      scrape_interval: "5m"   # Must match ACM's metrics collection interval

When enabled: true, Kiali uses the configured scrape_interval and retention_period values directly, rather than querying Prometheus’s /api/v1/status/config and /api/v1/status/runtimeinfo endpoints to discover them. This is necessary because Thanos does not expose these Prometheus configuration endpoints.

Why these values matter:

scrape_interval: Kiali’s UI uses this value to compute PromQL rate() intervals and query step sizes. The rate interval must be large enough to contain at least two data points for rate() to produce results. With ACM, data points arrive in Thanos at the ACM collection interval (default 5 minutes), not at the local Prometheus scrape interval (typically 15-30 seconds). If scrape_interval is set too low (e.g., “30s”), the computed rate windows will be too narrow to capture two ACM data points, causing Kiali’s metrics tab to show empty charts even though data exists in Thanos.

Critical: Set scrape_interval to match the ACM metrics collection interval (default "5m"), not the local Prometheus scrape interval. The ACM collection interval is configured via spec.observabilityAddonSpec.interval in the MultiClusterObservability CR on the hub cluster. If you have customized this value, set scrape_interval to match.

retention_period: Used to limit time range queries to available data. ACM defaults to 365d retention when spec.advanced.retentionConfig is not explicitly configured in the MultiClusterObservability CR. If using the default, set retention_period to “365d”. If configuring custom retention, use at least 10d minimum (a Thanos requirement for downsampling to function). Always match retention_period to your actual ACM retention configuration. The “14d” value shown in examples here is used for demonstration.

Multi-Cluster Setup

For multi-cluster service mesh deployments with ACM:

1. Metrics Aggregation (Handled by ACM)

ACM automatically aggregates metrics from all managed clusters. Each cluster’s metrics include a cluster label with the cluster name (the metadata.name of the ManagedCluster resource). To get a list of all the clusters managed by ACM, run oc get managedcluster on the hub cluster.

Kiali can filter metrics by cluster using query_scope. The query_scope configuration adds label filters to every Prometheus query:

external_services:
  prometheus:
    # Example 1: Filter to a single cluster
    query_scope:
      cluster: "east-cluster"

    # Example 2: Filter by mesh_id and cluster
    query_scope:
      mesh_id: "mesh-1"
      cluster: "east-cluster"

Each key-value pair in query_scope is added as key="value" to every query. For example, cluster: "east-cluster" adds cluster="east-cluster" to all PromQL queries.

2. Remote Cluster Access (For Workload/Config Data)

While metrics come from ACM’s central Thanos, Kiali still needs direct API access to each cluster for:

Workload and service discovery
Istio configuration validation
Kubernetes resource details

Create remote cluster secrets as described in the multi-cluster setup guide.

3. External Deployment Model

For multi-cluster with ACM, if you deploy Kiali on the hub cluster (or on a separate management cluster), you will typically want to run Kiali in external deployment mode:

clustering:
  ignore_home_cluster: true  # Kiali is external to mesh

kubernetes_config:
  cluster_name: "<management-cluster-name>"  # Unique name for the cluster where Kiali runs

See the External Kiali guide for complete external deployment instructions.

Certificate Management

Automatic Rotation

ACM-issued certificates (stored in the observability-grafana-certs secret in the ACM observability namespace) have 1-year validity and are automatically rotated by ACM before expiration. When certificates are rotated:

ACM updates the observability-grafana-certs secret in open-cluster-management-observability namespace
You must update the acm-observability-certs secret in Kiali’s namespace with the new certificate data. Options include:
- Re-run the extraction commands from Step 1: Obtain mTLS Certificates from ACM manually
- Use an ACM ConfigurationPolicy with hub cluster templating to automatically distribute and update the secret to the cluster where Kiali runs (see ACM Governance documentation for details)
Kubernetes updates the mounted files in Kiali pod (within 60 seconds after the secret update)
Kiali automatically uses new certificates on next connection (no pod restart needed)

Using Custom Certificates

If you prefer to use your own certificate infrastructure instead of ACM’s certificates:

Generate/obtain certificates signed by a CA trusted by ACM Observatorium API
Configure ACM to trust your CA (consult ACM documentation)
Create the acm-observability-certs secret with your certificates

Verification

Check Certificate Configuration

# Verify secret exists
oc get secret acm-observability-certs -n ${KIALI_NAMESPACE}

# Check certificate expiration
oc get secret acm-observability-certs -n ${KIALI_NAMESPACE} \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -enddate

# Verify CA bundle
oc get configmap kiali-cabundle -n ${KIALI_NAMESPACE} \
  -o jsonpath='{.data.additional-ca-bundle\.pem}' | \
  openssl x509 -noout -subject

Check Kiali Logs

Verify certificates are loaded successfully:

oc logs -n ${KIALI_NAMESPACE} deployment/kiali | grep -i "credential\|certificate"

# Expected output (at "info" log level):
# INF Loaded [1] valid CA certificate(s) from [/kiali-cabundle/additional-ca-bundle.pem]
#
# Additional output (at "debug" log level):
# DBG Credential file path configured: [/kiali-override-secrets/prometheus-cert/tls.crt]
# DBG Credential file path configured: [/kiali-override-secrets/prometheus-key/tls.key]

Test Metrics

Generate mesh traffic in one of your managed clusters
Wait for the initial warm-up period (approximately twice the ACM collection interval; default ~10 minutes) for metrics to propagate to Thanos and for enough data points to accumulate for rate calculations. The graph may appear sooner (after ~5 minutes).
Access Kiali UI and navigate to a workload
Verify metrics appear in the Metrics tab and traffic graph

Ambient Mode: If you are using Ambient mode:

Ztunnel-only traffic (no waypoint): You’ll see TCP metrics and traffic edges in the graph, but HTTP details (response codes, latency) will not be available.
Traffic through waypoints: You’ll see full L7 metrics, same as sidecar mode.

Verify Metrics in Thanos Directly

Test that metrics exist in Thanos (from within the hub cluster). The following are different queries you can run to obtain metrics data from the backend metric datastore used by ACM.

Note: These commands use jq to format JSON output. If you don’t have jq installed, simply omit | jq . to see the full, unfiltered and raw JSON.

# List available metric names (Kiali uses istio_*, pilot_*, and envoy_* metrics)
oc get --raw "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/label/__name__/values" | jq -r '.data[] | select(startswith("istio_") or startswith("pilot_") or startswith("envoy_"))'

# Count timeseries for key Istio metrics (shows which metrics have data and how many unique timeseries)
oc get --raw "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/query?query=count%20by%20(__name__)%20({__name__=~%22istio_requests_total|istio_tcp.*total%22})" | jq -r '.data.result[] | "\(.metric.__name__): \(.value[1])"'

# Query Istio request metrics with full details (limited to first result to show structure)
oc get --raw "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/api/v1/query?query=istio_requests_total" | jq '.data.result |= .[0:1]'

Troubleshooting

Empty Graph or No Metrics

Symptom: Kiali shows an empty graph, “No metrics” in the metrics tab, or both.

Causes and Solutions:

scrape_interval too low: If thanos_proxy.scrape_interval is set lower than the ACM collection interval (e.g., “30s” instead of “5m”), Kiali’s rate calculations will use windows too narrow to capture enough data points from Thanos
- Solution: Set thanos_proxy.scrape_interval to match the ACM collection interval (default “5m”). See Thanos Proxy Mode for details
Still in warm-up period: After deploying a new application, it takes approximately twice the ACM collection interval (~10 minutes by default) before enough data points exist for rate calculations
- Solution: Wait for the warm-up period to elapse
Metrics not allowlisted: ACM doesn’t collect metrics by default
- Solution: Create observability-metrics-custom-allowlist ConfigMap with uwl_metrics_list.yaml key in source namespace
PodMonitor missing: Prometheus not scraping Istio data plane components
- Solution: Create istio-proxies-monitor PodMonitor in each mesh namespace (including the ztunnel namespace and namespaces with waypoint proxies if using Ambient mode)
UWM not enabled: User Workload Monitoring not configured
- Solution: Enable enableUserWorkload: true in cluster-monitoring-config ConfigMap in openshift-monitoring namespace
Missing source/destination labels: The graph builds its topology from workload and namespace labels in the metrics. Verify Istio metrics have proper labels
Namespace not selected: Ensure the namespace is selected in the graph’s namespace dropdown
Query scope mismatch: Check query_scope cluster names match actual cluster label values

See also the Why is my graph empty? FAQ for additional troubleshooting information.

TLS/Certificate Errors

Symptom: Kiali logs show “x509: certificate signed by unknown authority” or “tls: bad certificate”

Solutions:

Verify CA bundle: Ensure kiali-cabundle ConfigMap has the correct CA
```
oc get configmap kiali-cabundle -n ${KIALI_NAMESPACE} -o yaml
```

Check certificate chain: Verify client cert is signed by expected CA

oc get secret acm-observability-certs -n ${KIALI_NAMESPACE} \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -issuer

Verify projected volume: Check both ConfigMaps are mounted

oc exec -n ${KIALI_NAMESPACE} deploy/kiali -- ls -la /kiali-cabundle/
# Should show: additional-ca-bundle.pem, service-ca.crt

Connection Refused / Timeout

Symptom: Kiali cannot reach Observatorium API

Solutions:

Verify route exists:

oc get route observatorium-api -n open-cluster-management-observability

Check ACM is ready (should return “True”):

oc get mco observability -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'

Test connectivity (should return “OK”):

oc get --raw "/api/v1/namespaces/open-cluster-management-observability/services/http:observability-thanos-query-frontend:9090/proxy/-/ready"

Check NetworkPolicies: Ensure no policies block egress from Kiali’s namespace

Ambient Mode: No HTTP Metrics

Symptom: Ambient mode workloads show TCP traffic in Kiali but no HTTP metrics (response codes, latency)

Possible causes:

No waypoint deployed: Ztunnel only provides L4 (TCP) metrics. Deploy a waypoint proxy for L7 (HTTP) visibility.
Missing waypoint PodMonitor: Even with a waypoint, metrics won’t be collected without a PodMonitor:
- Verify waypoint pod exists: oc get pods -n <namespace> -l gateway.networking.k8s.io/gateway-class-name=istio-waypoint
- Create PodMonitor in the waypoint’s namespace (same config as sidecar PodMonitor)
Missing allowlist in waypoint namespace: Create a ConfigMap with the name observability-metrics-custom-allowlist in the namespace where the waypoint runs (see Metrics Allowlist Configuration)

Ambient Mode: No Ztunnel Metrics

Symptom: Ambient mode workloads show no traffic at all in Kiali

Possible causes:

Missing ztunnel PodMonitor: Create istio-proxies-monitor PodMonitor in the ztunnel namespace
Wrong ztunnel namespace: Verify ztunnel location: oc get pods -l app=ztunnel -A
Missing allowlist: Create a ConfigMap with the name observability-metrics-custom-allowlist in the ztunnel namespace (see Metrics Allowlist Configuration)

Reference

This example represents a fully configured Kiali installation using ACM Observability via the Observatorium API with mTLS:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: <kiali-namespace>
spec:
  clustering:
    ignore_home_cluster: true  # External deployment

  kubernetes_config:
    cluster_name: "<management-cluster-name>"

  external_services:
    prometheus:
      url: "<observatorium-api-url>"

      auth:
        type: none
        cert_file: "secret:acm-observability-certs:tls.crt"
        key_file: "secret:acm-observability-certs:tls.key"

      thanos_proxy:
        enabled: true
        retention_period: "14d"
        scrape_interval: "5m"

Required Kubernetes resources:

---
# mTLS client certificates (from ACM)
# Data extracted from Secret observability-grafana-certs in namespace open-cluster-management-observability
apiVersion: v1
kind: Secret
metadata:
  name: acm-observability-certs
  namespace: <kiali-namespace>
type: Opaque
data:
  tls.crt: <base64-encoded-certificate>  # From observability-grafana-certs secret, tls.crt key
  tls.key: <base64-encoded-key>          # From observability-grafana-certs secret, tls.key key

---
# Server CA trust (from ACM)
# Data extracted from Secret observability-client-ca-certs (or observability-server-ca-certs) in namespace open-cluster-management-observability
apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: <kiali-namespace>
data:
  additional-ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    <ACM Observability CA certificate>  # From ca.crt or tls.crt key (see Step 2 for extraction commands)
    -----END CERTIFICATE-----

Additional Resources

Docs: Header strategy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The header strategy assumes a reverse proxy is in front of Kiali, such as OpenUnison or OAuth2 Proxy, injecting the user’s identity into each request to Kiali as an Authorization header. This token can be an OpenID Connect token or any other token the cluster recognizes.

In addition to a user token, the header strategy supports impersonation headers. If the impersonation headers are present in the request, then Kiali will act on behalf of the user specified by the impersonation (assuming the token supplied in the Authorization header is authorized to do so).

The header strategy allows for namespace access control.

The header strategy is only supported for single cluster.

Set-up

The header strategy will work with any Kubernetes cluster. The token provided must be supported by that cluster. For instance, most “on-prem” clusters support OpenID Connect, but cloud hosted clusters do not. For clusters that don’t support a token, the impersonation headers can be injected by the reverse proxy.

spec:
  auth:
    strategy: header

The header strategy doesn’t have any additional configuration.

HTTP Header

The header strategy looks for a token in the Authorization HTTP header with the Bearer prefix. The HTTP header should look like:

Authorization: Bearer TOKEN

Where TOKEN is the appropriate token for your cluster. This TOKEN will be submitted to the API server via a TokenReview to validate the token ONLY on the first access to Kiali. On subsequent calls the TOKEN is passed through directly to the API server.

Security Considerations

Network Policies

A policy should be put in place to make sure that the only “client” for Kiali is the authenticating reverse proxy. This helps limit potential abuse and ensures that the authenticating reverse proxy is the source of truth for who accessed Kiali.

Short Lived Tokens

The authenticating reverse proxy should inject a short lived token in the Authorization header. A shorter lived token is less likely to be abused if leaked. Kiali will take whatever token is passed into the reqeuest, so as tokens are regenerated Kiali will use the new token.

Impersonation

TokenRequest API

The authenticating reverse proxy should use the TokenRequest API instead of static ServiceAccount tokens when possible while using impersonation. The ServiceAccount that can impersonate users and groups is privileged and having it be short lived cuts down on the possibility of a token being leaked while it’s being passed between different parts of the infrastructure.

Drop Incoming Impersonation Headers

The authenticating proxy MUST drop any headers it receives from a remote client that match the impersonation headers. Not only do you want to make sure that the authenticating proxy can’t be overriden on which user to authenticate, but also what groups they’re a member of.

Docs: Install via Helm

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Helm is a popular tool that lets you manage Kubernetes applications. Applications are defined in a package named Helm chart, which contains all of the resources needed to run an application.

Kiali has a Helm Charts Repository at https://kiali.org/helm-charts. Two Helm Charts are provided:

The kiali-operator Helm Chart installs the Kiali operator which in turn installs Kiali when you create a Kiali CR.
The kiali-server Helm Chart installs a standalone Kiali without the need of the Operator nor a Kiali CR.

The kiali-server Helm Chart does not provide all the functionality that the Kiali Operator provides. Some features you read about in the documentation may only be available if you install the Kiali Server using the Kiali Operator (see this FAQ for details). Therefore, although the kiali-server Helm Chart is actively maintained, it is not recommended and is only provided for convenience. If using Helm, the recommended method is to install the kiali-operator Helm Chart and then create a Kiali CR to let the Operator deploy Kiali.

Make sure you have the helm command available by following the Helm installation docs.

Helm version 3.10 is the minimum required Helm version. Older versions will not work. Newer versions have not been tested.

Adding the Kiali Helm Charts repository

Add the Kiali Helm Charts repository with the following command:

$ helm repo add kiali https://kiali.org/helm-charts

All helm commands in this page assume that you added the Kiali Helm Charts repository as shown.

If you already added the repository, you may want to update your local cache to fetch latest definitions by running:

$ helm repo update

Installing Kiali using the Kiali operator

This installation method gives Kiali access to existing namespaces as well as namespaces created later. See Namespace Management for more information.

Once you’ve added the Kiali Helm Charts repository, you can install the latest Kiali Operator along with the latest Kiali server by running the following command:

$ helm install \
    --set cr.create=true \
    --set cr.namespace=istio-system \
    --set cr.spec.auth.strategy="anonymous" \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator

The --namespace kiali-operator and --create-namespace flags instructs to create the kiali-operator namespace (if needed), and deploy the Kiali operator on it. The --set cr.create=true and --set cr.namespace=istio-system flags instructs to create a Kiali CR in the istio-system namespace. Since the Kiali CR is created in advance, as soon as the Kiali operator starts, it will process it to deploy Kiali. After Kiali has started, you can access Kiali UI through ‘http://localhost:20001’ by executing kubectl port-forward service/kiali -n istio-system 20001:20001 because of --set cr.spec.auth.strategy="anonymous". But realize that anonymous mode will allow anyone to be able to see and use Kiali. If you wish to require users to authenticate themselves by logging into Kiali, use one of the other auth strategies.

The Kiali Operator Helm Chart is configurable. Check available options and default values by running:

$ helm show values kiali/kiali-operator

You can pass the --version X.Y.Z flag to the helm install and helm show values commands to work with a specific version of Kiali.

The kiali-operator Helm Chart mirrors all settings of the Kiali CR as chart values that you can configure using regular --set flags. For example, the Kiali CR has a spec.server.web_root setting which you can configure in the kiali-operator Helm Chart by passing --set cr.spec.server.web_root=/your-path to the helm install command.

For more information about the Kiali CR, see the Creating and updating the Kiali CR page.

Operator-Only Install

To install only the Kiali Operator, omit the --set cr.create and --set cr.namespace flags of the helm command previously shown. For example:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator

This will omit creation of the Kiali CR, which you will need to create later to install Kiali Server. This option is good if you plan to do large customizations to the installation.

Installing Multiple Instances of Kiali

By installing a single Kiali operator in your cluster, you can install multiple instances of Kiali by simply creating multiple Kiali CRs. For example, if you have two Istio control planes in namespaces istio-system and istio-system2, you can create a Kiali CR in each of those namespaces to install a Kiali instance in each control plane.

If you wish to install multiple Kiali instances in the same namespace, or if you need the Kiali instance to have different resource names than the default of kiali, you can specify spec.deployment.instance_name in your Kiali CR. The value for that setting will be used to create a unique instance of Kiali using that instance name rather than the default kiali. One use-case for this is to be able to have unique Kiali service names across multiple Kiali instances in order to be able to use certain routers/load balancers that require unique service names.

Since the spec.deployment.instance_name field is used for the Kiali resource names, including the Service name, you must ensure the value you assign this setting follows the Kubernetes DNS Label Name rules. If it does not, the operator will abort the installation. And note that because Kiali uses this as a prefix (it may append additional characters for some resource names) its length is limited to 40 characters.

Standalone Kiali installation

To install the Kiali Server without the operator, use the kiali-server Helm Chart:

$ helm install \
    --namespace istio-system \
    kiali-server \
    kiali/kiali-server

The kiali-server Helm Chart mirrors all settings of the Kiali CR as chart values that you can configure using regular --set flags. For example, the Kiali CR has a spec.server.web_fqdn setting which you can configure in the kiali-server Helm Chart by passing the --set server.web_fqdn flag as follows:

$ helm install \
    --namespace istio-system \
    --set server.web_fqdn=example.com \
    kiali-server \
    kiali/kiali-server

Upgrading Helm installations

If you want to upgrade to a newer Kiali version (or downgrade to older versions), you can use the regular helm upgrade commands. For example, the following command should upgrade the Kiali Operator to the latest version:

$ helm upgrade \
    --namespace kiali-operator \
    --reuse-values \
    kiali-operator \
    kiali/kiali-operator

WARNING: No migration paths are provided. However, Kiali is a stateless application and if the helm upgrade command fails, please uninstall the previous version and then install the new desired version.

By upgrading the Kiali Operator, existent Kiali Server installations managed with a Kiali CR will also be upgraded once the updated operator starts.

Managing configuration of Helm installations

After installing either the kiali-operator or the kiali-server Helm Charts, you may be tempted to manually modify the created resources to modify the installation. However, we recommend using helm upgrade to update your installation.

For example, assuming you have the following installation:

$ helm list -n kiali-operator
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
kiali-operator  kiali-operator  1               2021-09-14 18:00:45.320351026 -0500 CDT deployed        kiali-operator-1.40.0   v1.40.0

Notice that the current installation is version 1.40.0 of the kiali-operator. Let’s assume you want to use your own mirrors of the Kiali Operator container images. You can update your installation with the following command:

$ helm upgrade \
    --namespace kiali-operator \
    --reuse-values \
    --set image.repo=your_mirror_registry_url/owner/kiali-operator-repo \
    --set image.tag=your_mirror_tag \
    --version 1.40.0 \
    kiali-operator \
    kiali/kiali-operator

Make sure that you specify the --reuse-values flag to take the configuration of your current installation. Then, you only need to specify the new settings you want to change using --set flags.

Make sure that you specify the --version X.Y.Z flag with the version of your current installation. Otherwise, you may end up upgrading to a new version.

Uninstalling

Removing the Kiali operator and managed Kialis

If you used the kiali-operator Helm chart, first you must ensure that all Kiali CRs are deleted. For example, the following command will agressively delete all Kiali CRs in your cluster:

$ kubectl delete kiali --all --all-namespaces

The previous command may take some time to finish while the Kiali operator removes all Kiali installations.

Then, remove the Kiali operator using a standard helm uninstall command. For example:

$ helm uninstall --namespace kiali-operator kiali-operator
$ kubectl delete crd kialis.kiali.io

You have to manually delete the kialis.kiali.io CRD because Helm won’t delete it.

If you fail to delete the Kiali CRs before uninstalling the operator, a proper cleanup may not be done.

Known problem: uninstall hangs (unable to delete the Kiali CR)

Typically this happens if not all Kiali CRs are deleted prior to uninstalling the operator. To force deletion of a Kiali CR, you need to clear its finalizer. For example:

$ kubectl patch kiali kiali -n istio-system -p '{"metadata":{"finalizers": []}}' --type=merge

This forces deletion of the Kiali CR and will skip uninstallation of the Kiali Server. Remnants of the Kiali Server may still exist in your cluster which you will need to manually remove.

Removing standalone Kiali

If you installed a standalone Kiali by using the kiali-server Helm chart, use the standard helm uninstall commands. For example:

$ helm uninstall --namespace istio-system kiali-server

Docs: Kiali MCP

Mon, 01 Jan 0001 00:00:00 +0000

Kiali MCP is an integration that allows MCP-capable AI assistants to query (and optionally manage) Kiali-related data by calling tools exposed by an MCP server.

The implementation is provided as part of the Kubernetes MCP Server upstream and also for Openshift MCP server. It exposes a kiali toolset (see upstream guide: docs/KIALI.md).

Prerequisites

A reachable Kiali endpoint (Route/Ingress/Service URL).
Kubernetes credentials available to the MCP server (kubeconfig or in-cluster config).

Enable the `kiali` toolset

Create a TOML config file and enable kiali in toolsets.

toolsets = ["core", "kiali"]

[toolset_configs.kiali]
url = "https://kiali.example" # Endpoint/route to reach the Kiali console
# insecure = true  # optional: allow insecure TLS (not recommended in production)
# certificate_authority = "/path/to/ca.crt"  # CA bundle for Kiali's TLS cert

Notes:

If url is https:// and insecure = false, you must provide certificate_authority.
Authentication to Kiali is performed using the server’s Kubernetes credentials (it obtains/uses a bearer token for Kiali calls).

Connect from an MCP client

How you wire this into a specific client depends on the client, but the core idea is the same: start the MCP server with your kubeconfig and your TOML config.

Example (conceptual) command:

kubernetes-mcp-server --config /path/to/config.toml --read-only

Once connected, your assistant can use the Kiali tools (for example: mesh graph, metrics, traces, workload logs) to power a chatbot-like experience outside the Kiali UI (for example, in an IDE).

Docs: Deployment Options

Mon, 01 Jan 0001 00:00:00 +0000

There are other, more complex deployment settings, described in dedicated pages:

All examples on this page are focused on the Kiali CR (when installing via the Kiali operator). Remember that Helm charts mirror all these configurations.

Kiali and Istio installation namespaces

By default, the Kiali operator installs Kiali in the same namespace where the Kiali CR is created. However, it is possible to specify a different namespace for installation:

spec:
  deployment:
    namespace: "custom-kiali-namespace"

Log level and format

By default, Kiali will print up to INFO-level messages in simple text format. You can change the log level, output format, and time format as in the following example:

spec:
  deployment:
    logger:
      # Supported values are "trace", "debug", "info", "warn", "error" and "fatal"
      log_level: error  
      # Supported values are "text" and "json".
      log_format: json  
      time_field_format: "2006-01-02T15:04:05Z07:00"

The syntax for the time_field_format is the same as the Time.Format function of the Go language.

The json format is useful if you are parsing logs of your applications for further processing.

In Kiali, there are some special logs called audit logs that are emitted each time a user creates, updates or deletes a resource through Kiali. Audit logs are INFO-level messages and are enabled by default. If audit logs are too verbose, you can disable them without reducing the log level as follows:

spec:
  server:
    audit_log: false

Kiali instance name

If you plan to install more than one Kiali instance on the same cluster, you may need to configure an instance name to avoid conflicts on created resources:

spec:
  deployment:
    instance_name: "secondary"

The instance_name will be used as a prefix for all created Kiali resources. The exception is the kiali-signing-key secret which will always have the same name and will be shared on all deployments of the same namespace, unless you specify a custom secret name.

Since the instance_name will be used as a name prefix in resources, it must follow Kubernetes naming constraints.

Since Kubernetes resources cannot be renamed, you cannot change the instance_name of an existing Kiali installation. The workaround is to uninstall Kiali and re-install with the desired instance_name.

Resource requests and limits

You can set the amount of resources available to Kiali using the spec.deployment.resources attribute, like in the following example:

spec:
  deployment:
    resources:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "1Gi"
        cpu: "500m"

Please, read the Managing Resources for Containers section in the Kubernetes documentation for more details about possible configurations.

Custom labels and annotations on the Kiali pod and service

Although some labels and annotations are set on the Kiali pod and on its service (depending on configurations), you can add additional ones. For the pod, use the spec.deployment.pod_labels and spec.deployment.pod_annotations attributes. For the service, you can only add annotations using the spec.deployment.service_annotations attribute. For example:

spec:
  deployment:
    pod_annotations:
      a8r.io/repository: "https://github.com/kiali/kiali"
    pod_labels:
      sidecar.istio.io/inject: "true"
    service_annotations:
      a8r.io/documentation: "https://kiali.io/docs/installation/deployment-configuration"

Kiali page title (browser title bar)

If you have several Kiali installations and you are using them at the same time, there are good chances that you will want to identify each Kiali by simply looking at the browser’s title bar. You can set a custom text in the title bar with the following configuration:

spec:
  installation_tag: "Kiali West"

The installation_tag is any human readable text of your desire.

Kubernetes scheduler settings

Replicas and automatic scaling

By default, only one replica of Kiali is deployed. If needed, you can change the replica count like in the following example:

spec:
  deployment:
    replicas: 2

If you prefer automatic scaling, creation of an HorizontalPodAutoscaler resource is supported. For example, the following configuration automatically scales Kiali based on CPU utilization:

spec:
  deployment:
    hpa:
      api_version: "autoscaling/v1"
      spec:
        minReplicas: 1
        maxReplicas: 2
        targetCPUUtilizationPercentage: 80

You must omit the scaleTargetRef field of the HPA spec, because this field will be populated by the Kiali operator (or by Helm) depending on other configuration.

Read the Kubernetes Horizontal Pod Autoscaler documentation to learn more about the HPA.

Allocating the Kiali pod to specific nodes of the cluster

You can constrain the Kiali pod to run on a specific set of nodes by using some of the standard Kubernetes scheduler configurations.

The simplest option is to use the nodeSelector configuration which you can configure like in the following example:

spec:
  deployment:
    node_selector:
      worker-type: infra

You can also use the affinity/anti-affinity native Kubernetes feature if you prefer its more expressive syntax, or if you need more complex matching rules. The following is an example for configuring node affinity:

spec:
  deployment:
    affinity:
      node:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: worker-type
              operator: In
              values:
              - infra

Similarly, you can also configure pod affinity and pod anti-affinity using the spec.deployment.affinity.pod and spec.deployment.affinity.pod_anti attributes.

Finally, if you want to run Kiali in a node with taints, the following is an example to configure tolerations:

spec:
  deployment:
    tolerations: # Allow to run Kiali in a tainted master node
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"

Read the following Kubernetes documentation to learn more about these configurations:

Priority class of the Kiali pod

If you are using priority classes in your cluster, you can specify the priority class that will be set on the Kiali pod. For example:

spec:
  deployment:
    priority_class_name: high-priority

For more information about priority classes, read Pod Priority and Preemption in the Kubernetes documentation.

Adding host aliases to the Kiali pod

If you need to provide some static hostname resolution in the Kiali pod, you can use HostAliases to add entries to the /etc/hosts file of the Kiali pod, like in the following example:

spec:
  deployment:
   host_aliases:
   - ip: 192.168.1.100
     hostnames:
     - "foo.local"
     - "bar.local"

HTTP server

Kiali is served over HTTP. You can configure a few options of the HTTP server. The following are the defaults, but you can change them to suit your needs.

spec:
  server:
    # Listen/bind address of the HTTP server. By default it is empty, which means to
    # listen on all interfaces.
    address: ""

    # Listening port of the HTTP server. If you change it, also Kiali's Kubernetes
    # Service is affected to use this port.
    port: 20001

    # Use GZip compression for responses larger than 1400 bytes. You may want to disable
    # compression if you are exposing Kiali via a reverse proxy that is already
    # doing compression.
    gzip_enabled: true

    # For development purposes only. Controls if "Cross-Origin Resourse Sharing" is
    # enabled. 
    cors_allow_all: false

There is one additional spec.server.web_root option that affects the HTTP server, but that one is described in the Specifying route settings section of the Instalation guide.

Metrics server

Kiali emits metrics that can be collected by Prometheus. Most of these metrics are performance measurements.

The metrics server is enabled by default and listens on port 9090:

spec:
  server:
    metrics_enabled: true
    metrics_port: 9090

The bind address is the same as the HTTP server. Thus, make sure that the HTTP Server and the metrics server are not configured to the same port.

Docs: Install via OperatorHub

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The OperatorHub is a website that contains a catalog of Kubernetes Operators. Its aim is to be the central location to find Operators.

The OperatorHub relies in the Operator Lifecycle Manager (OLM) to install, manage and update Operators on any Kubernetes cluster.

The Kiali Operator is being published to the OperatorHub. So, you can use the OLM to install and manage the Kiali Operator installation.

Installing the Kiali Operator using the OLM

Go to the Kiali Operator page in the OperatorHub: https://operatorhub.io/operator/kiali.

You will see an Install button at the right of the page. Press it and you will be presented with the installation instructions. Follow these instructions to install and manage the Kiali Operator installation using OLM.

Afterwards, you can create the Kiali CR to install Kiali.

Installing the Kiali Operator in OpenShift

The OperatorHub is bundled in the OpenShift console. To install the Kiali Operator, simply go to the OperatorHub in the OpenShift console and search for the Kiali Operator. Then, click on the Install button and follow the instruction on the screen.

Afterwards, you can create the Kiali CR to install Kiali.

Docs: OpenID Connect strategy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The openid authentication strategy lets you integrate Kiali to an external identity provider that implements OpenID Connect, and allows users to login to Kiali using their existing accounts of a third-party system.

If your Kubernetes cluster is also integrated with your OpenId provider, then Kiali’s openid strategy can offer namespace access control.

Kiali only supports the authorization code flow of the OpenId Connect spec.

Requirements

The Kiali’s signing key needs to be 16, 24 or 32 byte long. If you install Kiali via the operator and don’t set a custom signing key, the operator should create a 16 byte long signing key.

If you don’t need namespace access control support, you can use any working OpenId Server where Kiali can be configured as a client application.

If you do need namespace access control support, you need either:

A Kubernetes cluster configured with OpenID connect integration, which results in the API server accepting tokens issued by your identity provider.
A replacement or reverse proxy for the Kubernetes cluster API capable of handling the OIDC authentication.

The first option is preferred if you can manipulate your cluster API server startup flags, which will result in your cluster to also be integrated with the external OpenID provider.

The second option is provided for cases where you are using a managed Kubernetes and your cloud provider does not support configuring OpenID integration. Kiali assumes an implementation of a Kubernetes API server. For example, a community user has reported to successfully configure Kiali’s OpenID strategy by using kube-oidc-proxy which is a reverse proxy that handles the OpenID authentication and forwards the authenticated requests to the Kubernetes API.

Set-up with namespace access control support

Assuming you already have a working Kubernetes cluster with OpenId integration (or a working alternative like kube-oidc-proxy), you should already had configured an application or a client in your OpenId server (some cloud providers configure this app/client automatically for you). You must re-use this existing application/client by adding the root path of your Kiali instance as an allowed/authorized callback URL. If the OpenID server provided you a client secret for the application/client, or if you had manually set a client secret, issue the following command to create a Kubernetes secret holding the OpenId client secret:

kubectl create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n $NAMESPACE

where $NAMESPACE is the namespace where you installed Kiali and $CLIENT_SECRET is the secret you configured or provided by your OpenId Server. If Kiali is already running, you may need to restart the Kiali pod so that the secret is mounted in Kiali.

It’s worth emphasizing that to configure OpenID integration you must re-use the OpenID application/client that you created for your Kubernetes cluster. If you create a new application/client for Kiali in your OpenId server, Kiali will fail to properly authenticate users.

Then, to enable the OpenID Connect strategy, the minimal configuration you need to set in the Kiali CR is like the following:

spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      issuer_uri: "https://openid.issuer.com"

This assumes that your Kubernetes cluster is configured with OpenID Connect integration. In this case, the client-id and issuer_uri attributes must match the --oidc-client-id and --oidc-issuer-url flags used to start the cluster API server. If these values don’t match, users will fail to login to Kiali.

If you are using a replacement or a reverse proxy for the Kubernetes API server, the minimal configuration is like the following:

spec:
  auth:
    strategy: openid
    openid:
      api_proxy: "https://proxy.domain.com:port"
      api_proxy_ca_data: "..."
      client_id: "kiali-client"
      issuer_uri: "https://openid.issuer.com"

The value of client-id and issuer_uri must match the values of the configuration of your reverse proxy or cluster API replacement. The api_proxy attribute is the URI of the reverse proxy or cluster API replacement (only HTTPS is allowed). The api_proxy_ca_data is the public certificate authority file encoded in a base64 string, to trust the secure connection.

Set-up with no namespace access control support

Register Kiali as a client application in your OpenId Server. Use the root path of your Kiali instance as the callback URL. If the OpenId Server provides you a client secret, or if you manually set a client secret, issue the following command to create a Kubernetes secret holding the OpenId client secret:

kubectl create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n $NAMESPACE

Then, to enable the OpenID Connect strategy, the minimal configuration you need to set in the Kiali CR is like the following:

spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      disable_rbac: true
      issuer_uri: "https://openid.issuer.com"

As namespace access control is disabled, all users logging into Kiali will share the same cluster-wide privileges.

Additional configurations

Configuring the displayed user name

The Kiali front-end will, by default, retrieve the string of the sub claim of the OpenID token and display it as the user name. You can customize which field to display as the user name by setting the username_claim attribute of the Kiali CR. For example:

spec:
  auth:
    openid:
      username_claim: "email"

If you enabled namespace access control, you will want the username_claim attribute to match the --oidc-username-claim flag used to start the Kubernetes API server, or the equivalent option if you are using a replacement or reverse proxy of the API server. Else, any user-friendly claim will be OK as it is purely informational.

Configuring requested scopes

By default, Kiali will request access to the openid, profile and email standard scopes. If you need a different set of scopes, you can set the scopes attribute in the Kiali CR. For example:

spec:
  auth:
    openid:
      scopes:
      - "openid"
      - "email"
      - "groups"

The openid scope is forced. If you don’t add it to the list of scopes to request, Kiali will still request it from the identity provider.

Configuring authentication timeout

When the user is redirected to the external authentication system, by default Kiali will wait at most 5 minutes for the user to authenticate. After that time has elapsed, Kiali will reject authentication. You can adjust this timeout by setting the authentication_timeout with the number of seconds that Kiali should wait at most. For example:

spec:
  auth:
    openid:
      authentication_timeout: 60 # Wait only one minute.

Configuring allowed domains

Some identity providers use a shared login and regardless of configuring your own application under your domain (or organization account), login can succeed even if the user that is logging in does not belong to your account or organization. Google is an example of this kind of provider.

To prevent foreign users from logging into your Kiali instance, you can configure a list of allowed domains:

spec:
  auth:
    openid:
      allowed_domains:
      - example.com
      - foo.com

The e-mail reported by the identity provider is used for the validation. Login will be allowed if the domain part of the e-mail is listed as an allowed domain; else, the user will be rejected. Naturally, you will need to configure the email scope to be requested.

There is a special case: some identity providers include a hd claim in the id_token. If this claim is present, this is used instead of extracting the domain from the user e-mail. For example, Google Workspace (aka G Suite) includes this hd claim for hosted domains.

Using an OpenID provider with a self-signed certificate

If your OpenID provider is using a self-signed certificate, you can disable certificate validation by setting the insecure_skip_verify_tls to true in the Kiali CR:

spec:
  auth:
    openid:
      insecure_skip_verify_tls: true

You should use self-signed certificates only for testing purposes.

However, if your organization or internal network has an internal trusted certificate authority (CA), and your OpenID server is using a certificate issued by this CA, you can configure Kiali to trust certificates from this CA rather than disabling verification.

See the TLS Configuration page for detailed instructions on configuring custom CA certificates. You can use either the global additional-ca-bundle.pem key (which makes the CA trusted for all HTTPS connections) or the OpenID-specific openid-server-ca.crt key in the kiali-cabundle ConfigMap.

Using an HTTP/HTTPS Proxy

In some network configurations, there is the need to use proxies to connect to the outside world. OpenID requires outside world connections to get metadata and do key validation, so you can configure it by setting the http_proxy and https_proxy keys in the Kiali CR. They use the same format as the HTTP_PROXY and HTTPS_PROXY environment variables.

spec:
  auth:
    openid:
      http_proxy: http://USERNAME:PASSWORD@10.0.1.1:8080/
      https_proxy: https://USERNAME:PASSWORD@10.0.0.1:8080/

Passing additional options to the identity provider

When users click on the Login button on Kiali, a redirection occurs to the authentication page of the external identity provider. Kiali sends a fixed set of parameters to the identity provider to enable authentication. If you need to add an additional set of parameters to your identity provider, you can use the additional_request_params setting of the Kiali CR, which accepts key-value pairs. For example:

spec:
  auth:
    openid:
      additional_request_params:
        prompt: login

The prompt parameter is a standard OpenID parameter. When the login value is passed in this parameter, the identity provider is instructed to ask for user credentials regardless if the user already has an active session because of a previous login in some other system.

If your OpenId provider supports other non-standard parameters, you can specify the ones you need in this additional_request_params setting.

Take into account that you should not add the client_id, response_type, redirect_uri, scope, nonce nor state parameters to this list. These are already in use by Kiali and some already have a dedicated setting.

Provider-specific instructions

Using with Keycloak

When using OpenId with Keycloak, you will need to enable the Standard Flow Enabled option on the Client (in the Administration Console):

The Standard Flow described on the options is the same as the authorization code flow from the rest of the documentation.

Using with Google Cloud Platform / GKE OAuth2

If you are using Google Cloud Platform (GCP) and its products such as Google Kubernetes Engine (GKE), it should be straightforward to configure Kiali’s OpenID strategy to authenticate using your Google credentials.

First, you’ll need to go to your GCP Project and to the Credentials screen which is available at (Menu Icon) > APIs & Services > Credentials.

On the Credentials screen you can select to create a new OAuth client ID.

If you’ve never setup the OAuth consent screen you will need to do that before you can create an OAuth client ID. On screen you’ll have multiple warnings and prompts to walk you through this.

On the Create OAuth client ID screen, set the Application type to Web Application and enter a name for your key.

Then enter in the Authorized Javascript origins and Authorized redirect URIs for your project. You can enter in localhost as appropriate during testing. You can also enter multiple URIs as appropriate.

After clicking Create you’ll be shown your newly minted client id and secret. These are important and needed for your Kiali CR yaml and Kiali secrets files.

You’ll need to update your Kiali CR file to include the following auth block.

spec:
  auth:
    strategy: "openid"
    openid:
      client_id: "<your client id from GCP>"
      disable_rbac: true
      issuer_uri: "https://accounts.google.com"
      scopes: ["openid", "email"]
      username_claim: "email"

Don’t get creative here. The issuer_uri should be https://accounts.google.com.

Finally you will need to create a secret, if you don’t have one already, that sets the oidc-secret for the openid flow.

apiVersion: v1
kind: Secret
metadata:
  name: kiali
  namespace: istio-system
  labels:
    app: kiali
type: Opaque
data:
  oidc-secret: "<base64 encode your client secret from GCP and enter here>"

Once all these settings are complete just set your Kiali CR and the Kiali secret to your cluster. You may need to refresh your Kiali Pod to set the Secret if you add the Secret after the Kiali pod is created.

Using with OpenShift and an external OIDC provider

Starting with OpenShift 4.20, you can configure your OpenShift cluster to authenticate users against an external OpenID Connect (OIDC) provider instead of using the built-in OAuth server. This is sometimes called “Bring Your Own OIDC” (BYO OIDC). When OpenShift is configured this way, Kiali can use the openid authentication strategy with full namespace access control support.

This section applies when you want to use an external OIDC provider (such as Keycloak, Okta, Auth0, or others) with OpenShift. If you want to use OpenShift’s built-in OAuth server, use the openshift authentication strategy instead.

Prerequisites

OpenShift 4.20 or later
An external OIDC provider configured and accessible from your OpenShift cluster
The OIDC provider must be configured as an authentication source for both OpenShift and Kiali (they share the same provider)
A certificate-based kubeconfig or long-lived service account token for emergency cluster access (the built-in OAuth will be disabled)

Step 1: Configure OpenShift for external OIDC authentication

First, configure your OpenShift cluster to use your external OIDC provider. This involves modifying the cluster’s Authentication resource to specify your OIDC provider details.

Refer to the official OpenShift documentation for detailed instructions: Enabling direct authentication with an external OIDC identity provider.

The key configuration elements include:

Issuer URL: The URL of your OIDC provider
Client ID: The OAuth2 client ID registered with your OIDC provider
Audiences: The list of acceptable audiences for tokens (must include your Kiali client ID)
Username claim mapping: How the OIDC token claims map to Kubernetes usernames
Username prefix: Optional prefix to distinguish OIDC users (e.g., oidc:)
CA certificate: If your OIDC provider uses a private CA
webhookTokenAuthenticator: null: Must be set when type is OIDC

When OpenShift is configured with external OIDC authentication, the built-in OAuth server is disabled. This means:

Users cannot log in using OpenShift’s standard login page
The OAuthClient API becomes unavailable
Keep a certificate-based kubeconfig (or other long-lived admin credentials) available for emergency access because the normal OAuth login paths and OAuth APIs are unavailable in this mode

Ensure your OIDC provider is properly configured and you have set up RBAC policies before enabling external OIDC authentication.

Step 2: Configure user RBAC

When using external OIDC with OpenShift, user identities in Kubernetes are derived from the OIDC token claims. OpenShift typically adds a configurable prefix to the username (e.g., oidc:) to distinguish OIDC users from other identity sources.

For example, if your OIDC provider returns user@example.com in the email claim and you configured a prefix of oidc:, the Kubernetes username becomes oidc:user@example.com.

Create RBAC resources to grant users access to the namespaces they need. See Namespace access control for details on the required privileges.

Example Role and RoleBinding to grant a user access to the istio-system namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kiali-user-access
  namespace: istio-system
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods/log
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kiali-user-access
  namespace: istio-system
subjects:
- kind: User
  name: "oidc:user@example.com"  # Use the prefixed username
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: kiali-user-access
  apiGroup: rbac.authorization.k8s.io

The username prefix (e.g., oidc:) is configured in OpenShift’s Authentication resource under spec.oidcProviders[].claimMappings.username.prefix.prefixString. Make sure your RBAC resources use the same prefixed username format.

Step 3: Create the OIDC client secret

If your OIDC provider requires a client secret, create a Kubernetes secret to store it:

oc create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n istio-system

Replace $CLIENT_SECRET with the client secret from your OIDC provider.

Step 4: Configure the CA certificate (if needed)

If your OIDC provider uses a certificate issued by a private CA (not a public CA), you need to configure Kiali to trust it. Create a ConfigMap with the CA certificate:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system
data:
  openid-server-ca.crt: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgIQAqxcJmoLQ...
    ... (your OIDC provider's CA certificate) ...
    -----END CERTIFICATE-----

See TLS Configuration for more details on configuring custom CA certificates.

Step 5: Configure the Kiali CR

Configure Kiali to use the openid authentication strategy. The client_id and issuer_uri must match the values configured in OpenShift’s Authentication resource:

spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      issuer_uri: "https://your-oidc-provider.example.com"
      scopes:
      - "openid"
      - "email"
      username_claim: "email"

The client_id must be listed in the audiences array of your OpenShift OIDC configuration, and the issuer_uri must exactly match the issuer URL configured in OpenShift. If these don’t match, authentication will fail.

Important configuration notes:

username_claim: Should match the claim mapping configured in OpenShift (commonly email or preferred_username)
scopes: Request the scopes that provide the claims you need (typically openid and email)
disable_rbac: Do not set this to true if you want per-user namespace access control. When disable_rbac is false (the default), Kiali uses the user’s OIDC token for Kubernetes API calls, enabling per-user RBAC.

Complete example

Here’s a complete example of the Kiali CR configuration for OpenShift with an external OIDC provider:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      issuer_uri: "https://your-oidc-provider.example.com"
      scopes:
      - "openid"
      - "email"
      username_claim: "email"

With the supporting resources:

# OIDC client secret (if required by your provider)
apiVersion: v1
kind: Secret
metadata:
  name: kiali
  namespace: istio-system
type: Opaque
stringData:
  oidc-secret: "your-client-secret-here"
---
# CA certificate (if using a private CA)
apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system
data:
  openid-server-ca.crt: |
    -----BEGIN CERTIFICATE-----
    ... (your CA certificate) ...
    -----END CERTIFICATE-----    
---
# RBAC for a user (repeat for each user/namespace combination)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kiali-user-access
  namespace: istio-system
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods/log
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kiali-user-access
  namespace: istio-system
subjects:
- kind: User
  name: "oidc:user@example.com"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: kiali-user-access
  apiGroup: rbac.authorization.k8s.io

Using with Azure: AKS and AAD

The OpenID authentication strategy can be used with Azure Kubernetes Service (AKS) and Azure Active Directory (AAD) with Kiali versions 1.33 and later. Prior Kiali versions do not support namespace access control on Azure.

AKS has support for a feature named AKS-managed Azure Active Directory, which enables integration between AKS and AAD. This has the advantage that users can use their AAD credentials to access AKS clusters and can also use Kubernetes RBAC features to assign privileges to AAD users.

However, Azure is implementing this integration via the Kubernetes Webhook Token Authentication rather than via the Kubernetes OpenID Connect Tokens authentication (see the Azure AD integration section in AKS Concepts documentation). Because of this difference, authentication in AKS behaves slightly different from a standard OpenID setup, but Kiali’s OpenID authentication strategy can still be used with namespace access control support by following the next steps.

First, enable the AAD integration on your AKS cluster. See the official AKS documentation to learn how. Once it is enabled, your AKS panel should show the following:

Create a web application for Kiali in your Azure AD panel:

Go to AAD > App Registration, create an application with a redirect url like https://<your-kiali-url>/kiali
Go to Certificates & secrets and create a client secret.
1. After creating the client secret, take note of the provided secret. Create a Kubernetes secret in your cluster as mentioned in the Set-up with namespace access control support section. Please, note that the suggested name for the Kubernetes Secret is kiali. If you want to customize the secret name, you will have to specify your custom name in the Kiali CR. See: secret_name in Kial CR Reference.
Go to API Permissions and press the Add a permission button. In the new page that appears, switch to the APIs my organization uses tab.
Type the following ID in the search field: 6dae42f8-4368-4678-94ff-3960e28e3630 (this is a shared ID for all Azure clusters). And select the resulting entry.
Select the Delegated permissions square.
Select the user.read permission.
Go to Authentication and make sure that the Access tokens checkbox is ticked.

Then, create or modify your Kiali CR and include the following settings:

spec:
  auth:
    strategy: "openid"
    openid:
      client_id: "<your Kiali application client id from Azure>"
      issuer_uri: "https://sts.windows.net/<your AAD tenant id>/"
      username_claim: preferred_username
      api_token: access_token
      additional_request_params:
        resource: "6dae42f8-4368-4678-94ff-3960e28e3630"

You can find your client_id and tenant_id in the Overview page of the Kiali App registration that you just created. See this documentation for more information.

Docs: OpenShift strategy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The openshift authentication strategy is the preferred and default strategy when Kiali is deployed on an OpenShift cluster.

When using the openshift strategy, a user logging into Kiali will be redirected to the login page of the OpenShift console. Once the user provides his OpenShift credentials, he will be redireted back to Kiali and will be logged in if the user has enough privileges.

The openshift strategy supports namespace access control.

The openshift strategy is supported for single and multi-cluster deployments.

Set-up

Since openshift is the default strategy when deploying Kiali in OpenShift, you shouldn’t need to configure anything. If you want to be verbose, use the following configuration in the Kiali CR:

spec:
  auth:
    strategy: openshift

The Kiali operator will make sure to setup the needed OpenShift OAuth resources to register Kiali as a client for the most common use-cases. The openshift strategy does have a few configuration settings that most people will never need but are available in case you have a situation where the customization is needed. See the Kiali CR Reference page for the documentation on those settings.

Multi-Cluster

There are some things to know when using the openshift strategy with Kiali in a multi-cluster environment.

Consistent Kiali Namespace and Instance-Name

The default namespace for Kiali is istio-system. But many users prefer to use a dedicated namespace for Kiali, such as kiali, kiali-server, etc. In a multi-cluster environment Kiali must be deployed in the same namespace on each cluster. Clusters that don’t have a Kiali deployment must still provide the namespace, to hold the remote cluster resources.

The default instance-name for kiali is kiali. Any change to the default must also be made consistently across all clusters.

Assuming Kiali is installed via the Kiali Operator. Any customization would be done via the following CR settings:

spec.deployment.namespace
spec.deployment.instance_name

It is recommended that the Kiali Operator be deployed on all clusters, even if Kiali itself is not deployed. This will ensure that the proper namespace and remote cluster resources are created. For clusters without Kiali, requiring only the remote cluster resources (for auth), configure the CR with:

spec.deployment.remote_cluster_resources_only: true

OpenShift OAuthClient Naming

OpenShift OAuth requires an OAuthClient resource on each cluster to be named <instance-name>-<namespace>. For example, if Kiali is installed with the default instance name kiali in namespace istio-system, the OAuthClient on every cluster must be named kiali-istio-system.

Both the Kiali Operator and the Kiali Server helm chart automatically create the OAuthClient with the correct name when they create the remote cluster resources. The kiali-prepare-remote-cluster.sh script also delegates to the Kiali Server helm chart for resource creation and will produce a correctly-named OAuthClient, provided you pass --kiali-resource-name and --remote-cluster-namespace values that match the Kiali instance name and namespace on the cluster where Kiali is deployed. If you are managing resources entirely manually, ensure the OAuthClient on the remote cluster is named consistently with the Kiali instance name and namespace.

If the OAuthClient names do not match across clusters, OAuth authentication will fail.

OAuthClient Redirect URIs for Remote Cluster Resources

When using remote_cluster_resources_only: true on a remote cluster with the openshift auth strategy, the Kiali Operator must create an OAuthClient resource but cannot automatically determine the redirect URI (since there is no Kiali server or route on the remote cluster). You must explicitly specify the redirect URI in the Kiali CR on the remote cluster via spec.auth.openshift.redirect_uris. Without this, the Kiali Operator will fail to reconcile with the error:

Redirect URIs for the Kiali Server OAuthClient are not specified via auth.openshift.redirect_uris;
this is required when creating remote cluster resources with auth.strategy of openshift.

The redirect URI must point back to the Kiali server on the cluster where Kiali is deployed. Critically, the URI must include the remote cluster’s name as a path suffix in the form https://<kiali-route-host>/api/auth/callback/<remote-cluster-name>. This is required so that the OAuth callback can correctly identify which cluster the login is for. Using the base /api/auth/callback path (without the cluster name) will result in the login failing with a http: named cookie not present error.

To determine the correct URI:

If Kiali is already deployed, run this on the cluster where Kiali is deployed: oc get route -l app.kubernetes.io/name=kiali -n <kiali-namespace> -o jsonpath='{..spec.host}'
If Kiali is not yet deployed, you can predict the route hostname from the cluster’s app domain by running this on the cluster where Kiali will be deployed: oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}'

The Kiali route hostname will be something like kiali-<namespace>.<app-domain>, so the full redirect URI will be something like https://kiali-<namespace>.<app-domain>/api/auth/callback/<remote-cluster-name>, where <remote-cluster-name> is the Istio cluster name of the remote cluster.

For example, if Kiali is deployed in namespace istio-system with instance_name kiali, the app domain is apps.east.example.com, and the remote cluster’s Istio cluster name is west, the Kiali CR on the remote cluster should include:

spec:
  auth:
    openshift:
      redirect_uris:
        - https://kiali-istio-system.apps.east.example.com/api/auth/callback/west
  deployment:
    remote_cluster_resources_only: true

When using the openshift strategy with multiple clusters, users must be logged into each cluster in order to access resources on that cluster. The Kiali UI provides a mechanism to log into remote clusters:

In your browser, navigate to the Kiali UI and log in using your credentials for the cluster where Kiali is deployed.
Once logged in, use the user profile dropdown in the Kiali UI to initiate login to each remote cluster. Kiali will redirect you to the remote cluster’s OpenShift login page.
Log in with your credentials for that remote cluster. You will be redirected back to the Kiali UI. Repeat step 2 for each additional remote cluster until you are logged into all clusters.

Currently, OpenShift OAuth does not provide SSO across clusters. Each cluster requires its own login. If you are having trouble logging into a remote cluster from within Kiali, try starting a fresh private/incognito browser tab to ensure there are no stale OAuth cookies from prior logins to the remote cluster’s OpenShift console.

Using an internal or self-signed certificate

If you have a multi-cluster Kiali deployment and the OAuth server is configured with an external IdP that uses an internal or self-signed certificate, you can configure Kiali to trust the server’s certificate by creating a ConfigMap named kiali-oauth-cabundle containing the CA certificate bundle for the server under the oauth-server-ca.crt key:

Note that if you are deploying Kiali with spec.deployment.instance_name set to a value that is different than the default of kiali, your ConfigMap name needs to be that instance name appended with “-oauth-bundle”. For example, if your instance name is “myserver” then the name of the ConfigMap must be myserver-oauth-cabundle.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-oauth-cabundle
  namespace: istio-system # This is Kiali's install namespace
data:
  oauth-server-ca.crt: <PEM encoded CA root certificate>

Kiali will automatically trust this root certificate for all HTTPS requests (not just OAuth). The certificate is loaded into Kiali’s global certificate pool. Kiali watches for changes to the CA bundle and automatically refreshes without requiring a pod restart. If you have multiple different CAs, for different clusters, include each as a separate block in the bundle.

For most use cases, you can simply add your CA to the kiali-cabundle ConfigMap under the additional-ca-bundle.pem key instead of creating a separate kiali-oauth-cabundle ConfigMap. Both approaches result in the CA being trusted globally.

Insecure setting

You should only use this setting for testing and not in a production environment.

You can disable certificate validation between Kiali and the remote OAuth server(s) by setting insecure_skip_verify_tls to true in the Kiali CR:

spec:
  auth:
    openshift:
      insecure_skip_verify_tls: true

Docs: The Kiali CR

Mon, 01 Jan 0001 00:00:00 +0000

The Kiali Operator watches the Kiali Custom Resource (Kiali CR), a custom resource that contains the Kiali Server deployment configuration. Creating, updating, or removing a Kiali CR will trigger the Kiali Operator to install, update, or remove Kiali.

If you want the operator to re-process the Kiali CR (called “reconciliation”) without having to change the Kiali CR’s spec fields, you can modify any annotation on the Kiali CR itself. This will trigger the operator to reconcile the current state of the cluster with the desired state defined in the Kiali CR, modifying cluster resources if necessary to get them into their desired state. Here is an example illustrating how you can modify an annotation on a Kiali CR:

$ kubectl annotate kiali my-kiali -n istio-system --overwrite kiali.io/reconcile="$(date)"

The Operator provides comprehensive defaults for all properties of the Kiali CR. Hence, the minimal Kiali CR does not have a spec:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali

Assuming you saved the previous YAML to a file named my-kiali-cr.yaml, and that you are installing Kiali in the same default namespace as Istio, create the resource with the following command:

$ kubectl apply -f my-kiali-cr.yaml -n istio-system

Often, but not always, Kiali is installed in the same namespace as Istio, thus the Kiali CR is also created in the Istio namespace.

Once created, the Kiali Operator should shortly be notified and will process the resource, performing the Kiali installation. You can wait for the Kiali Operator to finish the reconcilation by using the standard kubectl wait command and ask for it to wait for the Kiali CR to achieve the condition of Successful. For example:

kubectl wait --for=condition=Successful kiali kiali -n istio-system

You can check the installation progress by inspecting the status attribute of the created Kiali CR:

$ kubectl describe kiali kiali -n istio-system
Name:         kiali
Namespace:    istio-system
Labels:       <none>
Annotations:  <none>
API Version:  kiali.io/v1alpha1
Kind:         Kiali

  (...some output is removed...)

Status:
  Conditions:
    Last Transition Time:  2021-09-15T17:17:40Z
    Message:               Running reconciliation
    Reason:                Running
    Status:                True
    Type:                  Running
  Deployment:
    Instance Name:  kiali
    Namespace:      istio-system
  Environment:
    Is Kubernetes:       true
    Kubernetes Version:  1.27.3
    Operator Version:    v1.89.0
  Progress:
    Duration:    0:00:16
    Message:     5. Creating core resources
  Spec Version:  default
Events:        <none>

Never manually edit resources created by the Kiali Operator; only edit the Kiali CR.

You may want to check the example install page to see some examples where the Kiali CR has a spec and to better understand its structure. Most available attributes of the Kiali CR are described in the pages of the Installation and Configuration sections of the documentation. For a complete list, see the Kiali CR Reference.

It is important to understand the spec.deployment.cluster_wide_access setting in the CR. See the Namespace Management page for more information.

Once you created a Kiali CR, you can manage your Kiali installation by editing the resource using the usual Kubernetes tools:

$ kubectl edit kiali kiali -n istio-system

To confirm your Kiali CR is valid, you can utilize the Kiali CR validation tool.

Docs: The OSSMConsole CR

Mon, 01 Jan 0001 00:00:00 +0000

OpenShift ServiceMesh Console (aka OSSMC) provides a Kiali integration with the OpenShift Console; in other words it provides Kiali functionality within the context of the OpenShift Console. OSSMC is applicable only within OpenShift environments.

The main component of OSSMC is a plugin that gets installed inside the OpenShift Console. Prior to installing this plugin, you are required to have already installed the Kiali Operator and Kiali Server in your OpenShift environment. Please the Installation Guide for details.

There are no helm charts available to install OSSMC. You must utilize the Kiali Operator to install it. Installing the Kiali Operator on OpenShift is very easy due to the Operator Lifecycle Manager (OLM) functionality that comes with OpenShift out-of-box. Simply elect to install the Kiali Operator from the Red Hat or Community Catalog from the OperatorHub page in OpenShift Console.

The Kiali Operator watches the OSSMConsole Custom Resource (OSSMConsole CR), a custom resource that contains the OSSMC deployment configuration. Creating, updating, or removing a OSSMConsole CR will trigger the Kiali Operator to install, update, or remove OSSMC.

Never manually edit resources created by the Kiali Operator, only edit the OSSMConsole CR.

Creating the OSSMConsole CR to Install the OSSMC Plugin

With the Kiali Operator and Kial Server installed and running, you can install the OSSMC plugin in one of two ways - either via the OpenShift Console or via the “oc” CLI. Both methods are described below. You choose the method you want to use.

You should specify the spec.version field of the OSSMConsole CR, and its value must be the same version as that of the Kiali Server (i.e. it must match the spec.version of the Kiali Server’s Kiali CR). Normally, you can just set spec.version to default which tells the Kiali Operator to install OSSMC whose version is the same as that of the operator itself. Alternatively, you may specify one of the supported versions in the format vX.Y.

Installing via OpenShift Console

From the Kiali Operator details page in the OpenShift Console, create an instance of the “OpenShift Service Mesh Console” resource. Accept the defaults on the installation form and press “Create”.

Installing via “oc” CLI

To instruct the Kiali Operator to install the plugin, simply create a small OSSMConsole CR. A minimal CR can be created like this:

cat <<EOM | oc apply -f -
apiVersion: kiali.io/v1alpha1
kind: OSSMConsole
metadata:
  namespace: openshift-operators
  name: ossmconsole
spec:
  version: default
EOM

Note that the operator will deploy the plugin resources in the same namespace where you create this OSSMConsole CR - in this case openshift-operators but you can create the CR in any namespace.

For a complete list of configuration options available within the OSSMConsole CR, see the OSSMConsole CR Reference.

To confirm your OSSMConsole CR is valid, you can utilize the OSSMConsole CR validation tool.

Installation Status

After the plugin is installed, you can see the “OSSMConsole” resource that was created in the OpenShift Console UI. Within the operator details page in the OpenShift Console UI, select the OpenShift Service Mesh Console tab to view the resource that was created and its status. The CR status field will provide you with any error messages should the deployment of OSSMC fail.

Once the operator has finished processing the OSSMConsole CR, you must then wait for the OpenShift Console to load and initialize the plugin. This may take a minute or two. You will know when the plugin is ready when the OpenShift Console pops up this message - when you see this message, refresh the browser window to reload the OpenShift Console:

Uninstalling OSSMC

This section will describe how to uninstall the OpenShift Service Mesh Console plugin. You can uninstall the plugin in one of two ways - either via the OpenShift Console or via the “oc” CLI. Both methods are described in the sections below. You choose the method you want to use.

If you intend to also uninstall the Kiali Operator, it is very important to first uninstall the OSSMConsole CR and then uninstall the operator. If you uninstall the operator before ensuring the OSSMConsole CR is deleted then you may have difficulty removing that CR and its namespace. If this occurs then you must manually remove the finalizer on the CR in order to delete it and its namespace. You can do this via: oc patch ossmconsoles <CR name> -n <CR namespace> -p '{"metadata":{"finalizers": []}}' --type=merge

Uninstalling via OpenShift Console

Remove the OSSMConsole CR by navigating to the operator details page in the OpenShift Console UI. From the operator details page, select the OpenShift Service Mesh Console tab and then select the Delete option in the kebab menu.

Uninstalling via “oc” CLI

Remove the OSSMConsole CR via oc delete ossmconsoles <CR name> -n <CR namespace>. To make sure any and all CRs are deleted from any and all namespaces, you can run this command:

for r in $(oc get ossmconsoles --ignore-not-found=true --all-namespaces -o custom-columns=NS:.metadata.namespace,N:.metadata.name --no-headers | sed 's/  */:/g'); do oc delete ossmconsoles -n $(echo $r|cut -d: -f1) $(echo $r|cut -d: -f2); done

Docs: Accessing Kiali

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

After Kiali is succesfully installed you will need to make Kiali accessible to users. This page describes some popular methods of exposing Kiali for use.

If exposing Kiali in a custom way, you may need to set some configurations to make Kiali aware of how users will access Kiali.

The examples on this page assume that you followed the Installation guide to install Kiali, and that you installed Kiali in the istio-system namespace.

Accessing Kiali using port forwarding

This method should work in any kind of Kubernetes cluster.

You can use port-forwarding to access Kiali by running any of these commands:

# If you have oc command line tool
oc port-forward svc/kiali 20001:20001 -n istio-system
# If you have kubectl command line tool
kubectl port-forward svc/kiali 20001:20001 -n istio-system

These commands will block. Access Kiali by visiting https://localhost:20001/ in your preferred web browser.

Please note that this method exposes Kiali only to the local machine, no external users. You must have the necessary privileges to perform port forwarding.

Accessing Kiali through an Ingress

You can configure Kiali to be installed with an Ingress resource defined, allowing you to access the Kiali UI through the Ingress. By default, an Ingress will not be created. You can enable a simple Ingress by setting spec.deployment.ingress.enabled to true in the Kiali CR (a similar setting for the server Helm chart is available if you elect to install Kiali via Helm as opposed to the Kiali Operator).

Exposing Kiali externally through this spec.deployment.ingress mechanism is a convenient way of exposing Kiali externally but it will not necessarily work or be the best way to do it because the way in which you should expose Kiali externally will be highly dependent on your specific cluster environment and how services are exposed generally for that environment.

When installing on an OpenShift cluster, an OpenShift Route will be installed (not an Ingress). This Route will be installed by default unless you explicitly disable it via spec.deployment.ingress.enabled: false. Note that the Route is required if you configure Kiali to use the auth strategy of openshift (which is the default auth strategy Kiali will use when installed on OpenShift).

The default Ingress that is created will be configured for a typical NGinx implementation. If you have your own Ingress implementation you want to use, you can override the default configuration through the settings spec.deployment.ingress.override_yaml and spec.deployment.ingress.class_name. More details on customizing the Ingress can be found below.

The Ingress IP or domain name should then be used to access the Kiali UI. To find your Ingress IP or domain name, as per the minikube documentation, try the following command (though this may not work if using Minikube without the ingress addon):

kubectl get ingress kiali -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

If it doesn’t work, unfortunately, it depends on how and where you had setup your cluster. There are several Ingress controllers available and some cloud providers have their own controller or preferred exposure method. Check the documentation of your cloud provider. You may need to customize the pre-installed Ingress rule or expose Kiali using a different method.

Customizing the Ingress resource

The created Ingress resource will route traffic to Kiali regardless of the domain in the URL. You may need a more specific Ingress resource that routes traffic to Kiali only on a specific domain or path. To do this, you can specify route settings.

Alternatively, and for more advanced Ingress configurations, you can provide your own Ingress declaration in the Kiali CR. For example:

When installing on an OpenShift cluster, the deployment.ingress.override_yaml will be applied to the created Route. The deployment.ingress.class_name is ignored on OpenShift.

spec:
  deployment:
    ingress:
      class_name: "nginx"
      enabled: true
      override_yaml:
        metadata:
          annotations:
            nginx.ingress.kubernetes.io/secure-backends: "true"
            nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
        spec:
          rules:
            - http:
                paths:
                  - path: /kiali
                    backend:
                      serviceName: kiali
                      servicePort: 20001

Accessing Kiali in Minikube

If you enabled the Ingress addon, the default Ingress resource created by the installation (mentioned in the previous section) should be enough to access Kiali. The following command should open Kiali in your default web browser:

xdg-open https://$(minikube ip)/kiali

Accessing Kiali through a LoadBalancer or a NodePort

By default, the Kiali service is created with the ClusterIP type. To use a LoadBalancer or a NodePort, you can change the service type in the Kiali CR as follows:

spec:
  deployment:
    service_type: LoadBalancer

Once the Kiali operator updates the installation, you should be able to use the kubectl get svc -n istio-system kiali command to retrieve the external address (or port) to access Kiali. For example, in the following output Kiali is assigned the IP 192.168.49.201, which means that you can access Kiali by visiting http://192.168.49.201:20001 in a browser:

NAME    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                          AGE
kiali   LoadBalancer   10.105.236.127   192.168.49.201   20001:31966/TCP,9090:30128/TCP   34d

If you are using the LoadBalancer service type to directly expose the Kiali service, you may want to check the available options for the HTTP Server and Metrics server.

Accessing Kiali through an Istio Ingress Gateway

If you want to take advantage of Istio’s infrastructure, you can expose Kiali using an Istio Ingress Gateway. The Istio documentation provides a good guide explaining how to expose the sample add-ons. Even if the Istio guide is focused on the sample add-ons, the steps are the same to expose a Kiali installed using this Installation guide.

Accessing Kiali in OpenShift

By default, Kiali is exposed through a Route if installed on OpenShift. The following command should open Kiali in your default web browser:

xdg-open https://$(oc get routes -n istio-system kiali -o jsonpath='{.spec.host}')/console

Specifying route settings

If you are using your own exposure method or if you are using one of the methods mentioned in this page, you may need to configure the route that is being used to access Kiali.

In the Kiali CR, route settings are broken in several attributes. For example, to specify that Kiali is being accessed under the https://apps.example.com:8080/dashboards/kiali URI, you would need to set the following:

spec:
  server:
    web_fqdn: apps.example.com
    web_port: 8080
    web_root: /dashboards/kiali
    web_schema: https

If you are letting the installation create an Ingress resource for you, the Ingress will be adjusted to match these route settings. If you are using your own exposure method, these spec.server settings are only making Kiali aware of what its public endpoint is.

It is possible to omit these settings and Kiali may be able to discover some of these configurations, depending on your exposure method. For example, if you are exposing Kiali via LoadBalancer or NodePort service types, Kiali can discover most of these settings. If you are using some kind of Ingress, Kiali will honor X-Forwarded-Proto, X-Forwarded-Host and X-Forwarded-Port HTTP headers if they are properly injected in the request.

The web_root receives special treatment, because this is the path where Kiali will serve itself (both the user interface and its api). This is useful if you are serving multiple applications under the same domain. It must begin with a slash and trailing slashes must be omitted. The default value is /kiali for Kubernetes and / for OpenShift.

Usually, these settings can be omitted. However, a few features require that the Kiali’s public route be properly discoverable or that it is properly configured; most notably, the OpenID authentication.

Docs: Token strategy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The token authentication strategy allows a user to login to Kiali using the token of a Kubernetes ServiceAccount. This is similar to the login view of Kubernetes Dashboard.

The token strategy supports namespace access control.

The token strategy is only supported for single cluster.

Set-up

Since token is the default strategy when deploying Kiali in Kubernetes, you shouldn’t need to configure anything, unless your cluster is OpenShift. If you want to be verbose or if you need to enable the token strategy in OpenShift, use the following configuration in the Kiali CR:

spec:
  auth:
    strategy: token

The token strategy doesn’t have any additional configuration other than the session expiration time.

Docs: Advanced Install

Mon, 01 Jan 0001 00:00:00 +0000

Canary upgrades

During a canary upgrade where multiple controlplanes are present, Kiali will automatically detect both controlplanes. You can visit the mesh page to visualize your controlplanes during a canary upgrade.

Installing a Kiali Server of a different version than the Operator

When you install the Kiali Operator, it will be configured to install a Kiali Server that is the same version as the operator itself. For example, if you have Kiali Operator v1.34.0 installed, that operator will install Kiali Server v1.34.0. If you upgrade (or downgrade) the Kiali Operator, the operator will in turn upgrade (or downgrade) the Kiali Server.

There are certain use-cases in which you want the Kiali Operator to install a Kiali Server whose version is different than the operator version. Read the following section «Using a custom image registry» section to learn how to configure this setup.

Using a custom image registry

Kiali is released and published to the Quay.io container image registry. There is a repository hosting the Kiali operator images and another one for the Kiali server images.

If you need to mirror the Kiali container images to some other registry, you still can use Helm to install the Kiali operator as follows:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    --set image.repo=your.custom.registry/owner/kiali-operator-repo
    --set image.tag=your_custom_tag
    --set allowAdHocKialiImage=true
    kiali-operator \
    kiali/kiali-operator

Notice the --set allowAdHocKialiImage=true which allows specifying a custom image in the Kiali CR. For security reasons, this is disabled by default.

Then, when creating the Kiali CR, use the following attributes:

spec:
  deployment:
    image_name: your.custom.registry/owner/kiali-server-repo
    image_version: your_custom_tag

Change the default image

As explained earlier, when you install the Kiali Operator, it will be configured to install a Kiali Server whose image will be pulled from quay.io and whose version will be the same as the operator. You can ask the operator to use a different image by setting spec.deployment.image_name and spec.deployment.image_version within the Kiali CR (as explained above).

However, you may wish to alter this default behavior exhibited by the operator. In other words, you may want the operator to install a different Kiali Server image by default. For example, if you have an air-gapped environment with its own image registry that contains its own copy of the Kiali Server image, you will want the operator to install a Kiali Server that uses that image by default, as opposed to quay.io/kiali/kiali. By configuring the operator to do this, you will not force the authors of Kiali CRs to have to explicitly define the spec.deployment.image_name setting and you will not need to enable the allowAdHocKialiImage setting in the operator.

To change the default Kiali Server image installed by the operator, set the environment variable RELATED_IMAGE_kiali_default in the Kiali Operator deployment. The value of that environment variable must be the full image tag in the form repoName/orgName/imageName:versionString (e.g. my.internal.registry.io/mykiali/mykialiserver:v1.50.0). You can do this when you install the operator via helm:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    --set "env[0].name=RELATED_IMAGE_kiali_default" \
    --set "env[0].value=my.internal.registry.io/mykiali/mykialiserver:v1.50.0" \
    kiali-operator \
    kiali/kiali-operator

Development Install

This option installs the latest Kiali Operator and Kiali Server images which are built from the master branches of Kiali GitHub repositories. This option is good for demo and development installations.

helm install \
  --set cr.create=true \
  --set cr.namespace=istio-system \
  --set cr.spec.deployment.image_version=latest \
  --set image.tag=latest \
  --namespace kiali-operator \
  --create-namespace \
  kiali-operator \
  kiali/kiali-operator

Docs: Session options

Mon, 01 Jan 0001 00:00:00 +0000

There are two settings that are available for the user’s session. The first one is the session expiration time, which is only applicable to token and header authentication strategies:

spec:
  login_token:
    # By default, users session expires in 24 hours.
    expiration_seconds: 86400

The session expiration time is the amount of time before the user is asked to extend his session by another cycle. It does not matter if the user is actively using Kiali, the user will be asked if the session should be extended.

The second available option is the signing key configuration, which is unset by default, meaning that a random 16-character signing key will be generated and stored to a secret named kiali-signing-key, in Kiali’s installation namespace:

spec:
  login_token:
    # By default, create a random signing key and store it in
    # a secret named "kiali-signing-key".
    signing_key: ""

If the secret already exists (which may mean a previous Kiali installation was present), then the secret is reused.

The signing key is used on security sensitive data. For example, one of the usages is to sign HTTP cookies related to the user session to prevent session forgery.

If you need to set a custom fixed key, you can pre-create or modify the kiali-signing-key secret:

apiVersion: v1
kind: Secret
metadata:
  namespace: "kiali-installation-namespace"
  name: kiali-signing-key
type: Opaque
data:
  key: "<your signing key encoded in base64>"

The signing key must be 16, 24 or 32 bytes length. Otherwise, Kiali will fail to start.

If you prefer a different secret name for the signing key and/or a different key-value pair of the secret, you can specify your preferred names in the Kiali CR:

spec:
  login_token:
    signing_key: "secret:<secretName>:<secretDataKey>"

It is possible to specify the signing key directly in the Kiali CR, in the spec.login_token.signing_key attribute. However, this should be only for testing purposes. The signing key is sensitive and should be treated like a password that must be protected.

Docs: Example Install

Mon, 01 Jan 0001 00:00:00 +0000

This is a quick example of installing Kiali. This example will install the operator and two Kiali Servers - one server will require the user to enter credentials at a login screen in order to obtain read-write access and the second server will allow anonymous read-only access.

For this example, assume there is a Minikube Kubernetes cluster running with an Istio control plane installed in the namespace istio-system and the Istio Bookinfo Demo installed in the namespace bookinfo:

$ kubectl get deployments.apps -n istio-system
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
grafana                1/1     1            1           8h
istio-egressgateway    1/1     1            1           8h
istio-ingressgateway   1/1     1            1           8h
istiod                 1/1     1            1           8h
jaeger                 1/1     1            1           8h
prometheus             1/1     1            1           8h

$ kubectl get deployments.apps -n bookinfo
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
details-v1       1/1     1            1           21m
productpage-v1   1/1     1            1           21m
ratings-v1       1/1     1            1           21m
reviews-v1       1/1     1            1           21m
reviews-v2       1/1     1            1           21m
reviews-v3       1/1     1            1           21m

Install Kiali Operator via Helm Chart

First, the Kiali Operator will be installed in the kiali-operator namespace using the operator helm chart:

$ helm repo add kiali https://kiali.org/helm-charts
$ helm repo update kiali
$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator

Install Kiali Server via Operator

Next, the first Kiali Server will be installed. This server will require the user to enter a Kubernetes token in order to log into the Kiali dashboard and will provide the user with read-write access. To do this, a Kiali CR will be created that looks like this (file: kiali-cr-token.yaml):

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: "token"
  deployment:
    cluster_wide_access: false
    discovery_selectors:
      default:
      - matchLabels:
          kubernetes.io/metadata.name: bookinfo
    view_only_mode: false
  server:
    web_root: "/kiali"

This Kiali CR will command the operator to deploy the Kiali Server in the same namespace where the Kiali CR is (istio-system). The operator will configure the server to: respond to requests to the web root path of /kiali, enable read-write access, use the authentication strategy of token, and be given access to the bookinfo namespace:

$ kubectl apply -f kiali-cr-token.yaml

Get the Status of the Installation

The status of a particular Kiali Server installation can be found by examining the status field of its corresponding Kiali CR. For example:

$ kubectl get kiali kiali -n istio-system -o jsonpath='{.status}'

When the installation has successfully completed, the status field will look something like this (when formatted):

$ kubectl get kiali kiali -n istio-system -o jsonpath='{.status}' | jq
{
  "conditions": [
    {
      "ansibleResult": {
        "changed": 21,
        "completion": "2021-10-20T19:17:35.519131",
        "failures": 0,
        "ok": 102,
        "skipped": 90
      },
      "lastTransitionTime": "2021-10-20T19:17:12Z",
      "message": "Awaiting next reconciliation",
      "reason": "Successful",
      "status": "True",
      "type": "Running"
    }
  ],
  "deployment": {
    "discoverySelectorNamespaces": "bookinfo,istio-system",
    "instanceName": "kiali",
    "namespace": "istio-system"
  },
  "environment": {
    "isKubernetes": true,
    "kubernetesVersion": "1.28.0",
    "operatorVersion": "v1.88.0"
  },
  "progress": {
    "duration": "0:00:14",
    "message": "7. Finished all resource creation"
  }
}

Access the Kiali Server UI

The Kiali Server UI is accessed by pointing a browser to the Kiali Server endpoint and requesting the web root /kiali:

xdg-open http://$(minikube ip)/kiali

Because the auth.strategy was set to token, that URL will display the Kiali login screen that will require a Kubernetes token in order to authenticate with the server. For this example, you can use the token that belongs to the Kiali service account itself:

$ kubectl get secret -n istio-system $(kubectl get sa kiali-service-account -n istio-system -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 -d

The output of that command above can be used to log into the Kiali login screen.

Install a Second Kiali Server

The second Kiali Server will next be installed. This server will not require the user to enter any login credentials but will only provide a view-only look at the service mesh. To do this, a Kiali CR will be created that looks like this (file: kiali-cr-anon.yaml):

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: kialianon
spec:
  installation_tag: "Kiali - View Only"
  auth:
    strategy: "anonymous"
  deployment:
    cluster_wide_access: false
    discovery_selectors:
      default:
      - matchLabels:
          kubernetes.io/metadata.name: bookinfo
    view_only_mode: true
    instance_name: "kialianon"
  server:
    web_root: "/kialianon"

This Kiali CR will command the operator to deploy the Kiali Server in the same namespace where the Kiali CR is (kialianon). The operator will configure the server to: respond to requests to the web root path of /kialianon, disable read-write access, not require the user to authenticate, have a unique instance name of kialianon and be given access to the bookinfo namespace. The Kiali UI will also show a custom title in the browser tab so the user is aware they are looking at a “view only” Kiali dashboard. The unique deployment.instance_name is needed in order for this Kiali Server to be able to share access to the Bookinfo application with the first Kiali Server.

$ kubectl create namespace kialianon
$ kubectl apply -f kiali-cr-anon.yaml

The UI for this second Kiali Server is accessed by pointing a browser to the Kiali Server endpoint and requesting the web root /kialianon. Note that no credentials are required to gain access to this Kiali Server UI because auth.strategy was set to anonymous; however, the user will not be able to modify anything via the Kiali UI - it is strictly “view only”:

xdg-open http://$(minikube ip)/kialianon

Reconfigure Kiali Server

A Kiali Server can be reconfigured by simply editing its Kiali CR. The Kiali Operator will perform all the necessary tasks to complete the reconfiguration and reboot the Kiali Server pod when necessary. For example, to change the web root for the Kiali Server:

$ kubectl patch kiali kiali -n istio-system --type merge --patch '{"spec":{"server":{"web_root":"/specialkiali"}}}'

The Kiali Operator will update the necessary resources (such as the Kiali ConfigMap) and will reboot the Kiali Server pod to pick up the new configuration.

Uninstall Kiali Server

To uninstall a Kiali Server installation, simply delete the Kiali CR. The Kiali Operator will then perform all the necessary tasks to remove all remnants of the associated Kiali Server.

kubectl delete kiali kiali -n istio-system

Uninstall Kiali Operator

To uninstall the Kiali Operator, use helm uninstall and then manually remove the Kiali CRD.

You must delete all Kiali CRs in the cluster prior to uninstalling the Kiali Operator. If you fail to do this, uninstalling the operator will hang and remnants of Kiali Server installations will remain in your cluster and you will be required to perform some manual steps to clean it up.

$ kubectl delete kiali --all --all-namespaces
$ helm uninstall --namespace kiali-operator kiali-operator
$ kubectl delete crd kialis.kiali.io