Kiali – Features

Docs: Application Wizards

Mon, 01 Jan 0001 00:00:00 +0000

Istio Application Wizards

Kiali provides Actions to create, update and delete Istio configuration, driven by wizards.

Actions can be applied to a Service

Actions can also be applied to a Workload

And, actions are available for an entire Namespace

Service Actions

Kiali offers a robust set of service actions, with accompanying wizards.

Traffic Management: Request Routing

The Request Routing Wizard allows creating multiple routing rules.

Every rule is composed of a Request Matching and a Routes To section.
The Request Matching section can add multiple filters using HEADERS, URI, SCHEME, METHOD or AUTHORITY HTTP parameters.
The Request Matching section can be empty, in this case any HTTP request received is matched against this rule.
The Routes To section can specify the percentage of traffic that is routed to a specific workload.

Istio applies routing rules in order, meaning that the first rule matching an HTTP request (top-down) performs the routing. The Matching Routing Wizard allows changing the rule order.

Traffic Management: Fault Injection

The Fault Injection Wizard allows injecting faults to test the resiliency of a Service.

HTTP Delay specification is used to inject latency into the request forwarding path.
HTTP Abort specification is used to immediately abort a request and return a pre-specified status code.

Traffic Management: Traffic Shifting

The Traffic Shifting Wizard allows selecting the percentage of traffic that is routed to a specific workload.

Kiali also provides an analogous action for TCP traffic shifting.

Traffic Management: Request Timeouts

The Request Timeouts Wizard sets up request timeouts in Envoy, using Istio.

HTTP Timeout defines the timeout for a request.
HTTP Retry describes the retry policy to use when an HTTP request fails.

Traffic Management: Gateways

Traffic Management Wizards have an Advanced Options section that can be used to extend the scenario.

One available Advanced Option is to expose a Service to external traffic through an existing Gateway or to create a new Gateway for this Service.

Traffic Management: Circuit Breaker

Traffic Management Wizards allows defining Circuit Breakers on Services as part of the available Advanced Options.

Connection Pool defines the connection limits for an upstream host.
Outlier Detection implements the Circuit Breaker based on the consecutive errors reported.

Routing Rules Preview

Kiali provides a safe preview environment where users can review the complete YAML definition of the routing configuration and edit the configuration inline before creating.

Security: Traffic Policy

Traffic Management Advanced Options allows defining Security and Load Balancing settings.

TLS related settings for connections to the upstream service.
Automatically generate a PeerAuthentication resource for this Service.
Load balancing policies to apply for a specific destination.

Workload Actions

Automatic Sidecar Injection

A Workload can be individually managed to control the Sidecar Injection.

A default scenario is to indicate this at Namespace level but there can be cases where a Workload shouldn’t be part of the Mesh or vice versa.

Kiali allows users to alter the Deployment template and propagate this configuration into the Pods.

Namespace Actions

The Kiali Namespaces page (Kiali >= 2.23) offers several Namespace actions.

Show

Show actions navigate from a Namespace to its specific Graph, Applications, Workloads, Services or Istio Config pages.

Automatic Sidecar Injection

When Automatic Sidecar Injection is enabled in the cluster, a Namespace can be labeled to enable/disable the injection webhook, controlling whether new deployments will automatically have a sidecar.

Canary Istio upgrade

When Istio Canary revision is installed, a Namespace can be labeled to that canary revision, so the sidecar of canary revision will be injected into workloads of the namespace.

Security: Traffic Policies

Kiali can generate Traffic Policies based on the traffic for a namespace.

For example, at some point a namespace presents a traffic graph like this:

And a user may want to add Traffic Policies to secure that communication. In other words, to prevent traffic other than that currently reflected in the Graph’s Services and Workloads.

Using the Create Traffic Policies action on a namespace, Kiali will generate AuthorizationPolicy resources per every Workload in the Namespace.

Docs: Detail Views

Mon, 01 Jan 0001 00:00:00 +0000

Kiali provides filtered list views of all your service mesh definitions. Each view provides health, details, YAML definitions and links to help you visualize your mesh. There are list and detail views for:

Applications
Istio Configuration
Services
Workloads

Selecting an object from the list will bring you to its detail page. For Istio Config, Kiali will present its YAML, along with contextual validation information. Other mesh components present a variety of Tabs.

Overview Tab

Overview is the default Tab for any detail page. The overview tab provides detailed information, including health status, and a detailed mini-graph of the current traffic involving the component. The full set of tabs, as well as the detailed information, varies based on the component type.

Each Overview provides:

links to related components and linked Istio configuration.
health status.
validation information.
an Action menu for actions that can be taken on the component.
- several Wizards are available.

And also type-specfic information. For example:

Service detail includes Network information.
Workload detail provides backing Pod information.

Both Workload and Service detail can be customized to some extent, by adding additional details supplied as annotations. This is done through the additional_display_details field in the Kiali CR.

Traffic

The Traffic Tab presents a service, app, or workload’s Inbound and Outbound traffic in a table-oriented way:

Logs

Workload detail offers a special Logs tab. Kiali offers a special unified logs view that lets users correlate app and proxy logs. It can also add-in trace span information to help identify important traces based on associated logging. More powerful features include substring or regex Show/Hide, full-screen, and the ability to set proxy log level without a pod restart.

Metrics

Each detail view provides Inbound Metrics and/or Outbound Metrics tabs, offering predefined metric dashboards. The dashboards provided are tailored to the relevant application, workload or service level. Application and workload detail views show request and response metrics such as volume, duration, size, or tcp traffic. The service detail view shows request and response metrics for inbound traffic only.

Kiali allows the user to customize the charts by choosing the charted dimensions. It can also present metrics reported by either source or destination proxy metrics. And for troublshooting it can overlay trace spans.

Traces

Each detail view provides a Traces tab with a native integration with Jaeger. For more, see Tracing.

Dashboards

Kiali will display additional tabs for each applicable Built-In Dashboard or Custom Dashboard.

Built-in dashboards

Kiali comes with built-in dashboards for several runtimes, including Envoy, Go, Node.js, and others.

Envoy

The most important built-in dashboard is for Envoy. Kiali offers the Envoy tab for many workloads. The Envoy tab is actually a Built-In Dashboard, but it is very common as it applies to any workload injected with, or that is itself, an Envoy proxy. Being able to inspect the Envoy proxy is invaluable when troublshooting your mesh. The Envoy tab itself offers five subtabs, exposing a wealth of information.

Istio’s Envoy sidecars supply some internal metrics, that can be viewed in Kiali. They are different than the metrics reported by Istio Telemetry, which Kiali uses extensively. Some of Envoy’s metrics may be redundant.

Note that the enabled Envoy metrics can be tuned, as explained in the Istio documentation: it’s possible to get more metrics using the statsInclusionPrefixes annotation. Make sure you include cluster_manager and listener_manager as they are required.

For example, sidecar.istio.io/statsInclusionPrefixes: cluster_manager,listener_manager,listener will add listener metrics for more inbound traffic information. You can then customize the Envoy dashboard of Kiali according to the collected metrics.

Go

Contains metrics such as the number of threads, goroutines, and heap usage. The expected metrics are provided by the Prometheus Go client.

Example to expose built-in Go metrics:

        http.Handle("/metrics", promhttp.Handler())
        http.ListenAndServe(":2112", nil)

As an example and for self-monitoring purpose Kiali itself exposes Go metrics.

The pod annotation for Kiali is: kiali.io/dashboards: go

Kiali

Kiali has its own built-in dashboard that helps you observe performance of the Kiali server itself. To view this dashboard, navigate to either the application or workload view of the Kiali server and select the Kiali Internal Metrics tab to see Kiali’s own internal metrics:

Node.js

Contains metrics such as active handles, event loop lag, and heap usage. The expected metrics are provided by prom-client.

Example of Node.js metrics for Prometheus:

const client = require('prom-client');
client.collectDefaultMetrics();
// ...
app.get('/metrics', (request, response) => {
  response.set('Content-Type', client.register.contentType);
  response.send(client.register.metrics());
});

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/ratings

The pod annotation for Kiali is: kiali.io/dashboards: nodejs

Quarkus

Contains JVM-related, GC usage metrics. The expected metrics can be provided by SmallRye Metrics, a MicroProfile Metrics implementation. Example with maven:

    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-smallrye-metrics</artifactId>
    </dependency>

The pod annotation for Kiali is: kiali.io/dashboards: quarkus

Spring Boot

Three dashboards are provided: one for JVM memory / threads, another for JVM buffer pools and the last one for Tomcat metrics. The expected metrics come from Spring Boot Actuator for Prometheus. Example with maven:

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-core</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/details

The pod annotation for Kiali with the full list of dashboards is: kiali.io/dashboards: springboot-jvm,springboot-jvm-pool,springboot-tomcat

By default, the metrics are exposed on path /actuator/prometheus, so it must be specified in the corresponding annotation: prometheus.io/path: "/actuator/prometheus"

Thorntail

Contains mostly JVM-related metrics such as loaded classes count, memory usage, etc. The expected metrics are provided by the MicroProfile Metrics module. Example with maven:

    <dependency>
      <groupId>io.thorntail</groupId>
      <artifactId>microprofile-metrics</artifactId>
    </dependency>

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/productpage

The pod annotation for Kiali is: kiali.io/dashboards: thorntail

Vert.x

Several dashboards are provided, related to different components in Vert.x: HTTP client/server metrics, Net client/server metrics, Pools usage, Eventbus metrics and JVM. The expected metrics are provided by the vertx-micrometer-metrics module. Example with maven:

    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-micrometer-metrics</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

Init example of Vert.x metrics, starting a dedicated server (other options are possible):

      VertxOptions opts = new VertxOptions().setMetricsOptions(new MicrometerMetricsOptions()
          .setPrometheusOptions(new VertxPrometheusOptions()
              .setStartEmbeddedServer(true)
              .setEmbeddedServerOptions(new HttpServerOptions().setPort(9090))
              .setPublishQuantiles(true)
              .setEnabled(true))
          .setEnabled(true));

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/reviews

The pod annotation for Kiali with the full list of dashboards is: kiali.io/dashboards: vertx-client,vertx-server,vertx-eventbus,vertx-pool,vertx-jvm

Custom Dashboards

When the built-in dashboards don’t offer what you need, it’s possible to create your own. See Custom Dashboard Configuration for more information.

Docs: Health

Mon, 01 Jan 0001 00:00:00 +0000

Kiali help users know whether their service mesh is healthy. This includes the health of the mesh infrastructure itself, and the deployed application services.

Service Mesh Infrastructure Health

Users can quickly confirm the health of their infrastructure by looking at the Kiali Masthead. If Kiali detects any health issues with the infrastructure of the mesh, including multi-cluster setups, it will show an indication in the masthead, severity will be reflected via color, and hovering will show the detail:

For more detail on how Kiali tracks the Istio infrastructure status, see the Istio Status Feature.

Overview Health

The default Kiali page is an Overview Dashboard. This view will quickly allow you to identify components with issues, including clusters, Istio configuration, control planes and data planes. It provides a chart showing all applications grouped by health, and Service Insights showing the services with the top error rates and p95 latencies.

Graph Health

The Kiali Graph offers a rich visualization of your service mesh traffic. The health of Nodes and Edges is represented via a standard color system using shades of orange and red to reflect degraded and failure-level traffic health. Red or orange nodes or edges may need attention. The color of an edge represents the request health between the relevant nodes. Note that node shape indicates the type of component, such as service, workload, or app.

The health of nodes and edges is refreshed automatically based on the user’s desired refresh interval. The graph can also be paused to examine a particular state, or replayed to re-examine a particular time period.

Health Configuration

Kiali calculates health by combining the individual health of several indicators, such as pods and request traffic. The global health of a resource reflects the most severe health of its indicators.

Health Indicators

The table below lists the current health indicators and whether the indicator supports custom configuration for its health calculation.

Indicator	Supports Configuration
Pod Status	No
Traffic Health	Yes

Icons and colors

Kiali uses icons and colors to indicate the health of resources and associated request traffic.

No Health Information (NA)
Healthy
Degraded
Failure

Custom Request Health

There are times when Kiali’s default thresholds for traffic health do not work well for a particular situation. For example, at times 404 response codes are expected. Kiali has the ability to set powerful, fine-grained overrides for health configuration. For details, see Traffic Health Configuration.

Docs: Internationalization

Mon, 01 Jan 0001 00:00:00 +0000

Kiali is used worldwide and some users prefer to display Kiali in a language that they are more comfortable with than English. For this reason Kiali supports internationalization, and it can be localized into multiple languages.

Current supported languages are English, Chinese and Spanish.

Language Selector

Kiali provides a language selector in the Masthead to be able to switch Kiali between supported languages:

By default the language selector is hidden, and English is the default language.

In order to show the language selector a feature flag must be enabled in the Kiali CR:

spec:
  kiali_feature_flags:
    ui_defaults:
      i18n:
        language: en
        show_selector: true

You can also modify the default language shown in Kiali (values according to ISO 639-1 codes):

Language	Code
Chinese	zh
English	en
Spanish	es

As an example, this is how Kiali displays the Overview page in Spanish:

If you want to collaborate with us on adding a new language or improving the translation of an existing one, please refer to the internationalization section of the kiali project’s README for UI

Docs: Istio Ambient Mesh

Mon, 01 Jan 0001 00:00:00 +0000

Kiali provides visualization for Ambient Mesh components:

The Kiali Ambient features, as well as Ambient Mesh, are evolving. Some of these features are in alpha status. For enhancements or detected issues, don’t hesitate to open a GitHub issue.

Control Plane Ambient Mesh

When the control plane is in Ambient mode, Kiali will show an Ambient badge on the Namespaces page, for the control plane namespace, in the Type column. It will also be reflected in the control plane side-panel on the Mesh page. This badge indicates that Kiali has detected a ztunnel (the L4 component for Ambient) in the control plane namespace.

For Kiali to detect Ambient, it needs to have access to the namespace where ztunnel is deployed. This is usually the Istio control plane namespace (often istio-system), but on platforms such as OpenShift, it may differ.

Ambient Namespace

When a namespace is labeled with istio.io/dataplane-mode=ambient, it is enrolled in Ambient Mesh. On the Namespaces page, Kiali will show the number of ambient data planes, as well as the number of sidecar data planes.

Workloads in Ambient Mesh

When a workload, application, or service is part of the Ambient Mesh, a badge will appear in the namespace details. When hovering over this badge, further information about the workload will be displayed:

In Mesh: Indicating that it was included in Ambient, and the traffic is redirected to ztunnel to provide L4 features (L4 authorization and telemetry, and encrypted data transport)

In Mesh with waypoint enabled: Additionally, it can include the L7 badge which means that a waypoint proxy is deployed (providing additional L7 capabilities):

It is possible to check each pod protocol in the information tooltip. In Ambient, instead of TCP, it uses HBONE.

When the workload traffic is handled by a waypoint, the workload details will show a link to the proxy:

Kiali will correlate the ztunnel and waypoint logs related to the application and provide a checkbox to include them with the application logs. This ensures that all relevant information for the application is available in one place:

Waypoint proxy details

The workload details for a waypoint has specific waypoint data. It is identified with the L7 label:

The proxy status shows a new info message when some of the Discovery Services are IGNORED, and there are no other errors:

This condition is usually expected, but it is shown as an info in case it is not.

The waypoint proxy generates traces for the services for which it handles traffic, and this is where it can be checked, because the proxy generates the traces with the waypoint service name:

Waypoint proxies have a specific tab to show information about the Services and Workloads enrolled. The Labeled by label identifies where the waypoint label was added. It can be in the namespace, in the service or the workload.

For waypoint proxies, it is also possible to see the Envoy tab:

Ztunnel details

The workload details for a ztunnel workload has specific data. It has a new ztunnel tab containing the configuration for the services and workloads for which it handles traffic. It shows the same information that can be seen using the istioctl ztunnel-config, which can be useful for troubleshooting.

Ambient Telemetry

The Traffic graph generated with Ambient telemetry differs slightly from the usual graph, as the HTTP traffic and TCP traffic have different reporters.

The telemetry reported with sidecars represents the kind of traffic for the request (green edges for HTTP, blue edges for TCP). In Ambient, this information depends on the element reporting the Telemetry. The ztunnel will report all the traffic as TCP:

The following bookinfo namespace is in Ambient Mesh with a waypoint proxy enabled. Therefore, the telemetry is reported from ztunnel and from the waypoint, resulting in double edges connecting different nodes (Note that the Graph page toolbar offers a Traffic menu, letting you be selective about the protocols shown):

It is possible to filter the traffic by the Ambient reporter (ztunnel or waypoint) from the Traffic menu option:

There is an additional display option, waypoint proxies for the Ambient Mesh, that will display the waypoint proxies in the graph:

The waypoint proxies often serve as both the source and destination of traffic within the same workload, represented in the graph by bidirectional edges. When you click on an edge, the summary panel will display the waypoint proxy as the destination workload. However, you can also view the waypoint as the source by clicking on the double arrow icon located to the left of the “From/To” labels in the summary panel.

When the ingress waypoint routing is enabled on a service (istio.io/ingress-use-waypoint=true), the traffic goes from the gateway to the waypoint, instead of going to the service. In that case, the gateway node will show the waypoint icon:

Ambient Tracing

Ambient traces are emitted from the waypoint proxies. The traces involving a workload can be found looking for the waypoint service name.

In order to correlate the waypoint traces from a specific workload, app or service, Kiali looks for traces of the waypoint proxy in which the workload is enrolled. It then filters the traces related to the application using the operation name from the span.

The same approach is used to show the spans in the workload logs:

Also for the inbound and outbound metrics:

As the workload name is not part of the trace information, there are some gaps in the trace overlay (These gaps will hopefully be fixed in upstream Istio in a future Istio release - see this GitHub issue for details on that enhancement request). Also, for the workload view, there might be traces that are not part of a particular workload, but they are shown because they match the service name of the workload.

Starting with Istio 1.28, traces are reported using the service name instead of the waypoint name. Starting with Kiali 2.22, there is a configuration option, external_services.tracing.use_waypoint_name (disabled by default), that allows using the waypoint name as the service used for trace lookup.

Docs: Istio Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Kiali is more than observability, it also helps you to configure, update and validate your Istio service mesh.

The Istio configuration view provides advanced filtering and navigation for Istio configuration objects, such as Virtual Services and Gateways. Kiali provides inline config editing and powerful semantic validation for Istio resources.

Validations

Kiali performs a set of validations on your Istio Objects, such as Destination Rules, Service Entries, and Virtual Services. Kiali’s validations go above and beyond what Istio offers. Where Istio offers mainly static checks for well-formed definitions, Kiali performs semantic validations to ensure that the definitions make sense, across objects, and in some cases even across namespaces. Kiali validations are based on the runtime status of your service mesh.

Check the complete list of validations for further information.

Wizards

Kiali Wizards provide a way to apply an Istio service mesh pattern, letting Kiali generate the Istio Configuration. Wizards are launched from Kiali’s Action menus, located across the UI on relevant pages. Wizards can create and update Istio Config for Routing, Gateways, Security scenarios and more.

Istio Config Page Wizards

These Create Actions are available on the Istio Config page:

Authorization Wizards

Kiali allows creation of Istio AuthorizationPolicy resources:

Istio PeerAuthentication resources:

Istio RequestAuthentication resources:

Traffic Wizards

Kiali also allows creation of Istio Gateway resources.

Istio ServiceEntry resources:

Istio Sidecar resources:

K8s Gateway resources:

K8s Reference Grants resources:

Other Kiali Wizards

Kiali also has Wizards available from the Namespaces page (Kiali >= 2.23) and many details pages, such as Service Detail to create routing rules. The Kiali Travel Tutorial goes into several of these wizards.

Namespaces Page Wizards

The Namespaces page (Kiali >= 2.23) has namespace-specific actions for creating traffic policies:

Service Wizards

The Service Detail page offers several wizards to create traffic control config:

Docs: Istio Infrastructure Status

Mon, 01 Jan 0001 00:00:00 +0000

A service mesh simplifies application services by deferring the non-business logic to the mesh. But for healthy applications the service mesh infrastructure must also be running normally. Kiali monitors the multiple components that make up the service mesh, letting you know if there is an underlying problem.

A component status will be one of: Not found, Not ready, Unreachable, Not healthy and Healthy. Not found means that Kiali is not able to find the deployment. Not ready means no pods are running. Unreachable means that Kiali hasn’t been successfully able to communicate with the component (Prometheus, Grafana and Jaeger). Not healthy means that the deployment doesn’t have the desired amount of replicas running. Otherwise, the component is Healthy and it won’t be shown in the list.

Regarding the severity of each component, there are only two options: core or add-on. The core components are those shown as errors (in red) whereas the add-ons are displayed as warnings (in orange).

By default, Kiali checks that the core components “istiod”, “ingress”, and “egress” are installed and running in the control plane namespace, and that the add-ons “prometheus”, “grafana” and “jaeger” are available.

Mesh page

Detailed information about the Istio infrastructure status is displayed on the mesh page. It shows an infrastructure topology view with core and add-on components, their health, and how they are connected to each other.

Similar to the traffic graph, the left side of the page shows the topology view, while the right side displays information about the selected node. If no node is selected, global infrastructure information is shown, including the status, version, and cluster of every component.

Connection issues between Kiali and any component are indicated with a red dotted line and a red health indicator in the target side panel.

The specific information shown in the target side panel depends on the type of node selected:

Kiali

When you click on the Kiali node, you can check information such as the version, health status, and configuration values.

Istio control plane

When you click on the Istio control plane, you can check information such as the Istio version, mTLS status, outbound policy, CPU and memory metrics, configuration table, and more.

Control plane Namespace

When you click on the control plane namespace box, you can check the numbers of Namespaces managed by Control Planes, this will show the data plane migration status between different versions of Istio installations.

Data plane

When you click on the cluster data plane, you can check the basic information of each namespace belonging to that data plane (Istio configuration, traffic inbound/outbound), similar to what you can see on the overview page.

Add-on components

When you click on the “prometheus”, “grafana” or “jaeger” node, , its health status, version, and configuration values are displayed:

Docs: Multi-cluster

Mon, 01 Jan 0001 00:00:00 +0000

A basic Istio mesh deployment has a single control plane with a single data plane, deployed on a single Kubernetes cluster. But Istio supports a variety of advanced deployment models. It allows a mesh to span multiple primary (control plane) and/or remote (data plane only) clusters, and can use a single or multi-network approach. The only strict rule is that within a mesh service names are unique. A non-basic mesh deployment generally involves multiple clusters. See installation instructions for more detail on installing advanced mesh deployments.

A single Kiali install can currently work with at most one mesh, one metric store and one trace store but it can be configured for “single cluster” or “multi-cluster”. All clusters must be part of the same mesh and report to the same metric and trace store, whether directly or via some sort of aggregator. See the multi-cluster configuration page for more information on requirements.

For multi-cluster configurations, Kiali provides a unified view and management of your mesh across clusters.

Graph View: Cluster and Namespace Boxing

Istio provides cluster names in the traffic telemetry for multi-cluster installations. The Kiali graph can use this information to better visualize clusters and namespaces. The Display menu offers two multi-cluster related options: Cluster Boxes and Namespace Boxes. When enabled, either separately or together, the graph will generate boxes to help identify the relevant nodes and edges, and to see traffic traveling between them.

Each box type supports selection and provides a side-panel summary of traffic. Below we see a Bookinfo traffic graph for when Bookinfo services are deployed across the East and West clusters. The West cluster box is selected. You see traffic for all services and workloads across both clusters. Single cluster configurations will show some traffic across clusters but not all.

Unified Multi-cluster configuration

Unlike single-cluster configurations, multi-cluster configurations show list/details pages across all clusters.

List Views: Aggregated mesh view

With a multi-cluster Kiali configuration, you can view all apps, workloads, services, and Istio config in your mesh from a single place.

Detail Views: Dig into details across clusters

The detail pages provide the same functionality across all clusters that you have access to for a single cluster, including viewing logs, metrics, traces, envoy details, and more.

Overview: Cross cluster namespace info

The overview page shows namespace information across all configured clusters.

Mesh view: Cluster, Istio and data plane boxing

The mesh graph displays infrastructure information for multiple clusters, Istio control planes, and data planes according to the Istio deployment (primary-remote or multi-primary).

Docs: Security

Mon, 01 Jan 0001 00:00:00 +0000

Kiali gives support to better understand how mTLS is used in Istio meshes. Find those helpers in the Mesh page, Traffic Graph, Overview Page, and specific validations.

Mesh indicator

At the right panel of the Istio Control Plane from the Mesh page, Kiali shows a lock when the mesh has enabled mTLS for the whole service mesh. It means that all the communications in the mesh uses mTLS.

Kiali shows a hollow lock when either the mesh is configured in PERMISSIVE mode or there is a misconfiguration in the mesh-wide mTLS configuration.

Namespace locks

The Namespaces page shows all the available namespaces with aggregated data. Besides the health and validations, Kiali also shows the namespace-wide mTLS status. Similar to the Mesh page, it shows a lock when strict mTLS is enabled or an open lock when permissive. A red open lock is shown when mTLS is disabled. When the namespace doesn’t include an mTLS policy and it is inherited from the mesh, a down arrow is shown and the inherited mTLS is described in the badge.

Graph

The mTLS method is used to establish communication between microservices. In the graph, Kiali has the option to show which edges are using mTLS and with what percentage during the selected period. When an edge shows a lock icon it means at least one request with mTLS enabled is present. In case there are both mTLS and non-mTLS requests, the side-panel will show the percentage of requests using mTLS.

Enable the option in the Display dropdown, select the security badge.

Validations

Kiali has different validations to help troubleshoot configurations related to mTLS such as DestinationRules and PeerAuthentications.

Docs: Topology

Mon, 01 Jan 0001 00:00:00 +0000

Kiali offers multiple ways for users to examine their mesh Topology. Each combines several information types to help users quickly evaluate the health of their service architecture.

Overview

Kiali’s default landing page is the mesh Overview. It presents a high-level view of the mesh clusters, control planes, data planes, and configuration, for the user. It combines service and application information, along with telemetry, validations and health, to provide a holistic summary of the system behavior. It highlights potential issues that the user may need to investigate.

The Namespaces page (Kiali >= 2.23) provides numerous filtering, sorting and presentation options for the available namespaces. From here users can perform namespace-level Actions, or quickly navigate to more detailed views.

Graph

The Kiali Graph offers a powerful visualization of your mesh traffic. The topology combines real-time request traffic with your Istio configuration information to present immediate insight into the behavior of service mesh, allowing you to quickly pinpoint issues. Multiple Graph Types allow you to visualize traffic as a high-level service topology, a low-level workload topology, or as an application-level topology.

Graph nodes are decorated with a variety of information, pointing out various route routing options like virtual services and service entries, as well as special configuration like fault-injection and circuit breakers. It can identify mTLS issues, latency issues, error traffic and more. The Graph is highly configurable, can show traffic animation, and has powerful Find and Hide abilities.

You can configure the graph to show the namespaces and data that are important to you, and display it in the way that best meets your needs.

Health

Colors in the graph represent the health of your service mesh. A node colored red or orange might need attention. The color of an edge between components represents the health of the requests between those components. The node shape indicates the type of component such as services, workloads, or apps.

The health of nodes and edges is refreshed automatically based on the user’s preference. The graph can also be paused to examine a particular state, or replayed to re-examine a particular time period.

Side-Panel

The collapsible side-panel summarizes the current graph selection, or the graph as a whole. A single-click will select the node, edge, or box of interest. The side panel provides:

Charts showing traffic and response times.
Health details.
Links to fully-detailed pages.
Response Code and Host breakdowns.
Traces involving the selected component.

Node Detail

A single-click selects a graph node. A double-click drills in to show the node’s Detail Graph. The node detail graph visualizes traffic from the point-of-view of that node, meaning it shows only the traffic reported by that node’s Istio proxy.

You can return back to the main graph, or double-click to change to a different node’s detail graph.

Traffic Animation

Kiali offers several display options for the graph, including traffic animation.

For HTTP traffic, circles represent successful requests while red diamonds represent errors. The more dense the circles and diamonds the higher the request rate. The faster the animation the faster the response times.

TCP traffic is represented by offset circles where the speed of the circles indicates the traffic speed.

Ranking

Nodes can be ranked in the graph based on pre-defined criteria such as ’number of inbound edges’. Combined with the find/hide feature, this allows you to quickly highlight or filter for important workloads, services, and applications.

Rankings are normalized to fit between 1..100 and nodes may tie with each other in rank. Ranking starts at 1 for the top ranked nodes so when ranking nodes based on ’number of inbound edges’, the node(s) with the most inbound edges would have rank 1. Node(s) with the second most inbound edges would have rank 2. Each selected criteria contributes equally to a node’s ranking. Although 100 rankings are possible, only the required number of rankings are assigned, starting at 1.

Graph Types

Kiali offers four different traffic-graph renderings:

The workload graph provides the a detailed view of communication between workloads.
The app graph aggregates the workloads with the same app labeling, which provides a more logical view.
The versioned app graph aggregates by app, but breaks out the different versions providing traffic breakdowns that are version-specific.
The service graph provides a high-level view, which aggregates all traffic for defined services.

Replay

Graph replay allows you to replay traffic from a selected past time-period. This gives you a chance to thoroughly examine a time period of interest, or share it with a co-worker. The graph is fully bookmarkable, including replay.

Operation Nodes

Istio v1.6 introduced Request Classification. This powerful feature allows users to classify requests into aggregates, called “Operations” by convention, to better understand how a service is being used. If configured in Istio the Kiali graph can show these as Operation nodes. The user needs only to enable the “Operation Nodes” display option. Operations can span services, for example, “VIP” may be configured for both CarRental and HotelRental services. To see total “VIP” traffic then display operation nodes without service nodes. To see “VIP” traffic specific to each service then also enable the “Service Nodes” display option.

When selected, an Operation node also provides a side-panel view. And when double-clicked a node detail graph is also provided.

Because operation nodes represent aggregate traffic they are not compatible with Service graphs, which themselves are already logical aggregates. For similar reasons response time information is not available on edges leading into or out of operation nodes. But by selecting the edge the response time information is available in the side panel (if configured).

Operation nodes are represented as pentagons in the Kiali graph:

Docs: Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Kiali offers a native integration with two different Distributed Tracing platforms, Jaeger and Grafana Tempo. As such, users can access trace visualizations. But more than that, Kiali incorporates tracing into several correlated views, making your investment in trace data even more valuable.

For a quick glimpse at Kiali tracing features, see below. For a detailed explanation of tracing in Kiali, see this 3-part Trace my mesh blog series. There is a detailed guide to configure access to the different distributed tracing platform, Jaeger and Grafana Tempo.

Workload detail

When investigating a workload, click the Traces tab to visualize your traces in a chart. When selecting a trace Kiali presents a tab for trace detail, and a tab for span details. Kiali always tries to surface problem areas, Kiali uses a heatmap approach to help the user identify problem traces or spans.

Heatmaps

A heatmap that you see in the Workload’s Tracing tab is a matrix that compares a specific trace’s request duration against duration metrics aggregated over time.

Each trace has a corresponding heatmap matrix. Each cell in the matrix corresponds to a specific metric aggregate; the value and color of a cell represents the difference between that metric and the duration of the matrix’s associated trace.

For example, the top-right cell of the heatmap above shows that the duration of the request represented by the trace (5.96ms) was 17.5ms faster than the 99th percentile of all inbound requests to the workload within the last 10 minutes.

The color of a cell will range between red and green; the more green a cell is, the faster the duration of the trace was compared to the aggregate metric data. A red cell indicates the associated trace was much slower compared to the aggregate metric data and so examining that trace could help detect a potential bottleneck or problem.

Metric Correlation

Kiali offers span overlays on Metric charts. The user can simply enable the spans option to generate the overlay. Clicking any span will navigate back to the Traces tab, focused on the trace of interest.

Graph Correlation

Kiali users often use the Graph Feature to visualize their mesh traffic. In the side panel, When selecting a graph node, the user will be presented with a Traces tab, which lists available traces for the time period. When selecting a trace the graph will display an overlay for the trace’s spans. And the side panel will display span details and offer links back to the trace detail views.

Logs Correlation

Kiali works to correlate the standard pillars of observability: traces, metrics and logs. Kiali can present a unified view of log and trace information, in this way letting users use logs to identify traces of interest. When enabling the spans option Kiali adds trace entries to the workload logs view. Below, in time-sorted order, the user is presented with a unified view of application logs (in white), Envoy proxy logs (in gold), and trace spaces (in blue). Clicking a span of interest brings you to the detail view for the trace of interest.

Docs: Validation

Mon, 01 Jan 0001 00:00:00 +0000

Disabling validations

In certain environments, particularly those with a high volume of configurations or limited resources, the Istio validation process can be time and resource intensive, potentially causing delays. To prioritize speed and resource efficiency in such scenarios, Kiali offers the option to disable these validations by configuring the validation_reconcile_interval setting to “0s” within the Kiali CR.

Below is an example of a Kiali CR with validations disabled:

spec:
  external_services:
    istio:
      validation_reconcile_interval: "0s"

The complete list of validations:

AuthorizationPolicy

KIA0101 - Namespace not found for this rule

AuthorizationPolicy enables access control on workloads. Each policy effects only to a group of request. For instance, all requests started from a workload on a list of namespaces. The present validation points out those rules referencing a namespace that don’t exist in the cluster.

Resolution

Either remove the namespace from the list, correct if there is any typo or create a new namespace.

Severity

Warning

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
         - non-existing # warning
         - unexisting # warning
   to:
   - operation:
       methods: ["GET"]
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0102 - Only HTTP methods and fully-qualified gRPC names are allowed

An AuthorizationPolicy has an Operation field where is defined the oprations allowed for a request. In the method field are listed all the allowed methods that request can have. This validation appears when a problem is found in there. The only methods accepted are: either HTTP valid methods or fully-qualified names of gRPC service in the form of “/package.service/method”

Resolution

Either change or remove the violating method. It has to be either a HTTP valid method or a fully-qualified names of a gRPC service in the form of “/package.service/method”

Severity

Warning

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       methods:
         - "GET"
         - "/package.service/method"
         - "WRONG" # Warning
         - "non-fully-qualified-grpc" # Warning
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0104 - This host has no matching entry in the service registry

AuthorizationPolicy enables access control on workloads. Each policy effects only to a group of request going to a specific destination. For instance, allow all the request going to details host.

The present validation points out those rules referencing a host that don’t exist in the authorization policy namespace. Kiali considers services, virtual services and service entries. Those hosts that refers to hosts outside of the object namespace will be presented with an unknown error.

Resolution

Either remove the host from the list, correct if there is any typo or deploy a new service, service entry or a virtual service pointing to that host.

Severity

Error

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       hosts:
         - wrong # Error
         - ratings
         - details.default
         - reviews.default.svc.cluster.local
         - productpage.outside # Unknown
         - google.com # Service Entry present. No error
         - google.org # Service Entry not present, wrong domain. Error.
       methods:
         - "GET"
         - "/package.service/method"
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0105 - This field requires mTLS to be enabled

AuthorizationPolicy has a Source field which specifies the source identities of a request. The Source field accepts principals and namespaces which require mTLS be enabled.

A validation Error message on a principals or namespaces fields means that mTLS is not enabled.

This validation appears only when autoMtls is disabled.

Resolution

Either remove this field or enable autoMtls.

Severity

Error

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: bookinfo
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       methods: ["GET"]
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0106 - Service Account not found for this principal

AuthorizationPolicy has a Source field which specifies the source identities of a request. The Source field allows principals to be specified - a list of peer identities derived from the peer certificate. A peer identity is in the format of <TRUST_DOMAIN>/ns/<NAMESPACE>/sa/<SERVICE_ACCOUNT>, for example, cluster.local/ns/default/sa/productpage.

A validation Error message on a principal value means that, while the specified Service Account may exist, it is not referenced by any Pod in the system.

Resolution

Correct the principal to refer to an existing Service Account, make sure that the value is in correct format without a typo, and make sure at least one Pod references the Service Account.

Severity

Error

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       hosts:
         - ratings
         - details.default
         - reviews.default.svc.cluster.local
       methods:
         - "GET"
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0107 - Service Account for this principal found on a remote cluster

A validation Warning message on a principal value means, that the specified Service Account was found in a cluster different from that of the AuthorizationPolicy.

Resolution

Kiali currently does not verify if the SPIRE is configured on the workload of the remote cluster.

Severity

Warning

Destination rules

KIA0201 - More than one DestinationRules for the same host subset combination

This validation warning could be a result of duplicate definition of the same subsets as well as from rules that apply to all subsets. Also, a combination of one Destination Rule (DR) applying to all subsets and another defining behavior for only some subsets triggers this validation warning.

Istio silently ignores the duplicate subsets and merge these destination rules without letting the user know. Only the first seen rule (by Istio) per subset is used and information from multiple definitions is not merged. While the routing might work correctly, this is most likely a configuration error. It may lead to a undesired behavior if one of the offending rules is removed or modified and that is probably not the intention of the deployer of this service. Also, if the two offending destination rules have different policies for traffic routing the wrong one might be used.

Resolution

Either merge the settings to a single DR or split the subsets in such a way that they do not interleave. This ensures that the routing behavior stays consistent.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews-dr1
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews-dr2
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1

KIA0202 - This host has no matching entry in the service registry (service, workload or service entries)

Istio applies traffic rules for services after the routing has happened. These can include different settings such as connection pooling, circuit breakers, load balancing, and detection. Istio can define the same rules for all services under a host or different rules for different versions of the service. The host must have a service that is defined in the platform’s service registry or as a ServiceEntry. Short names are extended to include ‘.namespace.cluster’ using the namespace of the destination rule, not the service itself. FQDN is evaluated as is. It is recommended to use the FQDN to prevent any confusion.

If the host is not found, Istio ignores the defined rules.

Resolution

Correct the host to point to a correct service, in this namespace or with FQDN to other namespaces, or deploy the missing service to the mesh.

Severity

Error

There is an exception to the severity level: It will be shown as a Warning when OutboundTrafficPolicy Mode for MeshConfig is set to ALLOW_ANY.

Example

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: notpresent
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3

KIA0203 - This subset’s labels are not found in any matching host

Istio applies traffic rules for services after the routing has happened. These can include different settings such as connection pooling, circuit breakers, load balancing, and detection. Istio can define the same rules for all services under a host or different rules for different versions of the service. The host must a service that is defined in the platform’s service registry or as a ServiceEntry. Short names are extended to include ‘.namespace.cluster’ using the namespace of the destination rule, not the service itself. FQDN is evaluated as is. It is recommended to use the FQDN to prevent any confusion.

Subsets can override the global settings defined in the DR for a host.

If the host is not found, Istio ignores the defined rules.

If the not found subset is not referenced in any Virtual Service, the severity of this error is changed to Info.

Resolution

Correct the host to point to a correct service, in this namespace or with FQDN to other namespaces, or deploy the missing service to the mesh. If the hostname is equal to the one used otherwise in the DR, consider removing the duplicate host resolution.

Also, verify that the labels are correctly matching a workload with the intended service.

Severity

Error

Example

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v10
  - name: v2
    labels:
      notfoundlabel: v2
  - name: v3
    labels:
      version: v3

KIA0204 - mTLS settings of a non-local Destination Rule are overridden

Istio allows you to define DestinationRule at three different levels: mesh, namespace and service level. A mesh may have multiple DRs. In case of having two DestinationRules on the first one is at a lower level than the second one, the first one overrides the TLS values of the second one.

Resolution

This validation aims to warn Kiali users that they may be disabling/enabling mTLS from the higher DestinationRule. Merging the TLS settings to one of the DestinationRules is the only way to fix this validation. However, this is a valid scenario so it might be impossible to remove this warning.

Severity

Warning

Example

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3

KIA0205 - PeerAuthentication enabling mTLS at mesh level is missing

Istio has the ability to define mTLS communications at mesh level. In order to do that, Istio needs one DestinationRule and one PeerAuthentication. The DestinationRule configures all the clients of the mesh to use mTLS protocol on their connections. The PeerAuthentication defines what authentication methods that can be accepted on the workload of the whole mesh. If the PeerAuthentication is not found or doesn’t exist and the mesh-wide DestinationRule is on ISTIO_MUTUAL mode, all the communication returns 500 errors.

Resolution

Add a PeerAuthentication within the istio-system namespace without specifying targets but setting peers mtls mode to STRICT or PERMISSIVE. The PeerAuthentication should be like this.

Severity

Error

Example

# AutoMtls disabled, no PeerAuthentication at mesh-level defined
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

KIA0206 - PeerAuthentication enabling namespace-wide mTLS is missing

Istio has the ability to define mTLS communications at namespace level. In order to do that, Istio needs both a DestinationRule and a PeerAuthentication targeting all the clients/workloads of the specific namespace. The PeerAuthentication allows mTLS authentication method for all the workloads within a namespace. The DestinationRule defines all the clients within the namespace to start communications in mTLS mode. If the PeerAuthentication is not found and the DestinationRule is on STRICT mode in that namespace but there is the DestinationRule enabling mTLS, all the communications within that namespace returns 500 errors.

Resolution

A PeerAuthentication enabling mTLS method is needed for the workloads in the namespace. Otherwise all the clients start mTLS connections that those workloads won’t be ready to manage. Add a PeerAuthentication without specifying targets but setting mTLS mode to STRICT or PERMISSIVE in the same namespace as the DestinationRule.

Severity

Error

Example

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "enable-mtls"
  namespace: "bookinfo"
spec:
  host: "*.bookinfo.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

KIA0207 - PeerAuthentication with TLS strict mode found, it should be permissive

Istio needs both a DestinationRule and PeerAuthentication to enable mTLS communications. The PeerAuthentication configures the authentication method accepted for all the targeted workloads. The DestinationRule defines which is the authentication method that the clients of specific workloads has to start communications with.

Resolution

Kiali has found that there is a DestinationRule sending traffic without mTLS authentication method. There are two different ways to fix this situation. You can either change the PeerAuthentication applying to PERMISSIVE mode or change the DestinationRule to start communications using mTLS.

Severity

Error

Example

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "disable-mtls"
  namespace: "bookinfo"
spec:
  host: "*.bookinfo.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: DISABLE
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "enable-mtls"
  namespace: "bookinfo"
spec:
  host: "*.bookinfo.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "bookinfo"
spec:
  mtls:
    mode: STRICT

KIA0208 - PeerAuthentication enabling mTLS found, permissive mode needed

Kiali found a DestinationRule starting communications without TLS but there was a PeerAuthentication allowing all services in the mesh to accept only requests in mTLS.

Resolution

There are two ways to fix this situation. You can either change the PeerAuthentication to enable PERMISSIVE mode to all the workloads in the mesh or change the DestinatonRule to enable mTLS instead of disabling it (change the mode to ISTIO_MUTUAL).

Severity

Error

Example

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "bookinfo"
spec:
  host: "*.bookinfo.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: DISABLE
---
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT

KIA0209 - DestinationRule Subset has not labels

A DestinationRule subset without labels may miss the destination endpoint linked with a specific workload.

If there is any other subset with valid labels, the severity of this warning is changed to Info.

Resolution

Validate that a subset is properly configured.

Severity

Warning

K8s Gateway API

Note that with the support of K8s Gateway API, a new mechanism of subsetting is introduced. Which means, that each version of a service should have a separate Service pointing to that particular version. And instead of the usage of DestinationRules, there should be a K8s HTTPRoute object created, referencing to Services per version in it’s rules.

Gateways

KIA0301 - More than one Gateway for the same host port combination

Gateway creates a proxy that forwards the inbound traffic for the exposed ports. If two different gateways expose the same ports for the same host, this creates ambiguity inside Istio as either of these gateways could handle the traffic. This is most likely a configuration error. This check is done across all namespaces the user has access to.

There is one exception: when both gateways points to a different ingress. Then the ambiguity doesn’t exist and, in consequence, no validation is shown. Kiali considers that two gateways points to the same ingress if they share the exact same selector.

Resolution

Remove the duplicate gateway entries or merge the two gateway definitions into a single one.

When one of the duplicate Gateways has a wildcard in hosts, there is an option ‘skip_wildcard_gateway_hosts’ in Kiali CR, by setting it to ’true’, it will ignore Gateways with wildcards in hosts during validation. As Istio considers such a Gateway with a wildcard in hosts as the last in order, after the Gateways with FQDN in hosts.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway # Validation shown
  namespace: bookinfo
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway-copy # Validation shown
  namespace: bookinfo2
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway-diff-ingress # No validations shown
  namespace: bookinfo
spec:
  selector:
    istio: ingressgateway-pub # Using different ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---

KIA0302 - No matching workload found for gateway selector in this namespace

This validation checks the current namespace for matching workloads as this is recommended, and potentially in the future required, by the Istio. Excluded from this check are the default “istio-ingressgateway” and “istio-egressgateway” workloads which are included in Istio by default.

Although your traffic might be correctly routed to a workload in other namespace, this is not a guaranteed behavior and thus a warning is flagged in such cases also.

Resolution

Deploy the missing workload or fix the selector to target a correct location.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
spec:
  selector:
    app: nonexisting # workload doesn't exist in the namespace
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

Mesh Policies

KIA0401 - Mesh-wide Destination Rule enabling mTLS is missing

Istio has the ability to define mTLS communications at mesh level. In order to do that, Istio needs one DestinationRule and one PeerAuthentication. The DestinationRule configures all the clients of the mesh to use mTLS protocol on their connections. The PeerAuthentication defines what authentication methods can be accepted on the workload of the whole mesh. If the DestinationRule is not found or doesn’t exist and the PeerAuthentication is on STRICT mode, all the communication returns 500 errors.

Resolution

Add a DestinationRule with “*.cluster” host and ISTIO_MUTUAL as tls trafficPolicy mode. The DestinationRule should be like this.

Severity

Error

Example

# Make sure there isn't any DestinationRule enabling meshwide mTLS
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT

PeerAuthentication

KIA0501 - Destination Rule enabling namespace-wide mTLS is missing

Istio has the ability to define mTLS communications at namespace level. In order to do that, Istio needs one DestinationRule and one PeerAuthentication. The DestinationRule configures all the clients of the namespace to use mTLS protocol on their connections. The PeerAuthentication defines what authentication methods can be accepted on a specific group of workloads. PeerAuthentications without target field specified will target all the workloads within its namespace. If the DestinationRule is not found or doesn’t exist in the namespace and the namespace-wide PeerAuthentication is on STRICT mode, all the communication will return 500 errors.

Resolution

Add a DestinationRule with “*.namespace.svc.cluster.local” host and ISTIO_MUTUAL as tls trafficPolicy mode. The DestinationRule should be like this.

Severity

Error

Example

# Make sure there isn't any DestinationRule enabling meshwide mTLS
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "bookinfo"
spec:
  mtls:
    mode: STRICT

KIA0505 - Destination Rule disabling namespace-wide mTLS is missing

PeerAuthentication objects are used to define the authentication methods that a set of workloads can accept: Mutual, Istio Mutual, Simple or Disabled.

This validation warns the scenario where there is one PeerAuthentication at namespace level with DISABLE mode but there is DestinationRule at namespace or mesh level enabling mTLS. With this scenario, all the traffic flowing between the services in that namespace will fail.

Resolution

You can either change the namespace/mesh-wide Destination Rule to DISABLE mode or change the current PeerAuthentication to allow mTLS (mode STRICT or PERMISSIVE).

Severity

Error

Example

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "bookinfo"
spec:
  mtls:
    mode: DISABLE
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "enable-mtls"
  namespace: bookinfo
spec:
  host: "*.bookinfo.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

KIA0506 - Destination Rule disabling mesh-wide mTLS is missing

PeerAuthentication objects are used to define the authentication methods that a set of workloads can accept: Mutual, Istio Mutual, Simple or Disabled.

This validation warns the scenario where there is one PeerAuthentication at mesh level with DISABLE mode but there is DestinationRule at mesh level enabling mTLS. With this scenario, all the traffic flowing between the services in that namespace will fail.

Resolution

You can either change the mesh-wide Destination Rule to DISABLE mode or change the current PeerAuthentication to allow mTLS (mode STRICT or PERMISSIVE).

Severity

Error

Example

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: DISABLE
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "enable-mtls"
  namespace: bookinfo
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Ports

KIA0601 - Port name must follow [-suffix] form

Istio requires the service ports to follow the naming form of ‘protocol-suffix’ where the ‘-suffix’ part is optional. If the naming does not match this form (or is undefined), Istio treats all the traffic TCP instead of the defined protocol in the definition. Dash is a required character between protocol and suffix. For example, ‘http2foo’ is not valid, while ‘http2-foo’ is (for http2 protocol).

Resolution

Rename the service port name field to follow the form and the traffic flows correctly.

Severity

Error

Example

apiVersion: v1
kind: Service
metadata:
  name: ratings-java-svc
  namespace: bookinfo
  labels:
    app: ratings
    service: ratings-svc
spec:
  ports:
  - port: 9080
    name: wrong-http
  selector:
    app: ratings-java
    version: v1

KIA0602 - Port appProtocol must follow form

Istio also optionally supports the appProtocol in service ports, following the form of ‘protocol’. When port name field does not contain the protocol the appProtocol field is considered as a protocol. If the naming does not match this form, Istio treats all the traffic TCP instead of the defined protocol in the definition.

Resolution

Rename the service port appProtocol field to follow the form and the traffic flows correctly.

Severity

Error

Example

apiVersion: v1
kind: Service
metadata:
  name: ratings-java-svc
  namespace: bookinfo
  labels:
    app: ratings
    service: ratings-svc
spec:
  ports:
  - port: 3306
    name: database
    appProtocol: wrong-mysql
  selector:
    app: ratings-java
    version: v1

Services

KIA0701 - Deployment exposing same port as Service not found

Service definition has a combination of labels and port definitions that are not matching to any workloads. This means the deployment will be unsuccessful and no traffic can flow between these two resources. The port is read from the Service ‘TargetPort’ definition first and if undefined, the ‘Port’ field is used as Kubernetes defaults the ‘TargetPort’ to ‘Port’. If the ‘TargetPort’ is using a integer, the port numbers are compared and if the ‘TargetPort’ is a string, the deployment’s portName is used for comparison.

Resolution

Fix the port definitions in the workload or in the service definition to ensure they match.

Severity

Warning

Example

Invalid example with port definitions unmatched:

apiVersion: v1
kind: Service
metadata:
  name: ratings-java-svc
  namespace: ratings-java
  labels:
    app: ratings
    service: ratings-svc
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings-java
    version: v1
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ratings-java
  namespace: ratings-java
  labels:
    app: ratings-java
    version: v1
spec:
  replicas: 1
  template:
    metadata:
      annotations:
         sidecar.istio.io/inject: "true"
      labels:
        app: ratings-java
        version: v1
    spec:
      containers:
      - name: ratings-java
        image: pilhuhn/ratings-java:f
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

Valid example using targetPort definition matching:

apiVersion: v1
kind: Service
metadata:
  name: ratings-java-svc
  namespace: ratings-java
  labels:
    app: ratings
    service: ratings-svc
spec:
  ports:
  - port: 9080
    targetPort: 8080
    name: http
  selector:
    app: ratings-java
    version: v1
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ratings-java
  namespace: ratings-java
  labels:
    app: ratings-java
    version: v1
spec:
  replicas: 1
  template:
    metadata:
      annotations:
         sidecar.istio.io/inject: "true"
      labels:
        app: ratings-java
        version: v1
    spec:
      containers:
      - name: ratings-java
        image: pilhuhn/ratings-java:f
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

Sidecars

KIA1004 - This host has no matching entry in the service registry

The Sidecar resources are used for configuring the sidecar proxies in the service mesh. IstioEgressListener specifies the properties of an outbound traffic listener on the sidecar proxy attached to a workload instance.

In the hosts field, there is the list of hosts exposed to the workload. Each host in the list have the namespace/dnsName format where both namespace and dnsName may have non-obvious values. namespace may be either ., ~, * or an actual namespace name. dnsName has to be a FQDN representing a service, virtual service or a service entry. This FQDN may use the wildcard character.

See more information about the syntax of both namespace and dnsName into istio documentation.

Resolution

Make sure there is a service, virtual service or service entry matching with the host.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: servicenotfound
  namespace: bookinfo
spec:
  workloadSelector:
    labels:
      app: reviews
  egress:
  - port:
      number: 3306
      protocol: MYSQL
      name: egressmysql
    captureMode: NONE
    bind: 127.0.0.1
    hosts:
    - "bookinfo/*.bookinfo.svc.cluster.local" # Bookinfo running into bookinfo ns
    - "default/kiali.io" # Service entry present in the namespace
    - "bookinfo/bogus.bookinfo.svc.cluster.local" # Bogus service into bookinfo doesn't exist
    - "bogus-ns/reviews.bookinfo.svc.cluster.local" # Cross-namespace validation: unable to verify validity

KIA1006 - Global default sidecar should not have workloadSelector

The Sidecar resources are used for configuring the sidecar proxies in the service mesh. By default, all the sidecars are configured with the default sidecar instance specified in the control plane namespace (usually istio-system). In case there are sidecar resources in the namespaces where your applications are, this default sidecar resource won’t be considered. The sidecar in your namespace will be applied.

Having workloadSelector in your global default sidecar won’t make any effect in the other sidecars living outside of the control plane namespace.

Resolution

Make sure you don’t have the workloadSelector in this global sidecar resource. In case you need specific settings for specific workloads, move those settings to the sidecar resources in your application namespaces.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: default
  namespace: istio-system
spec:
  workloadSelector: # Default sidecar can't have labels
    labels:
      version: v1
  egress:
  - port:
      number: 3306
      protocol: MYSQL
      name: egressmysql
    captureMode: NONE
    bind: 127.0.0.1
    hosts:
    - "bookinfo/reviews.bookinfo.svc.cluster.local"
    - "bookinfo/details.bookinfo.svc.cluster.local"

KIA1007 - OutboundTrafficPolicy shown with empty mode value is ambiguous

Due to issues with the Istio client and the protobuf library it uses, the way some defaults are handled becomes ambiguous. When a Sidecar resource spec.outboundTrafficPolicy.mode is left unset or is explicitly set to REGISTRY_ONLY, the Kiali UI will show the value as unset (e.g. if nothing is set inside outboundTrafficPolicy, its value will be shown as {}). In this case, you are not guaranteed to know what the value of mode truly is. So in the case where Kiali UI shows mode as empty, you cannot know if Istio will be using a value of REGISTRY_ONLY or ALLOW_ANY.

Resolution

You cannot determine the value of mode using the Kiali UI when this condition arises. You must inspect the Sidecar object using other means to determine the value (e.g. use kubectl get sidecar your-side-car-name -o jsonpath='{.spec.outboundTrafficPolicy.mode}').

Severity

Informational

Example

Both of these Sidecar resources will show mode as unset/empty in the Kiali UI YAML editor:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
...
spec:
  outboundTrafficPolicy:
    mode: REGISTRY_ONLY

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
...
spec:
  # according to Istio documentation, the default will be ALLOW_ANY
  outboundTrafficPolicy: {}

VirtualServices

KIA1101 - DestinationWeight on route doesn’t have a valid service (host not found)

VirtualService routes matching requests to a service inside your mesh. Routing can also match a subset of traffic to a certain version of it for example. Any service inside the mesh must be targeted by its name, the IP address are only allowed for hosts defined through a Gateway. Host must be in a short name or FQDN format. Short name will evaluate to VS’ namespace, regardless of where the actual service might be placed.

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews-cp
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

KIA1107 - Subset not found

VirtualService routes matching requests to a service inside your mesh. Routing can also match a subset of traffic to a certain version of it for example. The subsets referred in a VirtualService have to be defined in one DestinationRule.

If one route in the VirtualService points to a subset that doesn’t exist Istio won’t be able to send traffic to a service.

Resolution

Fix the routes that points to a non existing subsets. It might be fixing a typo in the subset’s name or defining the missing subset in a DestinationRule.

Severity

Error

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: nosubset
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

KIA1108 - Preferred nomenclature: /

A virtual service may include a list of gateways which the defined routes should be applied to. Gateways in other namespaces may be referred to by /; specifying a gateway with no namespace qualifier is the same as specifying the VirtualService’s namespace.

Resolution

Move the nomenclature of the gateways into the supported Istio form: /

Example

kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: bookinfo
  namespace: bookinfo
spec:
  hosts:
    - '*'
  gateways:
    - bookinfo-gateway.bookinfo.svc.cluster.local # unsupported format
    - bookinfo/bookinfo-gateway # works
  http:
    - match:
        - uri:
            exact: /productpage
        - uri:
            prefix: /static
        - uri:
            exact: /login
        - uri:
            exact: /logout
        - uri:
            prefix: /api/v1/products
      route:
        - destination:
            host: productpage
            port:
              number: 9080
---
kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
spec:
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 80
        protocol: HTTP
  selector:
    istio: ingressgateway

WorkloadEntries

KIA1201 - Missing one or more addresses from matching WorkloadEntries

This validation shows, that the address field’s value of Workload Entry is not matching to any address of Service Entry.

Resolution

Add missing Service Entry which address will match the Workload Entry’s address.

Severity

Warning

Example

apiVersion: networking.istio.io/v1beta1
kind: WorkloadEntry
metadata:
  name: ratings-v1
  namespace: bookinfo
spec:
  serviceAccount: ratings-vm
  address: 3.3.3.3
  labels:
    app: ratings
    version: v1
  ports:
    http: 9080
---
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: ratings-unmatching-address
  namespace: bookinfo
spec:
  addresses:
    - 4.4.4.4 # This IP is not in any WorkloadEntry. It needs 3.3.3.3 to work.
  hosts:
    - ratings
  location: MESH_INTERNAL
  resolution: STATIC
  ports:
    - number: 9080
      name: http
      protocol: HTTP
      targetPort: 9080
  workloadSelector:
    labels:
      app: ratings-unmatching

K8s Routes

KIA1401 - Route is pointing to a non-existent or inaccessible K8s gateway

Gateway API Protocol Route could be pointing to a [k8s] Gateway that the Route wants to be attached to. When the namespace field is not specified it takes Gateways from the current Route’s namespace. Here the error indicates that the referenced Gateway is not found in the provided namespace. If the [k8s] Gateway is in another namespace, then that gateway should be shared to the Route’s namespace. The Gateway API supports cross-namespace routing, allowing Gateways and Routes to be deployed into different namespaces with routes attaching to Gateways across namespace boundaries.

Resolution

Fix the parentRefs field to target to an existing gateway. Or share the Gateway to the namespace where the Route is located.

Severity

Error

Example

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1alpha2
metadata:
  name: httproute
  namespace: bookinfo
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      namespace: istio-system
      name: gatewayapi
  hostnames:
    - details

KIA1402 - Reference doesn’t have a valid service (Service name not found)

Gateway API Route could be pointing to a Service inside your mesh the Route sends the traffic to. A Service name should be specified, not a hostname. This Service can be a certain version of a parent Service, but in that case a separate Service is required to be created. When the namespace field is not specified it takes Service from the current Route’s namespace. In a case of referencing to a Service from remote namespace, a ReferenceGrant object needs to be created to enable cross namespace references. Here the error indicates that the referenced Service is not found in the provided namespace or the ReferenceGrant is missing (in a case of remote namespace).

Resolution

Correct the backendRefs name to point to a correct Service (in this namespace or to other namespaces):

Deploy the missing Service to the mesh, create a ReferenceGrant object in a case of remote namespace.
Or remove the configuration linking to that non-existing Service.

Severity

Error

Example

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1alpha2
metadata:
  name: httproute
  namespace: bookinfo
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      namespace: istio-system
      name: gatewayapi
  hostnames:
    - reviews
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /get
      backendRefs:
        - group: ''
          kind: Service
          name: reviews-v1
          namespace: default
          port: 9080
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: refgrant
  namespace: default
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: bookinfo
  to:
  - group: ""
    kind: Service

Workloads

KIA1301 - This workload is not covered by any authorization policy

Istio Authorization Policy enables access control on workloads in the mesh. Auth Policy selector will match with workloads in the same namespace as the authorization policy. If the authorization policy is in the root namespace, the selector will additionally match with workloads in all namespaces. This validation shows, that the selector match did not happen.

Resolution

Add Autorization Policy which selector matches with Workload’s label selector.

Severity

Warning

Warning

Generic

KIA0002 - More than one selector-less object in the same namespace

This validation refers to the usage of the selector. Selector-less Istio objects are those objects that don’t have the selector field specified. Therefore, objects that apply to all the workloads of a namespace (or whole mesh if the namespace is the same as the control plane namespace).

This validation warns you that you have two different objects living in the same namespace. This may leave an non-deterministic or unexpected behavior on the workloads of the namespace.

Resolution

The natural solution is to merge both objects. In case there are different behaviors you want to apply, consider to define the selector field targeting a specific set of workloads.

Severity

Error

Example

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "bookinfo"
spec:
  mtls:
    mode: STRICT
---
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "duplicate"
  namespace: "bookinfo"
spec:
  mtls:
    mode: STRICT

KIA0003 - More than one object applied to the same workload

This validation refers to the usage of the selector. In this field are defined the labels of the workloads that this object will be applied to. It might be one or more workloads in the same namespace.

Choose existing and accessible namespace to export to.

Severity

Error

K8s Gateway

KIA1501 - More than one K8s Gateway for the same host and port combination

A k8s Gateway defines a point where traffic can be translated to Services, within the cluster. This is defined through listeners or addresses. This validation warns when finding multiple Listener definitions for the same port and host combination, in different k8s Gateways. In this case the traffic handling can be in conflict.

Warning

Workload Groups

KIA1701 - Service Account not found in this namespace

WorkloadGroup describes a collection of workload instances. It enables specifying the properties of a single workload for bootstrap and provides a template for WorkloadEntry, using the specified service account from the same namespace.

A validation Warning message on a template means that, while the specified serviceAccount may exist, it is not referenced by any Pod in the same namespace.

Resolution

Correct the template to refer to an existing Service Account from the same namespace, make sure that the value is in correct format without a typo, and make sure at least one Pod references the Service Account.

Severity

Warning

Example

apiVersion: networking.istio.io/v1
kind: WorkloadGroup
metadata:
  name: ratings-vm
  namespace: bookinfo
  labels:
    app: ratings-vm
spec:
  template:
    serviceAccount: default

KIA1702 - More than one Workload Group with duplicate labels found in the same namespace

The set of labels from Workload Group spec metadata will be associated with each workload instance during the bootstrap process.

A validation Warning message means that the labels set in this Workload Group spec metadata are also used in other Workload Group within this namespace.

Resolution

Set a unique set of labels.

Severity

Warning