Install Kuadrant on an OpenShift cluster¶
Note
You must perform these steps on each OpenShift cluster that you want to use Kuadrant on.
In this document we use AWS route 53 as the example setup.
Warning
Kuadrant uses a number of labels to search and filter resources on the cluster.
All required labels are formatted as kuadrant.io/*.
Removal of any labels with the prefix may cause unexpected behaviour and degradation of the product.
Prerequisites¶
- OpenShift Container Platform 4.16.x or later with community Operator catalog available.
- AWS/Azure or GCP with DNS capabilities.
- Accessible Redis instance.
Procedure¶
Step 1 - Set up your environment¶
We use env vars for convenience only here. If you know these values you can setup the required yaml files in anyway that suites your needs.
export AWS_ACCESS_KEY_ID=xxxxxxx # Key ID from AWS with Route 53 access
export AWS_SECRET_ACCESS_KEY=xxxxxxx # Access key from AWS with Route 53 access
export REDIS_URL=redis://user:xxxxxx@some-redis.com:10340 # A Redis cluster URL
Set the version of Kuadrant to the latest released version: https://github.com/Kuadrant/kuadrant-operator/releases/
Step 2 - Install Gateway API v1¶
Before you can use Kuadrant, you must install Gateway API v1 as follows:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml
Step 3 - Install cert-manager¶
Before you can use Kuadrant, you must install cert-manager. Cert-Manager is used by kuadrant to manage TLS certificates for your gateways.
The minimum supported version of cert-manager is v1.14.0.
Install one of the different flavours of the Cert-Manager.
Install community version of the cert-manager¶
Consider installing cert-manager via OperatorHub, which you can do from the OpenShift web console.
More installation options at cert-manager.io
Install cert-manager Operator for Red Hat OpenShift¶
You can install the cert-manager Operator for Red Hat OpenShift by using the web console.
Note: Before using Kuadrant's
TLSPolicyyou will need to setup a certificate issuer refer to the cert-manager docs for more details
Step 4 - (Optional) Install and configure Istio with the Sail Operator¶
Note
Skip this step if planing to use Envoy Gateway as Gateway API provider
Kuadrant integrates with Istio as a Gateway API provider. You can set up an Istio-based Gateway API provider by using the Sail Operator.
Install Istio¶
To install the Istio Gateway provider, run the following commands:
kubectl apply -f - <<EOF
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
name: sail
namespace: gateway-system
spec:
upgradeStrategy: Default
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: sailoperator
namespace: gateway-system
spec:
channel: candidates
installPlanApproval: Automatic
name: sailoperator
source: community-operators
sourceNamespace: openshift-marketplace
EOF
Check the status of the installation as follows:
When ready, the status will change from installing to complete.
Configure Istio¶
To configure the Istio Gateway API provider, run the following command:
kubectl apply -f - <<EOF
apiVersion: sailoperator.io/v1alpha1
kind: Istio
metadata:
name: default
spec:
namespace: gateway-system
updateStrategy:
type: InPlace
inactiveRevisionDeletionGracePeriodSeconds: 30
version: v1.23.0
values:
pilot:
autoscaleEnabled: false
EOF
More details on what can be configured can be found by executing
And in documention : Istio Config
Wait for Istio to be ready as follows:
Step 5 - (Optional) Install Envoy Gateway as a Gateway API provider¶
Note
Skip this step if planing to use Istio as Gateway API provider
Note
There are several ways to install Envoy Gateway (via egctl, Helm chart or Kubernetes yaml) - this is just an example for starting from a bare Kubernetes cluster.
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.1.0 -n envoy-gateway-system --create-namespace
Enable EnvoyPatchPolicy feature:
TMP=$(mktemp -d)
kubectl get configmap -n envoy-gateway-system envoy-gateway-config -o jsonpath='{.data.envoy-gateway\.yaml}' > ${TMP}/envoy-gateway.yaml
yq e '.extensionApis.enableEnvoyPatchPolicy = true' -i ${TMP}/envoy-gateway.yaml
kubectl create configmap -n envoy-gateway-system envoy-gateway-config --from-file=envoy-gateway.yaml=${TMP}/envoy-gateway.yaml -o yaml --dry-run=client | kubectl replace -f -
kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system
Wait for Envoy Gateway to become available::
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
Step 6 - Optional: Configure observability and metrics (Istio only)¶
Kuadrant provides a set of example dashboards that use known metrics exported by Kuadrant and Gateway components to provide insight into different components of your APIs and Gateways. While not essential, it is recommended to set these up. First, enable monitoring for user-defined projects. This will allow the scraping of metrics from the gateway and Kuadrant components. The example dashboards and alerts for observing Kuadrant functionality use low-level CPU metrics and network metrics available from the user monitoring stack in OpenShift. They also use resource state metrics from Gateway API and Kuadrant resources.
To scrape these additional metrics, you can install a kube-state-metrics instance, with a custom resource configuration as follows:
kubectl apply -f https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/main/config/observability/openshift/kube-state-metrics.yaml
kubectl apply -k https://github.com/Kuadrant/gateway-api-state-metrics/config/kuadrant?ref=0.6.0
To enable request metrics in Istio and scrape them, create the following resource:
kubectl apply -f https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/refs/heads/main/config/observability/prometheus/monitors/istio/service-monitor-istiod.yaml
Some example dashboards show aggregations based on the path of requests. By default, Istio metrics don't include labels for request paths. However, you can enable them with the below Telemetry resource. Note that this may lead to a high cardinality label where multiple time-series are generated, which can have an impact on performance and resource usage.
kubectl apply -f https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/main/config/observability/openshift/telemetry.yaml
You can configure scraping of metrics from the various Kuadrant operators with the below resources.
kubectl create ns kuadrant-system
kubectl apply -f https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/refs/heads/main/config/observability/prometheus/monitors/operators.yaml
Note
There is 1 more metrics configuration that needs to be applied so that all relevant metrics are being scraped. That configuration depends on where you deploy your Gateway later. The steps to configure that are detailed in the follow on Secure, protect, and connect guide.
The example Grafana dashboards and alerts for observing Kuadrant functionality use low-level CPU metrics and network metrics available from the user monitoring stack in OpenShift. They also use resource state metrics from Gateway API and Kuadrant resources.
For Grafana installation details, see installing Grafana on OpenShift. That guide also explains how to set up a data source for the Thanos Query instance in OpenShift. For more detailed information about accessing the Thanos Query endpoint, see the OpenShift documentation.
Note
For some dashboard panels to work correctly, HTTPRoutes must include a "service" and "deployment" label with a value that matches the name of the service & deployment being routed to. eg. "service=myapp, deployment=myapp". This allows low level Istio & envoy metrics to be joined with Gateway API state metrics.
Step 7 - Setup the catalogsource¶
Before installing the Kuadrant Operator, you must enter the following commands to set up secrets that you will use later.
If you haven't aleady created the kuadrant-system namespace during the optional observability setup, do that first:
Set up a CatalogSource as follows:
kubectl apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: kuadrant-operator-catalog
namespace: kuadrant-system
spec:
sourceType: grpc
image: quay.io/kuadrant/kuadrant-operator-catalog:${KUADRANT_VERSION}
displayName: Kuadrant Operators
publisher: grpc
updateStrategy:
registryPoll:
interval: 45m
EOF
Step 8 - Install the Kuadrant Operator¶
To install the Kuadrant Operator, enter the following command:
kubectl apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: kuadrant-operator
namespace: kuadrant-system
spec:
channel: stable
installPlanApproval: Automatic
name: kuadrant-operator
source: kuadrant-operator-catalog
sourceNamespace: kuadrant-system
---
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
name: kuadrant
namespace: kuadrant-system
spec:
upgradeStrategy: Default
EOF
Wait for the Kuadrant Operators to be installed as follows:
After some time, this command should return complete.
Set up a DNSProvider¶
The example here is for AWS Route 53. It is important the secret for the DNSProvider is setup in the same namespace as the gateway.
kubectl -n ingress-gateway create secret generic aws-credentials \
--type=kuadrant.io/aws \
--from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
--from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
For more details on other providers take a look at DNS Providers
Setup TLS DNS verification credential¶
When setting up TLS certs via provider like lets-encrypt, cert-manager will do a DNS based verification. To allow it do this, it will need a similar credential to that one set up for the DNSProvider, but it needs to be created in the cert-manager namespace where the operator is installed.
kubectl -n cert-manager create secret generic aws-credentials \
--type=kuadrant.io/aws \
--from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
--from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
Step 9 - Install Kuadrant Components¶
To trigger your Kuadrant deployment, enter the following command:
kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
name: kuadrant
namespace: kuadrant-system
EOF
Wait for Kuadrant to be ready as follows:
This will setup and configure a number of Kuadrant subcomponents. Some of these can also take additional configuration:
- Authorino (Enforcement Component for AuthPolicy)
- Learn More: (Authorino CRD)[https://docs.kuadrant.io/latest/authorino-operator/#the-authorino-custom-resource-definition-crd]
- Limitador (Enforcement Component for RateLimitPolicy)
- Learn More:(Limitador CRD)[https://docs.kuadrant.io/latest/limitador-operator/#features]
- DNS Operator (Enforcement Component for DNSPOlicy)
Configuring Redis Storage for Limitador¶
Redis credentials for storage of rate limiting counters¶
In this installation we will show how to configure ratelimiting counters to be stored in redis. Before we go further we need to setup a redis secret to use later:
Update limitador config¶
To configure redis storage for Limatador, we must update the Limitador custom resource to use the secret we created:
You can run a command like the one below to add this configuration:
kubectl patch limitador limitador --type=merge -n kuadrant-system -p '
spec:
storage:
redis:
configSecretRef:
name: redis-credential
'
Check that limitador is back to ready:
Kuadrant is now ready to use.
Step 10 - Configure the Kuadrant Console Plugin¶
When running on OpenShift, the Kuadrant Operator will automatically install and configure the Kuadrant dynamic console plugin.
Enable the Console Plugin¶
To enable the Kuadrant console plugin:
- Log in to OpenShift or OKD as an administrator.
- Switch to the Admin perspective.
- Navigate to Home > Overview.
- In the Dynamic Plugins section of the status box, click View all.
- In the Console plugins area, find the
kuadrant-console-pluginplugin. It should be listed but disabled. - Click the Disabled button next to the
kuadrant-console-pluginplugin. - Select the Enabled radio button, and then click Save.
- Wait for the plugin status to change to Loaded.
Once the plugin is loaded, refresh the console. You should see a new Kuadrant section in the navigation sidebar.