Kubernetes has revolutionized how organizations deploy and manage containerized applications, making it easier to orchestrate and scale applications across clusters. However, it comes with its own set of challenges, such as complexity, over-provisioning of resources, infra management, customization, etc.
In this blog, we will
- Deploy Virtual K8s Clusters (a.k.a. vCluster) using helm on top of vSphere Kubernetes Service (VKS) host K8s Cluster.
- vCluster will include
- Metrics & Events for Observability.
- Storage Class to allow dynamic provisioning of PVs.
- Core DNS server so that pods & services can locate each other by hostnames.
What is VMware vSphere Kubernetes Service, a.k.a. VKS?
VKS is VMware by Broadcom’s enterprise-grade, CNCF-certified and conformant Kubernetes runtime, which is included, supported, and available now with VMware Cloud Foundation.
What is Virtual K8s Cluster a.k.a. vCluster?
Virtual clusters (vCluster) are fully functional Kubernetes clusters nested inside a physical host cluster. It provides better isolation and flexibility to support multi-tenancy. Multiple teams can operate independently within the same physical infrastructure while minimizing conflicts, maximizing autonomy, and reducing costs.
We can call it a nested Kubernetes Cluster or Kubernetes on Kubernetes
Why Virtual K8s Clusters?
The CNCF identifies over-provisioning Kubernetes resources• as the primary cause of increased organisational cloud spend. vCluster tackles this by enabling the rapid creation of virtual clusters, which can isolate 100s of different tenant workloads within one cluster, enhancing density and utilization, and reducing cluster count and sprawl. Virtual clusters have their API server, which makes them better isolated than namespaces and more affordable than separate Kubernetes clusters.
•CNCF’s 2023 Cloud Native & K8s FinOps Microsurvey
Features/Benefits of Virtual K8s Clusters a.k.a. vCluster
vCluster provides the following features
Robust Security and Isolation
| Granular Permissions | vCluster users operate with minimized permissions in the host cluster, significantly reducing the risk of privileged access misuse. Within their vCluster, users have admin-level control, enabling them to manage CRDs, RBAC, and other security policies independently. |
| Isolated Control Plane | Each vCluster comes with its own dedicated API server and control plane, creating a strong isolation boundary. |
| Customizable Security Policies | Tenants can implement additional vCluster-specific governance, including OPA policies, network policies, resource quotas, limit ranges, and admission control, in addition to the existing policies and security measures in the underlying physical host cluster. |
Cost-effectiveness and reduced overhead
| Lightweight Infrastructure | Virtual clusters are significantly more lightweight than physical clusters, able to spin up in seconds, which contrasts sharply with the lengthy provisioning times often seen in cloud environments like EKS / GKE / AKS |
| Resource Efficiency: | By sharing the underlying host cluster’s resources, virtual clusters minimize the need for additional physical infrastructure, reducing costs and environmental impact. |
Improved Scalability
| Reduced API Server Load | Virtual clusters, each with its own dedicated API server, significantly reduce the operational load on the host cluster’s Kubernetes API server by isolating and handling requests internally. |
| Conflict-Free CRD Management | Independent management of CRDs within each virtual cluster eliminates the potential for CRD conflicts and version discrepancies, ensuring smoother operations and easier scaling as the user base expands |
What are the use cases for vClusters?
Several use cases make virtual clusters a key technology for advanced Kubernetes users and platform teams.
| Empower developers with self-service Kubernetes | Simplify Kubernetes access for developers through self-service virtual clusters, reducing human error and enhancing developer autonomy without compromising security and compliance requirements. This helps provide cloud like functionality in Onprem Cluster |
| Accelerate CI/CD with ephemeral Kubernetes clusters | Instantly create clean, new virtual Kubernetes clusters for each pull request, enabling fast, isolated testing and PR previews without wait times and the struggles of a shared test environment. |
| Diverse Kubernetes Environments | vCluster supports different Kubernetes versions and distributions (including K8s, K3s, and K0s), allowing version skews. This makes it possible to tailor each virtual cluster to specific requirements without impacting others. |
| Diverse Application Environments | Developers can deploy different versions of their environment. |
| Runs Anywhere |
Virtual clusters can run on VKS(TKC), EKS, GKE, AKS, OpenShift, etc on cloud, edge, and on-prem. As long as it’s a K8s cluster, you can run a virtual cluster on top of it.
|
Deploy Virtual K8s Clusters on VKS using Helm
Prerequisites
Before we start, ensure you have the following:
- A VKS Cluster is up and running(v1.28+). In my lab, I havea VKS Cluster with 1.29.4
- kubectl and Helm are installed locally.
- Access to the VKS cluster using the admin kubeconfig
Procedure
Step 1- Clone Repo
git clone https://github.com/munishpalmakhija/vcluster-deploy.git
Step 2 – Export required environment Variables
You will need to export 3 environment variables
export namespace_name=vcluster-automation-demo
export values_file=vcluster-values.yaml (You can modify based on your requirement in case you have any node selectors)
export KUBECONFIG=VKS-Cluster-Admin-Kubeconfig (This will be the host VKS cluster where vClusters will be created )
Step 3 – Execute bash script
Execute the bash script create_vcluster.sh
bash create_vcluster.sh $namespace_name $values_file $KUBECONFIG
It performs the following
- Create Namespace
- Create K8 service with Type LoadBalancer
- Deploys vcluster statefulset using helm with K8s version 1.32.1, using the default K8s distro (Remember I have a host cluster with version 1.29. That is the beauty )
- It enables the following inside the vcluster
- Coredns server
- Metrics
- Storage Class
- K8s events
Step 4 – Verify
Bash script should print the kubeconfig on stdout, which can be used to access the vCluster
You can also use the vcluster CLI to check the status. For more details, you can refer to their documentation
Conclusion
vCluster provides an efficient way to manage Kubernetes multi-tenancy, allowing developers and teams to work in isolated environments. By using Helm, deploying a vCluster is simple and integrates seamlessly with existing tooling used to manage Kubernetes Infrastructure. For more details, youcan refer to their documentation https://www.vcluster.com/docs/