I needed a Kubernetes cluster that could run continuously for platform engineering work - deploying services, testing GitOps workflows, running applications. The cluster needed to be production-grade in its automation and reliability, even if it wasn’t serving production traffic.
I chose k3s running on KVM/libvirt virtual machines. The entire deployment is automated: Terraform provisions three VMs, cloud-init configures networking and installs k3s, and within about five minutes a working cluster is operational. No manual steps, no configuration drift, completely reproducible.
This post walks through the architecture, the automation decisions, and the technical solutions that make it work. If you’re building Kubernetes infrastructure on virtualization platforms, some of these patterns might be useful.
The cluster runs three virtual machines managed by KVM/libvirt:
Each VM has 6 vCPUs, 10GB RAM, and 80GB disk. They run Ubuntu 24.04 LTS with the containerd runtime. The k3s version is pinned to v1.34.3+k3s1 - I’ll explain why that matters later.
The network uses libvirt’s default network (192.168.122.0/24) with static IP assignments:
Static IPs are configured via cloud-init’s network_config, not DHCP reservations. This eliminates any dependency on DHCP lease stability and ensures nodes always come up with the correct addresses.
The deployment is managed entirely through infrastructure as code. Packer builds the base Ubuntu image with k3s prerequisites. Terraform provisions the VMs using the libvirt provider. Cloud-init handles node-specific configuration and k3s installation. The entire process is declarative and reproducible.
The initial implementation used DHCP with MAC address reservations in libvirt’s network configuration. This worked, but introduced instability. Occasionally nodes would come up with different IPs or fail to get leases at all. Debugging network issues in a distributed system is painful when you’re not sure if the problem is application-level or infrastructure-level.
Static IPs configured via cloud-init eliminate this entire class of problems. Each VM’s network configuration is defined in its cloud-init network_config. The IP addresses are assigned before any services start. There’s no lease negotiation, no timing dependencies, no opportunity for the network layer to behave differently between deployments.
The trade-off is that you need to manage IP address allocation manually. For a three-node cluster, that’s trivial. For larger deployments, you’d want tooling to generate network_config from an IP allocation database. But the reliability improvement is worth it.
K3s supports installing from a “stable” channel, which automatically pulls the latest stable release. This seems convenient - you always get the newest version without manual updates.
In practice, this creates deployment instability. The k3s version can change between cluster deployments, introducing variables when troubleshooting. Different nodes might end up with different versions if they pull from the channel at different times. And k3s releases sometimes introduce breaking changes that affect existing workloads.
Pinning to a specific version (v1.34.3+k3s1 in this case) makes deployments reproducible. Every node gets exactly the same k3s binary. If I destroy and recreate the cluster six months from now, it will be identical to today’s deployment. When I’m ready to upgrade, I test the new version, update the pin, and deploy deliberately.
This is a standard practice in production environments. Version pinning trades convenience for predictability. For a platform that exists to demonstrate capabilities, predictability matters more than running the absolute latest release.
The k3s installation script runs automatically via cloud-init. If networking isn’t fully operational when k3s starts, the installation can fail in subtle ways - certificates might be generated with wrong IPs, the API server might bind to the wrong interface, nodes might not be able to join the cluster.
The cloud-init configuration includes network validation before k3s installation:
runcmd:
- |
# Wait for network interface to be operational
for i in {1..30}; do
IFACE=$(ip -o -4 route show to default | awk '{print $5}' | head -n1)
if [ -n "$IFACE" ]; then
echo "Network interface $IFACE is up"
break
fi
echo "Waiting for network interface... ($i/30)"
sleep 2
done
- |
# Verify DNS resolution
for i in {1..30}; do
if nslookup google.com > /dev/null 2>&1; then
echo "DNS resolution working"
break
fi
echo "Waiting for DNS... ($i/30)"
sleep 2
done
- |
# Install k3s after network is confirmed operational
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.34.3+k3s1 sh -s - server
This adds maybe 10-15 seconds to deployment time, but eliminates an entire class of timing-related failures. The k3s installation doesn’t start until the network is confirmed working. Simple, reliable, worth the wait.
The Packer template builds a base Ubuntu 24.04 image with k3s prerequisites installed. Early versions of this image had problems: hardcoded MAC addresses from the build environment, residual netplan configurations, cloud-init state that didn’t reset properly between VM deployments.
These issues caused VMs to come up with duplicate MAC addresses or incorrect network configurations. The solution was a cleanup script that runs at the end of the Packer build:
# Remove machine-specific identifiers
rm -f /etc/machine-id
rm -f /var/lib/dbus/machine-id
# Clean cloud-init state
cloud-init clean --logs --seed
# Remove netplan configs (cloud-init will regenerate)
rm -f /etc/netplan/*.yaml
# Clean logs
find /var/log -type f -delete
This ensures the base image is truly generic. Each VM that boots from this image gets fresh identifiers, clean cloud-init state, and network configuration from its own cloud-init data source.
The Terraform configuration provisions VMs using the dmacvicar/libvirt provider. For each node, it creates:
The libvirt provider connects to a remote libvirtd instance over SSH. This means Terraform runs on my laptop but manages VMs on a separate Linux host. The provider handles the SSH connection transparently.
Key Terraform resources:
resource "libvirt_cloudinit_disk" "k3s_cp" {
name = "k3s-cp-01-cloudinit.iso"
user_data = templatefile("${path.module}/cloud-init/k3s-cp.yml.tpl", {...})
network_config = templatefile("${path.module}/cloud-init/network-config.yml.tpl", {...})
}
resource "libvirt_volume" "k3s_cp" {
name = "k3s-cp-01.qcow2"
base_volume_id = libvirt_volume.base.id
size = 85899345920 # 80GB
}
resource "libvirt_domain" "k3s_cp" {
name = "k3s-cp-01"
memory = 10240
vcpu = 6
cloudinit = libvirt_cloudinit_disk.k3s_cp.id
disk {
volume_id = libvirt_volume.k3s_cp.id
}
network_interface {
network_name = "default"
addresses = ["192.168.122.10"]
wait_for_lease = false
}
}
The wait_for_lease = false parameter is important. It tells Terraform not to wait for a DHCP lease, since we’re using static IPs. Without this, Terraform would hang waiting for a lease that will never come.
The cloud-init configuration has two parts: user_data (what to do) and network_config (how to configure networking).
The network_config is straightforward - static IP, gateway, DNS servers:
version: 2
ethernets:
ens3:
addresses:
- 192.168.122.10/24
gateway4: 192.168.122.1
nameservers:
addresses:
- 8.8.8.8
- 1.1.1.1
The user_data is more complex. For the control plane:
#cloud-config
hostname: k3s-cp-01
fqdn: k3s-cp-01.local
users:
- name: ubuntu
ssh_authorized_keys:
- ${ssh_public_key}
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
runcmd:
# Network validation (shown earlier)
# K3s installation
- curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.34.3+k3s1 sh -s - server
# Wait for k3s to be ready
- |
for i in {1..60}; do
if sudo k3s kubectl get nodes > /dev/null 2>&1; then
echo "k3s control plane is ready"
break
fi
sleep 5
done
For worker nodes, the configuration is similar but joins the cluster instead of initializing it:
runcmd:
# Network validation
# K3s agent installation
- |
curl -sfL https://get.k3s.io | \
INSTALL_K3S_VERSION=v1.34.3+k3s1 \
K3S_URL=https://192.168.122.10:6443 \
K3S_TOKEN=${k3s_token} \
sh -s - agent
The k3s token comes from Terraform variables and is templated into the cloud-init configuration. In a production environment, you’d want to manage this more securely (maybe HashiCorp Vault or AWS Secrets Manager), but for a demonstration cluster, templating it directly works fine.
From a clean state, deploying the cluster:
# Terraform provisions VMs and attaches cloud-init
terraform apply
# Wait ~5 minutes for cloud-init to complete
# VMs boot, network configures, k3s installs
# Verify cluster is operational
ssh ubuntu@192.168.122.10 'sudo k3s kubectl get nodes -o wide'
The entire process takes about five minutes. Most of that is waiting for VMs to boot and cloud-init to run. The actual k3s installation is fast once the network is ready.
Destroying and recreating the cluster:
terraform destroy
terraform apply
# Another 5 minutes, identical cluster
Complete reproducibility. No manual steps. No configuration drift.
This cluster is the foundation for platform services. The next steps are:
Install Flux GitOps controllers to manage platform and application deployments declaratively. Flux will sync from Git repositories and keep the cluster state in sync with the desired state defined in code.
Deploy Big Bang platform services - the DoD-maintained DevSecOps baseline that provides Istio service mesh, Prometheus monitoring, GitLab CI/CD, and other core capabilities. This gives the cluster a production-grade platform layer.
Add applications - once the platform is operational, deploy actual applications to demonstrate the full stack working together.
AWS compatibility - This architecture is designed with Kubernetes portability in mind. The application layer uses standard Kubernetes primitives. Environment-specific infrastructure differences (storage classes, load balancers, ingress) are handled through Kustomize overlays. The same manifests and GitOps workflows that work on k3s can deploy to AWS EKS.
Future posts will cover the Flux and Big Bang deployment process, and eventually demonstrate AWS integrations using services like Direct Connect, Storage Gateway, and Outposts.
Static networking is worth the small amount of extra configuration. DHCP adds moving parts and timing dependencies. For infrastructure that needs to be reliable, static IPs eliminate an entire class of problems.
Version pinning trades convenience for predictability. Always getting the latest version sounds good until you need to debug why a deployment behaves differently than it did last month.
Network validation before service installation prevents subtle failures. The 10-15 seconds spent confirming DNS works and interfaces are up saves hours of debugging certificate problems and API server binding issues.
Clean base images are not optional. Residual state in images causes mysterious problems that are hard to diagnose. Taking the time to properly clean machine IDs, cloud-init state, and network configs is essential.
Automation enables experimentation. Being able to destroy and recreate the cluster in 5 minutes means you can test ideas without fear. If something breaks, just rebuild. This changes how you approach learning and troubleshooting.
The cluster runs continuously now, stable and operational. It’s ready for the platform services layer. The automation is solid enough that I trust it, which means I can focus on building the interesting parts - the platform and applications - instead of fighting infrastructure problems.
]]>Running PostgreSQL in a multi-tenant configuration is a powerful cost-optimization strategy—especially in environments where dozens or hundreds of isolated workloads coexist. But as I wrote previously in Operational Realities of Running PostgreSQL, security isolation is only half the story.
PostgreSQL is extremely stable when respected, but it has sharp edges when pushed into resource contention. Multi-tenant architectures amplify those failure modes. Even with perfect security isolation (role-per-tenant, database-per-tenant, schema hardening), tenants still share a single set of physical resources:
If one tenant misbehaves, it can degrade the experience for all others—even without violating a single privilege boundary.
This post explains the operational guardrails required to ensure safe, predictable, and compliant multi-tenant PostgreSQL deployments. All guardrails described here are fully implemented and verifiable in the accompanying project:
Project: Multi-Tenant PostgreSQL Security & Operational Isolation
Multi-tenant PostgreSQL is only viable when both of these are true:
No tenant should ever be able to read or affect another tenant’s data.
No tenant should be able to destabilize the shared database server.
The first requirement is handled by:
public schemasearch_pathThe second requirement requires operational guardrails, which this post covers in detail.
Both sets of controls are implemented and actively tested in the project linked above.
Every PostgreSQL instance has a global connection budget (max_connections). All tenants draw from this shared pool.
A single tenant with:
…can exhaust all connections and knock the instance offline.
ALTER ROLE tenant_a_app CONNECTION LIMIT 2;
Small limits dramatically reduce blast radius.
The test suite spawns multiple concurrent connections and confirms one fails once the limit is exceeded.
A single long query—or a stuck transaction—can tie up CPU, I/O, locks, and memory.
ALTER ROLE tenant_a_app SET statement_timeout = '3s';
ALTER ROLE tenant_a_app SET lock_timeout = '2s';
ALTER ROLE tenant_a_app SET idle_in_transaction_session_timeout = '10s';
These serve as circuit breakers against runaway behavior.
SELECT pg_sleep(10) is used to confirm the timeout fires predictably.
Long-lived locks stop autovacuum from doing its job. The result:
In multi-tenant environments, all tenants suffer.
These are documented operational expectations for production RDS deployments.
PostgreSQL’s background processes operate at the instance level:
A high-churn tenant can degrade performance for everyone.
On AWS RDS, a snapshot contains all tenant databases.
This is essential for IL4 workloads.
public schemasearch_pathAvoid multi-tenant PostgreSQL when:
These constraints are architectural realities, not limitations of PostgreSQL itself.
Multi-tenant PostgreSQL can be secure, cost-effective, and IL4-aligned — but only when operational guardrails are enforced. These include:
The accompanying project provides a complete, reproducible reference architecture
Upcoming work: Terraform integration using the PostgreSQL provider, RDS automation, and CI/CD validation pipelines.
]]>Modern development environments move fast.
Package managers update. Libraries deprecate APIs. Defaults change. Previously working builds suddenly fail.
This post documents a real case of toolchain drift on macOS involving Homebrew and OpenSSL, and—more importantly—how to reason through recovery when the ecosystem moves out from under you.
This is not a recommendation to stay on old versions indefinitely.
It’s about getting unstuck responsibly.
Toolchain drift usually presents as:
In this case, the symptoms appeared after routine updates to Homebrew and OpenSSL.
Nothing in the application code changed.
The environment did.
On macOS, Homebrew:
OpenSSL:
When those two collide, downstream tooling often breaks first.
This is not negligence. It’s the cost of a fast-moving ecosystem.
At the moment of failure:
The goal was restoration of functionality, not architectural perfection.
The chosen approach was to:
This is a containment strategy, not a permanent fix.
The recovery involved:
Commands like the following were used during diagnosis and recovery:
brew info openssl
brew install openssl@1.1
brew unlink openssl
brew link openssl@1.1 --force
The exact commands matter less than the intent:
Restore the environment the software was built against.
Once the expected library versions were present again, the failures disappeared. Nothing about the application itself had changed. The mismatch between the application’s expectations and the system-provided libraries was the entire problem.
Most native tooling is sensitive to ABI and linking changes. Tools often assume specific library versions, paths, or symbols will exist. When those assumptions are violated, failures surface in ways that look unrelated to the actual cause.
By reverting to a known-good toolchain, the assumptions held again. The system returned to a stable state without modifying application code.
This is why toolchain drift so often manifests as “random” build failures. The failure is deterministic, but the dependency chain is opaque.
Downgrading or pinning dependencies is not without cost.
It can:
This approach should always be treated as temporary. It is a recovery technique, not a long-term strategy.
The important part is not the downgrade itself, but the discipline around it: documenting the change, understanding why it was necessary, and planning how to move forward.
With more time and less pressure, better long-term solutions include:
The goal is not to freeze the environment forever, but to control when and how it changes.
When toolchain drift causes failures:
Stability first. Improvements second.
Fast-moving ecosystems are powerful, but unforgiving.
Toolchain drift is not a personal failure or a lack of skill. It is a reminder that reproducibility is something you must design for.
Sometimes the correct move is not forward.
It is back — briefly, intentionally, and with full awareness of why.
]]>PostgreSQL is often treated like a dependency:
In reality, PostgreSQL is a stateful system with strong opinions about memory, disk, and durability. When those expectations aren’t met, performance problems and outages tend to look mysterious.
This post captures practical realities of running PostgreSQL in production—especially in containerized and Kubernetes environments—without turning into a tuning checklist.
PostgreSQL:
You don’t “embed” Postgres. You host it.
Treating it like a stateless service almost always leads to surprises.
One of the most common misconceptions is that PostgreSQL memory usage scales primarily with data size or query complexity.
In practice, it scales with connections.
Each connection:
A large number of idle connections can be just as harmful as active ones.
This is why:
When PostgreSQL is slow, adding CPU is often the first instinct.
In reality, PostgreSQL performance issues are more commonly caused by:
CPU becomes a bottleneck after those are addressed.
PostgreSQL’s durability guarantees rely heavily on the Write-Ahead Log (WAL).
This means:
Slow or inconsistent disks show up as:
This is especially important in virtualized or networked storage environments.
Running PostgreSQL in a container does not change how PostgreSQL works.
It still:
Common container mistakes include:
Containers change packaging, not physics.
Kubernetes can help manage PostgreSQL, but it does not remove operational requirements.
In Kubernetes:
If the storage layer is slow or misconfigured, PostgreSQL will faithfully surface those problems.
PostgreSQL defaults prioritize:
They are intentionally conservative.
This is good for safety, but it means:
Understanding why a setting exists matters more than memorizing values.
PostgreSQL is verbose when asked correctly.
Key signals include:
When Postgres is unhealthy, it usually tells you—just not always in the place people look first.
A few patterns show up repeatedly in troubled deployments:
None of these fail immediately. They fail under load.
PostgreSQL rewards understanding. It punishes assumptions.
Most PostgreSQL outages aren’t caused by bugs.
They’re caused by mismatches between:
Once you treat Postgres as a system with real physical constraints, its behavior becomes predictable—and manageable.
]]>CI/CD systems frequently need non-interactive access to Kubernetes clusters.
Historically, this was straightforward:
In Kubernetes 1.24 and later, that workflow quietly broke.
In a CI/CD environment:
This requires a long-lived credential.
Before Kubernetes 1.24:
Many pipelines relied on this behavior.
Starting in Kubernetes 1.24:
This improves security but breaks external CI workflows.
CI systems:
The ServiceAccount exists. RBAC is correct. But no token exists to authenticate with.
You can confirm this by inspecting the ServiceAccount:
kubectl get serviceaccount deployer-service-account -o jsonpath='{.secrets}'
If the output is empty, no token Secret exists.
Bound tokens:
Secret-based tokens:
When CI access is required, a token Secret can be created manually:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: deployer-sa-token
annotations:
kubernetes.io/service-account.name: deployer-service-account
type: kubernetes.io/service-account-token
EOF
Kubernetes will populate the token automatically.
Manually created tokens:
They should be treated as exceptions, not defaults.
More robust approaches include:
These eliminate static tokens entirely.
The system did not break. The defaults changed. The model finally caught up with security reality.
]]>kubectl feels simple once it works.
But when access breaks—or when you need to create access from scratch—the kubeconfig file suddenly becomes mysterious. Tokens, certificates, contexts, users, clusters: everything is there, but rarely explained clearly.
This post explains what a kubeconfig actually is, how it’s structured, and how to create or modify one intentionally instead of relying on magic.
A kubeconfig file is not credentials.
It is a configuration document that tells kubectl:
Think of it as a connection profile, not a secret store.
Every kubeconfig is built from four pieces.
Defines:
Defines:
Binds:
Tells kubectl which context to use by default.
Nothing works unless all four line up.
To see what kubectl is currently using:
kubectl config view
To see only the active context:
kubectl config current-context
To list all contexts:
kubectl config get-contexts
These commands are often enough to diagnose access confusion.
Creating a kubeconfig intentionally makes the model click.
kubectl config set-cluster example-cluster \
--server=https://api.example.internal:6443 \
--certificate-authority=/path/to/ca.crt
This tells kubectl where the API server is and how to trust it.
Example using a client certificate:
kubectl config set-credentials example-user \
--client-certificate=/path/to/client.crt \
--client-key=/path/to/client.key
Other authentication methods exist, but the structure is the same.
kubectl config set-context example-context \
--cluster=example-cluster \
--user=example-user \
--namespace=default
This binds identity to destination.
kubectl config use-context example-context
At this point, kubectl is fully configured.
By default, kubectl looks for:
~/.kube/config
You can override this with:
KUBECONFIG=/path/to/config kubectl get pods
Multiple kubeconfig files can be merged automatically via the KUBECONFIG environment variable.
Most access mistakes are context mistakes, not auth failures.
Common issues include:
Always check the context before acting.
A kubeconfig:
Authorization is enforced by:
If access is denied, the kubeconfig is usually fine—the permissions are not.
Once authentication is working, the next question is authorization.
kubectl auth can-i answers a simple but critical question:
“Is this identity allowed to do this action?”
kubectl auth can-i get pods
This checks whether the current context’s user is allowed to list pods in the current namespace.
kubectl auth can-i create deployments -n example-namespace
This avoids false assumptions caused by the active namespace.
kubectl auth can-i list nodes
This verifies permissions that are not namespace-bound.
kubectl auth can-i
Is the fastest way to distinguish between:
If the command returns no, authentication succeeded but permissions are insufficient.
If the command errors, the kubeconfig itself may be misconfigured.
Before debugging:
Run:
kubectl auth can-i <verb> <resource>
It turns RBAC from guesswork into something concrete.
Understanding kubeconfig reduces both mistakes and anxiety.
Once this clicks:
The file didn’t change—your understanding did.
]]>Most engineers know that there’s a difference between decimal and binary byte units.
Fewer engineers can confidently say:
This post explains byte size units in the way that’s actually useful in practice—without turning it into a standards lecture.
There are two byte size systems in common use:
Used primarily for:
1 KB = 1,000 bytes
1 MB = 1,000,000 bytes
1 GB = 1,000,000,000 bytes
1 TB = 1,000,000,000,000 bytes
These scale cleanly by powers of 10.
Used primarily by:
1 KiB = 1,024 bytes
1 MiB = 1,048,576 bytes
1 GiB = 1,073,741,824 bytes
1 TiB = 1,099,511,627,776 bytes
These scale by powers of 2.
The confusion comes from history.
For years, binary quantities were labeled using decimal names:
That shorthand stuck—long after it became misleading.
The IEC standard introduced:
Not to complicate things—but to be precise.
In practice, you’ll most often see:
This is why a “1 TB disk” doesn’t show up as “1 TB” in your OS.
Nothing is missing. Nothing is broken.
The units changed.
A disk advertised as 1 TB contains:
1,000,000,000,000 bytes
Your OS reports in GiB:
1,000,000,000,000 ÷ 1,073,741,824 ≈ 931 GiB
That ~7% difference is expected.
It’s not overhead. It’s arithmetic.
You should pay attention to units when:
This is especially true in:
You can often ignore the distinction when:
Just don’t mix unit systems mid-calculation.
A simple rule of thumb:
When precision matters, check the unit label explicitly.
If the tool says GiB, believe it.
This confusion persists because:
Understanding the distinction once prevents years of second-guessing.
It’s a small mental model with a long shelf life.
]]>Understanding how Kubernetes storage works is one thing.
Actually enabling that capability in a cluster is another.
If you want dynamic NFS-backed Persistent Volumes, Kubernetes needs a component that can:
That component is the NFS Subdir External Provisioner.
This post focuses on installing it intentionally using Helm—and understanding what you’re enabling when you do.
The NFS Subdir External Provisioner:
Kubernetes itself does not talk to NFS directly.
This provisioner is the bridge.
Before installing anything, you need:
If the NFS server isn’t healthy, this installation will succeed—but provisioning will not.
First, add the Helm repository that hosts the chart:
helm repo add nfs-subdir-external-provisioner \
https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm repo update
This makes the chart available locally.
The core installation uses helm install with a small but important set of values.
Example:
helm install nfs-provisioner \
nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
--namespace storage-system \
--create-namespace \
--set nfs.server=192.0.2.50 \
--set nfs.path=/exports/kubernetes \
--set storageClass.name=managed-nfs
Key values explained:
nfs.server
Address of the external NFS server
nfs.path
Base directory where subdirectories will be created
storageClass.name
The StorageClass PVCs will reference
This command installs the provisioner and registers a new StorageClass.
After installation, you should see:
Helm handles object creation, but you are responsible for understanding the consequences.
Check that the pod is running:
kubectl get pods -n storage-system
Confirm the StorageClass exists:
kubectl get storageclass
You should see the managed-nfs StorageClass listed.
Create a simple PVC referencing the StorageClass:
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-nfs-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: managed-nfs
resources:
requests:
storage: 1Gi
EOF
If provisioning works:
This confirms end-to-end functionality.
If the reported size appears smaller than expected, this is often a unit conversion issue rather than a provisioning failure.
See: Understanding Byte Size Units (Without Overthinking Them)
If things don’t work, check:
nfs.server and nfs.pathMost failures are external to Kubernetes.
This approach works well for:
ReadWriteManyIt may not be appropriate for:
NFS is a tool, not a default.
This installation enables the architecture described in:
How NFS-Backed Persistent Volumes Actually Work in Kubernetes
Understanding the model first makes this step predictable instead of magical.
Helm just applies the intent—you still own the system.
]]>Most Kubernetes resources delete cleanly.
Namespaces are the exception.
When a namespace gets stuck in Terminating, it’s usually not because Kubernetes is broken—it’s because Kubernetes is waiting for something else to finish its job.
Understanding why that happens requires understanding finalizers.
Deleting a namespace is not a single operation.
When you run:
kubectl delete namespace example-namespace
Kubernetes:
If any step stalls, the namespace remains in Terminating.
A finalizer is a promise.
It says:
“Do not delete this object until I have cleaned something up.”
Finalizers are commonly added by:
They exist to prevent data loss and orphaned infrastructure.
The downside: if the controller is gone or broken, the promise is never fulfilled.
Namespaces typically get stuck when:
At that point, Kubernetes is waiting for a cleanup step that will never occur.
First, verify the namespace state:
kubectl get namespace example-namespace
If it shows:
STATUS Terminating
Inspect it more closely:
kubectl describe namespace example-namespace
Often, you’ll see references to remaining resources or finalizers.
For deeper inspection:
kubectl get namespace example-namespace -o json
Look specifically at:
spec.finalizers
Commands like:
kubectl delete namespace example-namespace --force --grace-period=0
are commonly tried—and commonly ineffective.
That’s because:
Force only skips graceful termination, not cleanup guarantees.
⚠️ This is an administrative recovery action.
You are explicitly telling Kubernetes to stop waiting.
Proceed only when:
kubectl get namespace example-namespace -o json > namespace.json
Edit namespace.json and remove the finalizers field entirely.
Before:
"spec": {
"finalizers": [
"kubernetes"
]
}
After:
"spec": {}
kubectl replace --raw "/api/v1/namespaces/example-namespace/finalize" \
-f namespace.json
This bypasses the normal deletion workflow and tells the API server:
“Delete this namespace now.”
If successful, the namespace disappears immediately.
Removing finalizers means:
This is why this approach is corrective, not routine.
This approach is appropriate when:
In practice, this is often the only viable path forward.
A few practices reduce the odds of hitting this:
Finalizers are powerful—but they require discipline.
This is one of those Kubernetes behaviors that feels mysterious—until it isn’t.
Once you understand the contract, the fix becomes deliberate instead of desperate.
]]>Kubernetes Secrets are often introduced early, but rarely explained clearly.
Most examples focus on how to create a Secret, not:
This post focuses specifically on creating Secrets from the command line using kubectl create secret, and—just as importantly—when not to do that.
At a high level, kubectl create secret:
It does not:
It is a creation mechanism, not a secrets management system.
The most direct pattern uses literals:
kubectl create secret generic example-db-creds \
--from-literal=username=example_user \
--from-literal=password=example_password
This is useful for:
It is not ideal for long-lived or production secrets.
A more common pattern is file-based creation:
kubectl create secret generic example-config \
--from-file=application.yaml
This creates a Secret where:
This works well for:
But it still raises questions about where that file lives and how it’s protected.
Environment-style files can also be used:
kubectl create secret generic example-env \
--from-env-file=.env
This is convenient, but dangerous if:
.env files are committed accidentallyConvenience and risk scale together here.
By default, Secrets are created in the current namespace.
This is one of the most common failure modes.
Always be explicit:
kubectl create secret generic example-db-creds \
--from-literal=username=example_user \
--from-literal=password=example_password \
-n example-namespace
Secrets in the wrong namespace are indistinguishable from missing secrets.
You can confirm a Secret exists with:
kubectl get secret example-db-creds -n example-namespace
And inspect metadata with:
kubectl describe secret example-db-creds -n example-namespace
Avoid decoding values casually unless you need to verify wiring.
If you do decode, do it intentionally and clean up afterward.
This approach works well when:
It’s a mechanical tool, not a long-term strategy.
Problems arise when:
At scale, this approach does not age well.
As systems mature, secrets creation usually moves toward:
In those models:
kubectl create secret is often replacedThat’s not a failure—it’s progress.
kubectl create secret is about object creation, not securitySecrets are less about syntax and more about discipline.
Understanding the limits of your tools is part of operating Kubernetes responsibly.
]]>