Writing
Flight Patterns

Part III: Git Push to Production

At the end of Part II, Cloudflare Tunnel routes traffic to the cluster. Requests arrive. Services respond. But the question shifts: when code changes, how does it reach those pods?

Getting Code to the Cluster

kubectl apply from my laptop. The obvious starting point. Copy manifests, run the command, watch pods restart. No pipeline, no automation, no audit trail. You have to be at your desk, and "I forgot to deploy" becomes a failure mode. Fine for debugging, not a workflow.

CI pushes directly. GitHub Actions builds the image, then runs kubectl apply or helm upgrade against the cluster. Straightforward, but the pipeline needs cluster credentials—your kubeconfig lives in GitHub Secrets, and CI has production access. If the pipeline is compromised, so is the cluster.

Pull-based GitOps. An agent running inside the cluster watches the repo and pulls desired state from git. CI never touches the cluster. It builds images and updates manifests in the repo; the agent handles the rest. The repo becomes the audit trail—every deployment is a commit.

I went with GitOps because I wanted to learn it, and the model is genuinely simple once you stop fighting the YAML. Push to main, make coffee, come back to running pods. The two main options are ArgoCD and FluxCD. ArgoCD ships with a dashboard and an API server—more moving parts, more resource overhead, designed for teams that need visibility across many clusters. FluxCD is a set of controllers and nothing else. On a single node where every pod competes for the same resources, lighter wins.

The Reconciliation Loop

FluxCD watches the main branch. Every sixty seconds, it checks for new commits. When it finds one, it reconciles—compares the manifests in git to the cluster's actual state and applies whatever differs.

The manifests are split into an infrastructure Kustomization (monitoring, tunnel, databases) and an apps Kustomization (the services behind them). The apps layer declares the dependency explicitly:

# clusters/rookery/apps.yaml
spec:
path: ./infra/k3s/flux/apps
dependsOn:
  - name: infrastructure

Infrastructure must fully reconcile before applications deploy. Without this, pods start before the monitoring stack exists or before the tunnel is ready to route traffic. On a single node where everything competes for the same resources, ordering matters more than the docs suggest. I learned this the obvious way: deploying everything at once and watching pods crashloop while waiting for services that hadn't been created yet.

So the full sequence: a human pushes to main. GitHub Actions builds the changed images and a bot commits new image tags to the FluxCD manifests. Within sixty seconds, FluxCD detects the new commit. It reconciles infrastructure first—monitoring, tunnel, databases—then applications. The tunnel routes traffic to the new pods. Push to production in under five minutes, most of it Docker builds. The actual rollout takes about thirty seconds.

What Actually Changed

This is a monorepo. The homepage, the API gateway, the user service, the dashboard—six deployable services sharing TypeScript packages. A typo in a blog post should not rebuild the API gateway. A change to a shared utility should rebuild everything that imports it.

The first version of the pipeline didn't bother with any of this. Every push built every image. Six Docker builds, every time, whether anything changed or not. It worked, in the way that doing everything twice works—and the builds took long enough that I stopped ignoring it.

The fix uses pnpm's workspace-aware filtering to figure out what actually changed:

# Get all packages affected by changes since the last commit
AFFECTED_PACKAGES=$(pnpm ls -r --depth -1 --filter="...[$BASE_REF]" --json \
| jq -r '.[].name')


# Check if a specific app or any of its dependencies were affected
check_package() {
local pkg=$1
local output_var=$2


# Get this package's full dependency tree
local trigger_packages=$(pnpm ls -r --depth -1 --filter="${pkg}..." --json \
  | jq -r '.[].name')


# If any affected package is in the dependency tree, rebuild
local found=false
for affected in $AFFECTED_PACKAGES; do
  for trigger in $trigger_packages; do
    if [ "$affected" = "$trigger" ]; then
      found=true
      break 2
    fi
  done
done


echo "${output_var}=$found" >> $GITHUB_OUTPUT
}


check_package "@app/www" "www_web"
check_package "@gateway/service" "api_service"
check_package "@users/service" "user_service"
# ...

The ...[HEAD~1] filter syntax is pnpm's "what changed since this ref" query. It walks the dependency graph—if @website/utils changed, every package that depends on it shows up. Downstream build jobs use the results to decide which Docker images to build. Each Dockerfile uses turbo prune to build a minimal workspace subset, so the images stay small—the www image is just nginx serving static HTML.

A blog post change: one image, two minutes. A shared package change: everything, five minutes. A CI-only change: nothing built at all.

The Infinite Loop

The deployment pipeline builds images, then commits updated image tags to the FluxCD manifests so the cluster knows which versions to run. That commit is a push to main. Which triggers the pipeline. Which builds images. Which commits tags. Which triggers the pipeline.

There are a few ways to break the cycle. You can use a personal access token so the bot's commits come from a different identity. You can push tag updates to a separate branch that doesn't trigger CI. Or you can let the loop trigger and skip it on the way in:

jobs:
update-flux-manifests:
  if: github.actor != 'github-actions[bot]'

The bot's commit fires the workflow, the if catches it, and the run exits immediately. Simplest option, no extra tokens or branches.

After the human-triggered run builds images, a bot commits the new tags using yq:

update_tag() {
local file=$1
local built=$2
if [ "$built" = "true" ]; then
  yq e ".spec.values.image.tag = \"$TAG\"" -i "$file"
fi
}


update_tag infra/k3s/flux/apps/www-web.yaml "$WWW_WEB_BUILT"
update_tag infra/k3s/flux/apps/api-service.yaml "$API_SERVICE_BUILT"
# ...

The bot's commit triggers the pipeline again—but the if condition catches it and the workflow exits immediately. FluxCD sees the new commit, notices the image tags changed, and rolls out new pods. Loop broken.

The image tags follow the pattern selfhosted-sha-{commit_sha}. If something goes wrong, the tag on the running pod is a direct link to the exact source code that built it. No guessing which version is deployed. The SHA is right there.

Rough Edges

Secrets. The one thing that can't live in git. FluxCD solves this with valuesFrom—secrets live in Kubernetes, created once via kubectl create secret, and the HelmRelease references them by name:

valuesFrom:
- kind: Secret
  name: api-service-secrets-values
  valuesKey: googleClientId
  targetPath: secrets.googleClientId
- kind: Secret
  name: api-service-secrets-values
  valuesKey: googleClientSecret
  targetPath: secrets.googleClientSecret
- kind: Secret
  name: api-service-secrets-values
  valuesKey: jwtSecret
  targetPath: secrets.jwtSecret

The git repo never sees the credentials. The cluster holds them. If you destroy the cluster, you recreate four secrets from a password manager like a responsible adult.

The 60-second poll. FluxCD checks for new commits on an interval, not a webhook. Push, open Grafana, stare at the dashboard, wait. You know the deploy worked when the build finished. You know the cluster caught up when the pod restarts. The gap between those two moments is dead air. It's fine. It also never stops feeling slow.

No dry runs. There's no way to preview what FluxCD will do with a manifest change without pushing it to main. Flux has flux diff kustomization for local checks, but it requires cluster access, which I only have from my desk (Part II). In practice, I push and watch. For a personal project, the blast radius is small. For a team, you'd want a staging branch or a pull-request preview environment.

YAML volume. Each service needs a HelmRelease manifest, a values file, a Kustomization entry, a Dockerfile, a Helm chart, and CI workflow configuration. For six services, that's a lot of files that exist to say "deploy this container." It's the tax you pay for declarative infrastructure, and some days it feels like a lot of tax.

Setting all of this up took considerably longer than deploying with it does. But now deploying is the most boring part of my workflow—push, wait, done. That's the point. Boring deployment means interesting work happens somewhere else.