Skip to content
← Back to writing

March 20, 2026

All because I wanted an accordion

A scroll fix on my portfolio site cascaded into one sixteen-hour session that rebuilt three sites, stood up dev/staging/prod, and fixed five stacked bugs in one Docker image.

It started with a scroll problem. My portfolio site had a stack of blog posts, and on mobile you had to scroll past all of them to reach the contact section. I asked my AI assistant to add an accordion: show four posts, expand to see the rest, collapse back down, the kind of thing a junior dev knocks out before lunch.

Terminal showing the real git log: the accordion commit 06835d9 in mjashley-site at 00:42, then four homelabarr-ce fix commits from 01:57 to 02:33 — c39becb, e2536e5, b9333d4, ddb343b — the actual commit history behind the five stacked bugs.

Sixteen hours later I had rebuilt three production sites, migrated a container across hosts, stood up dev/staging/prod for every project, switched my AI orchestration model, turned on two-factor auth across eleven admin panels, wired CI/CD for four repos, and written a long infrastructure playbook so this could never silently happen again.

All because I wanted an accordion.

The accordion that broke everything

The accordion itself was about five minutes of work. One useState, slice the posts array, two buttons. Here is the actual diff that landed in Writing.tsx:

// src/components/Writing.tsx  (mjashley-site 06835d9)
+'use client'
+import { useState } from 'react'
+
 export default function Writing() {
+  const [expanded, setExpanded] = useState(false)
   const posts = [ /* ... */ ]
+  const visible = expanded ? posts : posts.slice(0, 4)
+  const hiddenCount = posts.length - 4
   return (
     <section id="writing" /* ... */>
-      {posts.map((post) => ( /* ... */ ))}
+      {visible.map((post) => ( /* ... */ ))}
+      {!expanded && (
+        <button onClick={() => setExpanded(true)}>Show {hiddenCount} more posts</button>
+      )}
+      {expanded && (
+        <button onClick={() => setExpanded(false)}>Show less</button>
+      )}
     </section>
   )
 }

That part was never the problem. Everything it touched on the way to deployment is what actually broke.

A while back my AI assistant had dispatched what it called a minor update to Claude Code, and the model on the other end rewrote my entire website. Blog posts gone. Hero rewritten. Pricing changed. Navigation restructured. Deployed straight to production while I was getting ready to drive to a meeting. I wrote about that disaster on its own. What I never wrote about is the recovery, which turned into the most productive single session of infrastructure work I have done.

The session

I sat down at my terminal around 7 PM. The site was broken, my assistant could not talk to Claude Code through Discord because a token had been dead for thirteen days and nobody noticed, and I had no staging, no deploy pipeline, and no guardrail against any of it happening again.

I am going to write "we" through most of this, because it genuinely was collaborative: me directing, Claude Code executing, the two of us tracing problems in real time.

Disaster recovery

First priority was getting the site back. The good version was on the remote branch. The model had only destroyed the local copy, so git reset --hard origin/main brought the code back. But several blog posts had never been committed and existed only as markdown in a shared folder.

I recreated those pages from the original markdown, updated the Writing component to list every entry, rebuilt the static site, rsynced to the Docker host, and redeployed. Then the site started 502'ing, because the Cloudflare tunnel routed by container name and the rewritten compose file had changed the container name. The site stayed down until I found that naming dependency.

That was one site. The same vague dispatch had also hit imogenlabs.ai (an uncommitted Astro migration, nearly lost for good) and the HomelabARR CE demo (an ARM image deployed to an x86 host, so the binary literally could not execute and crash-looped for hours). Three sites, three different failure modes, all from one loosely scoped task.

Dev, staging, and production

The root cause was obvious: changes went straight from code to production with nothing in between, so I built the whole chain.

Three Docker Compose files. A deploy script that enforces promotion: dev deploys freely, staging requires dev to be healthy, prod requires staging plus a confirmation. DNS records in Cloudflare for the subdomains. Traefik routing through file-based rules, not Docker labels, which cost me an hour of debugging because nothing routed until I realized this host configures Traefik entirely through files.

Six environments live in one evening: dev and staging for both mjashley.com and imogenlabs.ai, alongside the existing production domains.

The model switch

While I was untangling the deploy mess I found why my assistant could not dispatch to Claude Code at all: a Codex OAuth token had expired on March 6th. Thirteen days of silent failures, every dispatch falling back to the CLI without a single error anyone saw.

That made me look at the orchestration model itself. The assistant was running on a model that cost nothing but routed every API call through Singapore, so every orchestrator decision carried ten-plus seconds of latency. I had benchmarked NVIDIA's Nemotron models months earlier and written them off, but I had tested them through a model proxy that mangled the results. That night I tried Nemotron directly against NVIDIA's API with no middleman, and got three-second responses with clean structured JSON and correct routing. Turned out the proxy was the problem the whole time, not the model.

Switching was not clean. The orchestrator uses a provider/model naming convention where a slash splits provider from model ID, but NVIDIA's API wants nvidia/ as a prefix on the model field itself. The orchestrator was stripping that prefix, sending a bare model name, getting 404s, and silently falling back to a different model. The fix was renaming the provider so the full path resolved, which cost me three hours on a naming convention.

Mission Control migration

Mission Control is my ops dashboard: task board, an AI roundtable, infrastructure monitoring. It was running on the Mac Mini, which shares 16 GB of RAM with the orchestrator, Claude Code, and everything else. The Docker host had headroom. It already had a Dockerfile with standalone Next.js output, so I containerized it, migrated the SQLite database, set up the Traefik route with Authelia two-factor in front, and added the Cloudflare DNS record. About fifteen minutes from "this should move" to a live URL with 2FA.

While I was in there I noticed Plex had 32 GB allocated on the Proxmox host and was using under a gigabyte, so I dropped it to 8 GB and got two dozen gigabytes back with zero impact.

Two-factor authentication

I had been running Authelia for over a year with everything on single-factor. I did not know it supported TOTP. I was honestly considering migrating to Authentik just to get 2FA.

It had TOTP the whole time. I had simply never set any domain to a two_factor policy. One config change, and eleven admin panels now want a password plus an authenticator code: Portainer, Proxmox, Mission Control, Traefik, code-server, Guacamole, n8n, Bitwarden, Dozzle, Netdata, Glances.

CI/CD and auto-updates

Every site was deployed by rsyncing source to the Docker host and building locally. No pipeline, no automation, no notifications. So I set up GitHub Actions across four site repos: push to main triggers a multi-arch Docker build, pushes to GHCR, and posts to the right Discord channel. Then Watchtower with label-based opt-in, so production containers carrying the watchtower.enable label pull from GHCR on their own. Push to main, wait a few minutes, and the site updates itself, with no SSH, rsync, or manual step.

The accordion (finally)

After all of that, I built the accordion. The diff up top is the whole feature, and it was the fifteenth thing I shipped that night.

The five bugs hiding in one Docker image

Even after the infrastructure work, the HomelabARR CE demo was still broken: five bugs stacked like nesting dolls, every one of them a side effect of that unreviewed rewrite. Here is the real chain, with the commits, all landed the same night.

The frontend was hardcoded to call http://localhost:8092 for the backend. That worked on the old host, where both containers ran on one machine and the browser was on that same machine. On the new Docker host the browser is on a completely different computer, so localhost means nothing.

// src/lib/api.ts  (homelabarr-ce c39becb)
-const API_BASE_URL = 'http://localhost:8092';  // Backend server URL for development
+// production: relative URL so requests go through the nginx proxy (/api/ → backend)
+const API_BASE_URL = import.meta.env.DEV ? 'http://localhost:8092' : '/api';

But the nginx proxy did not strip the prefix. It forwarded /api/applications straight to the backend, which only has routes like /applications, so the /api/ had to be rewritten away.

# nginx.conf  (homelabarr-ce e2536e5)
-    location /api {
+    location /api/ {
         set $backend http://homelabarr-backend:8092;
+        rewrite ^/api/(.*) /$1 break;
         proxy_pass $backend;

But the backend crash-looped, because auth.js reached for require('crypto') inside an ES module. The rewrite had mixed CommonJS and ESM in the same file.

// server/auth.js  (homelabarr-ce e2536e5)
+import crypto from 'crypto';
-const JWT_SECRET = process.env.JWT_SECRET || require('crypto').randomBytes(32).toString('hex');
+const JWT_SECRET = process.env.JWT_SECRET || crypto.randomBytes(32).toString('hex');

But even with a clean boot, the catalog returned zero apps, because .dockerignore excluded the very directories the catalog reads from.

# .dockerignore  (homelabarr-ce b9333d4)
 # HomelabARR specific
-apps/
-scripts/
-traefik/
+# NOTE: apps/, scripts/, traefik/ are NOT ignored — backend needs them for CLI Bridge

And even once those directories were in the image, the CLI Bridge looked at process.cwd() (which is /app in the container) instead of honoring the configured path. The rewrite had dropped the env-var fallback, and verifyCLIInstallation() had been throwing on a missing apps directory instead of falling back.

// server/cli-bridge.js  (homelabarr-ce ddb343b)
-    this.cliPath = process.cwd();
+    this.cliPath = process.env.CLI_BRIDGE_HOST_PATH || process.cwd();

That same commit baked apps/, scripts/, and traefik/ into Dockerfile.backend so the image works standalone, and pinned PORT=8092 (the app had been defaulting to a different port and nginx was proxying to nothing). Its commit message put the symptom plainly:

Fixes app catalog returning 0 applications when deployed via GHCR image.

That was five bugs across five commits in one night, all downstream of a single AI rewrite that nobody reviewed before it went to production.

The real lesson

This session happened because I had no guardrails. A vague task got dispatched, a model rewrote everything it touched, and the changes went straight to prod. The cascade exposed every missing piece at once: no staging, no branches, no pipelines, no monitoring, no 2FA, no auto-updates, no map of the infrastructure.

The most lasting thing I built was the documentation, not any single fix: a long infrastructure reference that maps every host, container, port, domain, Traefik route, DNS record, secret, webhook, pipeline, and Authelia policy, so that any agent or future me can read one file and understand the whole system. It covers how to add a project, how to deploy, how to roll back, and where every secret lives. At the bottom it sketches the NixOS equivalent of each piece, because someday I want this to collapse into a single declarative config that rebuilds everything from bare metal.

I run my company the way I used to run engineering teams: sprints, story points, Jira boards, an SDLC documented in Confluence. Until that night, none of it was enforced on my own infrastructure. The cobbler's children had no shoes. Now they have a deploy pipeline, two-factor auth, auto-updating containers, Discord notifications, an accordion, and a written record of how all of it fits together.

mjashley.com · imogenlabs.ai