WRITING
243 posts
Building software, running infrastructure, working with AI. Written while shipping, not after.
243 posts
187 days with posts in the last year
The box that refuses to start with the wrong state
GeneralEight.ly OS now keeps everything a box writes in one state area, and the service refuses to start if that area is missing or foreign, because a fresh database on an existing array is worse than no start at all.
A backup that has to work while the database is being written
GeneralMemIndex had no backup engine, only a one-time cleanup copy. I built local snapshots and a restore command in about an hour, and every commit after the first one was a case the first version got wrong.
Three sites, one critical flaw, and a USB cable
GeneralAttacks on a Next.js flaw were reported worldwide, so I patched three of my own sites before 6 a.m. Later the Mac froze for two hours, and the cause was a cable.
What I took out of the repository before anyone could read it
GeneralA public repository is read line by line by strangers. On October 21 I removed a website, a docs folder, 41 audited files, every ticket number and the comments from MemIndex's source, and left the published history alone.
The pull request that could run code on my machines
BuildingBefore MemIndex goes public I asked who could make my CI machines run their code. The answer was anyone who could open a pull request from a fork, and the fix was one line repeated on every job.
Twenty-two thousand lines left the repo, and each one was parked first
AI & AgentsMemIndex is going public, so on October 19 I moved eight agent packages and deleted 21,998 lines of inherited features. The order mattered more than the diff.
A failure in five seconds is not a code failure
BuildingMemIndex pull requests kept failing typecheck in five to nine seconds. The code was fine. A CI runner had refused the job because its disk was 97 percent full, and the filler was my own backups.
CI reported green on a third of the suite
BuildingA test that built an object the wrong way made the whole test run exit with code 0 partway through, and CI called it a pass. Finding it changed how I count what a green check covers.
8 rules for a viral product I learned building Eight.ly
BuildingMarc Lou wrote 31 principles of a viral product. I kept eight, checked each one against the live eight.ly site, and wrote down where it passes and where it does not yet.
Getting 30 hours of lost notes back, and a site for the tool that holds them
BuildingReplaying a lost memory database from session transcripts, then building the seven-page MemIndex site with an indexability check in CI.
A test run deleted my memory database
GeneralA test suite for MemIndex removed the real data folder on my Mac mini at 11:17. What was lost, what came back, and the guard that now stands between any delete and a folder that matters.
The Create Container dialog offered three images that could never be created
InfrastructureA container dialog that listed images nobody could build, an error message that blamed the user, and a rate limiter that counted the cluster's own traffic as an attacker.
The second array took the first one's parity
GeneralCreating a second array on an Eight.ly OS box quietly erased the first array's parity settings. Here is how the fix was proven on a test machine with three arrays.
Six VMs paused and nobody told me
GeneralA full storage pool paused six virtual machines on one of my boxes, two of my sites went down, and Eight.ly said nothing. What filled the pool, what I changed, and a runner guard that took three runners offline the same day.
Rock'em Sock'em Agents, and the bug I found in Docker Agent
AI & AgentsA weekend game built from three AI agents talking over A2A turned up a real bug in Docker's open-source Docker Agent. I reproduced it, found the cause, and sent a fix upstream.
The day I stopped installing things by hand
GeneralSix Eight.ly OS releases in a day, a Services screen that can finally retire a hand-made job, and a sweep of the app catalogue that touched thousands of health checks.
File-sharing users never needed a shell
BuildingA file-sharing account on Eight.ly OS was created with a full login shell it had no use for. The fix shipped as a security release the same morning, along with a migration for every box that already had the old accounts.
Movviz, our first app request from outside
BuildingA user asked for one app in the HomelabARR CE catalogue. Taking it all the way to a public address through Cloudflare and Authelia surfaced a bug that had been dropping the ports of 539 apps, and a status message that gave up two minutes too early.
The cleanup job that broke 24 containers
InfrastructureMy domains went dark because the Shelver, Eight.ly's nightly cache mover, had been deleting empty folders inside running containers for five days. Here is how I found it, proved it on a throwaway VM, and fixed it.
The Coral worked. The address did not.
GeneralA real USB Coral on ms01 ran Frigate's detection at 8.39 ms from an App Store install. Then I opened frigate.mjashley.com, the way a customer would, and got nothing.
One ticket, as written
AI & AgentsI handed an AI agent an eight-point ticket for an MCP server on Eight.ly OS. It came back as four sub-tasks, three agents and a design call nobody asked for. Here is how that evening went, and what got built.
A zombie every ten seconds
AI & AgentsMy lab box rolled back an update, and while I was looking at why, its process panel reported 115,420 zombie processes. One health check line in the remote agent's image had been leaving them behind for weeks.
The cache that unmounted its own apps
InfrastructureDoubling the NAS cache with the product's own conversion worked, then systemd pulled the mounts out from under 18 containers. The fixes shipped the same afternoon.
One error message, six reviews
BuildingI capped AI code review at one round after it spent half an hour on a single licence error, then gave the code that touches parity three rounds four hours later.
The fan chip that was there all along
InfrastructureFor a month my production NAS showed no fan speeds because a probe said there was no chip. There was a chip. The kernel just did not know its name.
The day Copilot ran out of reviews
BuildingCopilot's review quota ran out in under a day, so I moved code review for every Eight.ly repo to Codex. Then one Calendar pull request took seven Codex reviews to merge.
Twelve hours on someone else's clock
InfrastructureA backup server release that sat overnight behind GitHub's storage quota, the walls behind that one, and the bug it uncovered once it finally shipped.
The backup server had never deleted anything
InfrastructureMoving 832 GB of backups off a full system disk on the R730xd, and finding out along the way why the disk kept filling.
The sprint that kept growing
BuildingA bug sprint that started with six tickets had 32 by Monday morning. What went wrong, the one-hour sprint that replaced it, and the rule I had to restate twice before it stuck.
The backup that never left the box
InfrastructureFor days I was told the backup problem was a drive I needed to buy. The real problem was that every backup bound for the backup server had to fit on the box's own boot disk first.
A check that could not see purple
GeneralEight.ly OS's screen-picture suite passed 169 of 169 with the dark theme repainted purple. The causes were a colour tolerance, an area budget, a chart race and a missing glyph, and fixing them took most of the day.
Signed the long way
Buildingv2.3.54 went out unsigned. Before cutting v2.3.55 I checked every runner, found that none of the 11 could sign, and built a release job that refuses to build on a machine that can't.
Five bugs, no shared cause
BuildingFive fixes that landed on Eight.ly OS, side by side, with nothing joining them but the repo they landed in.
The warning was in a run that was already red
Generalv2.3.54 went out without its signature files, and the pipeline said so. It said so inside a run that was red for a different reason, so nobody read it.
Two packages to six: what the widened race gate found
BuildingThe Go race detector in Eight.ly OS CI only covered two packages because nobody had measured the cost of covering more. I measured it, widened it to six, and the first run caught a real race.
The night the parked tickets got answers
GeneralTwo Eight.ly OS tickets had been parked for a human ruling after three failed review rounds. I wrote the rulings and both closed within hours, alongside four more tickets, and the day ended on a different kind of mistake.
Two fixes that were never one story
BuildingTwo separate fixes landed the same day in Eight.ly OS, one in a Go file and one in a React component, and I want to keep them separate.
Green means nothing, false means nothing
GeneralFive small fixes in Eight.ly OS, all about the same failure: the dashboard was showing a confident answer when it should have shown that it did not know.
Thirty-eight photos and a site that showed nothing
AI & AgentsThe day I photographed Eight.ly OS on a clean install, found two reboot bugs while doing it, and took eight.ly and imogenlabs.ai public to pages that did not use the pictures.
Reading my own installer as a stranger
GeneralA walk through a clean install of Eight.ly OS turned up an installer full of words only I understand. Forty-one pull requests later, plus the reboot that hung on a black screen.
The four hours I lost to my own to-do list
Infrastructurev2.3.51 shipped and encrypted backups finally landed on the backup server, then I found 30 things wrong with the installer and fixed none of them for four hours.
Walking every screen twice
BuildingA full pass over the trialbox dashboard in dark and light mode turned up fifty six defects, most of them the same bug in different clothes: the screen telling you something that isn't true.
The Light theme was never finished
GeneralA screen-by-screen walk of Eight.ly OS in Light mode turned up nine real defects: unreadable text, a dead help link, a doubled version string, and a service that could never have started.
Two things that answer as if nothing went wrong
GeneralTwo fixes in Eight.ly OS, both about a check that answers success when it never actually ran.
Five ways a failed probe was allowed to lie
GeneralA look at five open tickets in Eight.ly OS where a failed check was reporting a false positive instead of saying it failed.
Two ways I broke find and replace in one day
AI & AgentsThe licensing model for Eight.ly OS got signed, and the day I spent shipping it into copy taught me the same lesson twice.
Same value, three different truths
BuildingFive bugs in Eight.ly OS where a probe that failed and a probe that found nothing produced the exact same reading on screen.
A green check has an age
GeneralEleven tickets closed, three releases cut, 91 commits landed in eightly-os, and five mistakes in the same session that all trace back to the same habit.
Settings that saved and were never read
InfrastructureThe System Name field on the General tab had never worked. Neither had most of the API tab, or the external IP field, and the container error toast was swallowing what Docker actually said.
639 files that were not mine
BuildingA night of fixing hardware-truth bugs on the eightly-os General tab ended with a commit that swept up 639 files of my own in-progress work under someone else's ticket.
Seven releases in twenty-five hours
GeneralCutting a release used to be a careful afternoon. It is now boring enough to do seven times in a day, which took three fixes to the guard that was supposed to make it safe.
Replication, spindown, trim, and handing a device to a virtual machine
BuildingA day of the unglamorous features that separate a storage appliance from a Linux box with a web page on it.
Unplugging a cable locked the whole machine
InfrastructurePulling a Thunderbolt enclosure hard-locked the NAS and left the controller stuck until the power cord came out. Fixing that turned into a day about who owns the hardware in a box.
Four gates that could never fire
GeneralI have a set of automated rules that stop me doing the wrong thing. Four of them had never once been able to trigger, and every one of them had been reporting green for months.
The loading bars were the same colour as the card
GeneralI kept saying there was no loading state on the Storage page. There was one, it rendered, it animated, and it was painted the exact hex value of the surface behind it.
The array you can actually operate
GeneralStop the array. Start it. Add a disk. Replace a dead one. Test a drive before you trust it. All of that was either a 501 or did not exist.
Rendered, clickable, and doing nothing
GeneralEight buttons on the storage pages did nothing at all. Twenty-two date fields reported the year 1. A running burn-in said it had finished before the Roman Empire.
Nova on a four gigabyte Pi
AI & AgentsA Raspberry Pi 400 took ten minutes to fail at a question the model can answer in eighteen seconds. Every reason was a decision I had made on a machine with more memory.
A tool you cannot run is not a backup you cannot trust
GeneralMy backup verifier declared 55 healthy database dumps unrestorable, because it was missing a program. It was reporting the wrong failure, and that is worse than reporting none.
The hardlink that was not there
InfrastructureMoving the media chain onto the NAS was supposed to make imports free. Six separate bugs meant the system could not see a hardlink even when it was looking straight at one.
Forty-three terabytes off Unraid, without reformatting anything
GeneralThe machine I use every day now runs the operating system I sell. The disks were adopted in place, and the rollback was an Unraid USB stick I never had to use.
A real NAS found thirty-one things wrong in one day
InfrastructureEvery storage bug I fixed that day existed because I had only ever run the code against small arrays I built for the purpose.
Booting the ISO on the machine before trusting it with 43 terabytes
InfrastructureBefore converting my main NAS, I booted the release image on it and changed nothing. Three problems came out of an hour of doing nothing.
The installer wiped the right disk and left the wrong one bootable
GeneralInstalling to one disk in a machine with several is not the same as installing to a machine with one disk, and my installer had only ever been tested on the second kind.
Authenticated was not authorized
BuildingAn old audit note in my own docs said all 27 endpoints were gated. Twelve of them checked that you were logged in and never checked who you were.
Forty-eight containers, crown jewels last
InfrastructureMoving everything off an aging Docker host, in five waves, with the password vault and the git server going last on purpose.
Almost everything we published said something untrue
GeneralI read the documentation the way a buyer would, across nine repositories and four websites. The install command returned a 404, the architecture page described a stack we do not run, and the pricing page offered free tiers for a product where nothing is free.
The Arc card my dashboard could not see
BuildingA discrete Intel Arc A310 sitting in a running machine was reported as integrated graphics, because the detection code only ever looked for NVIDIA.
Merged is not shipped
BuildingTwo days getting Eight.ly ready to release. I merged a mountain of work and shipped almost none of it, because three separate times the code was not where I assumed it was.
The changelog that erased itself
BuildingA HomelabARR build that took 2 hours 17 minutes, a workflow that blanked a published page, and an alert that had been set up for a webhook nobody owned. One night of CI and ops fixes, and what each one had quietly been doing.
The demo anyone could lock
GeneralMy public demo hands out admin and admin on its own login screen. The first visitor to change that password would have locked out everyone after them, permanently, and nothing would have reset it.
The token in the install command was root on every agent
AI & AgentsAn enrollment token meant to be pasted into a shell turned out to be a 30-day credential that any agent's API would accept. Fixing it took two changes, and refusing a third took a design argument.
222 images and one address
GeneralEvery product screenshot moved into one repository behind a commit-pinned CDN. Capturing them honestly turned up a placeholder card, a date that read Dec 31 year 1, and a theme toggle that did nothing.
An ISO for hardware I do not own
GeneralEight.ly OS built its first bootable arm64 image, gated it behind a QEMU boot test in CI, and taught the App Store to admit which apps cannot run on the machine you are holding.
A key and nowhere to use it
GeneralThe purchase loop closed on August 8. On August 9 I found out that the thing it sells had no download page for a human to visit, and decided to stop gating the image bytes at all.
The extension outran the server
AI & AgentsMy password server answered every request in milliseconds while every browser extension spun on a skeleton vault. A tcpdump found the bug, and a two-agent team with a one-way radio closed out the night.
The first real order
BuildingA year into building Eight.ly OS, the whole commercial loop ran for the first time: a real checkout minted a key, the key arrived by email, and it activated a live appliance. It took sixteen fixes and two bugs only a real order could expose.
My docs said the box was spare. It was running production.
GeneralA stale fleet table nearly got the production hypervisor wiped. So August 7 became the day of making every document match the machine it describes, and adding guards so they cannot quietly diverge again.
Killing a product name without breaking a single install
GeneralThe entry tier of Eight.ly is now Eight.ly Standard everywhere a person reads it. The old string still lives in every installed box, and keeping it there was the whole job.
Two of the seventeen were half a log stack
InfrastructureFixing the App Store entries that shipped with no container image uncovered something bigger: the one-click monitoring stack had never collected a single log line.
The teardown worked and the disk filled anyway
InfrastructureA catalog sweep filled a 294GB disk while Docker reported 3.8GB of images. The bytes were pinned by 676 containerd leases that docker rmi cannot touch.
A machine I could ruin
InfrastructureTier 2 of the App Store sweep needed an appliance I could destroy. Building it headless surfaced a docker group, a reboot that did not reboot, and a sharding mistake I had to measure to find.
Sonarr was never installable
BuildingThe first trustworthy pass over all 3,246 App Store templates found 135 failures. Only 19 were real, and the worst of them had been shipping as installable for months.
Three bugs only hardware finds
BuildingA fence that powered a recovered node back off, an ISO that could not build at all, and live migration that was impossible between any two member nodes.
The stack Docker is actually running
InfrastructureFive fixes in one day that turned out to be the same bug: code that trusted a declaration on disk instead of asking the system what it was really doing.
Anyone could have minted a license
AI & AgentsThe endpoint that mints Eight.ly license keys had no authentication, and it sat behind a public vhost. Anyone could mint a lifetime OS key. Underneath it, the entitlement model was inverted.
The race detector that went red on its first run
BuildingI wired the Go race detector into CI. It passed five times locally, then went red on its first Linux run on a real production race my Mac could not see.
I ran fourteen agents through every subsystem and found the lies
BuildingA fleet-wide sweep found seventy-four bugs in one session. Half were places the dashboard said success when nothing happened, or success when the user had no permission to try.
Plex off Proxmox, and the backup I destroyed getting there
InfrastructureNinety-three tickets, a catalog that dropped 123 apps, three CI gates that were lying, and the migration that finally let me power off the machine my homelab has run on for years.
Twenty-three bugs from actually using it
BuildingI stopped writing features for Eight.ly OS and spent a day migrating my own estate onto it. The product broke in twenty-three places, and almost none of them were things a test would have caught.
Every VM disk was landing on the boot drive
InfrastructureEight.ly OS could not live-migrate a VM, and every disk it created went to the root filesystem. Fixing the first problem was how I found the second.
I power-killed a node to test failover
InfrastructureI rebuilt a three-node Eight.ly OS cluster to prove the fixes were real, then pulled the power on a running server to test high availability. The failover did not complete, and the reason was worth more than a pass.
I renamed a client's whole business out of its own website
InfrastructureCooper & Cross Logistics became Cooper Logistics LLC. Shipping that meant chasing the old brand and its dead domain out of the sitemap, the docs, the contact email, and the Dockerfile, then adding the real content a working logistics business needs.
I clustered three real servers through the GUI and fixed everything that broke
InfrastructureEight.ly OS v2.3.4 passed in a VM. Then I installed it on a NAS and two Dell T320s the way a customer would, and every bug real hardware and a real cluster hide came out.
My install ISO booted every UEFI box and stranded every BIOS one at a bare grub prompt
BuildingThe Eight.ly OS installer looked finished. Then I put the ISO in a legacy-BIOS Dell and got a bare grub> prompt with no modules. The bug was a lost GRUB prefix in a memdisk core image.
The experimental toggle was hiding four fake features
BuildingEight.ly OS had a Cluster, Ceph, SDN, and HA toggle that returned success and did nothing. I spent a day making them real and sweeping the rest of the OS for the same kind of lie.
The appliance has no SSH, so I drove its web terminal
BuildingEight.ly OS ships as a locked box with no shell. To reinstall it four times in a night and prove a batch of first-run fixes, I wrote a small client that speaks the web terminal's own WebSocket protocol.
The bugs I left for morning
BuildingA defect sweep on Eight.ly OS turned up around eighty issues. I shipped the twenty safe ones to hardware and wrote a note next to the four scariest: too dangerous to rush at 3am. Tonight I went back for them.
Seeing the screen with nobody there
InfrastructureA new tool that captures a Dell server's console with no KVM applet and no human at the keyboard, plus the storage UI finally telling the truth about what is on the box.
Six lines of driver doubled Nova on the Intel box
InfrastructureEight.ly OS runs Nova on whatever GPU the NAS has. The Intel Arc box felt slow, so I built a profiler to get a real number, and a Mesa swap took it from 33.6 to 65.8 tokens per second.
Roles, permissions, and real ZFS at last
BuildingThe biggest single day on the OS this month: a whole permission system that fails closed, a filesystem that hung the build in the morning and shipped by evening, and 247 lies in the type system deleted.
Teaching the OS which GPU it has
GeneralEight.ly's local assistant runs on whatever silicon the box happens to have. Making it pick the right inference backend for an NVIDIA, Intel, or AMD GPU, automatically, was the work of the day.
Making a NAS build its own array
InfrastructureThe whole point of a NAS operating system is turning a pile of disks into storage you can use. That shipped as a real dashboard flow, and the same day it came up running on the first bare-metal box.
The client renamed the company halfway through the build
BuildingA day after the logo, favicons, print files, and business cards were done, the client changed the company name. Here is everything one rename touches, and the quote form that had to reach real invoicing.
Building a logistics company's site so the owner can run it from his phone
BuildingA new client, a jobsite-delivery business in Atlanta, needed a real marketing site. The interesting part was not the design. It was building a content system a non-technical owner can actually publish to.
The day the toolchain became shippable
BuildingNine Jira apps that enforce my own development discipline had been code-complete for 48 days with no README between them. Shipping software is not the moment the code works. It is the moment a stranger can install it.
Fail open, fail honest
InfrastructureAn installer script that printed success while running on the wrong machine, a licensing check that could lock you out of your own NAS, and a wiki claiming features that had not shipped. One day, three ways software lies, and the fixes for all of them.
The installer that painted nothing
GeneralA perfectly booted, Secure Boot verified machine was rendering its setup screen as pure black. Fixing that was the start of turning the Eight.ly OS installer from a boot into a real first-run experience.
The streak died on a day I was working
GeneralMy GitHub graph went gray during one of the busiest weeks of the year. The work was real. It was just landing where the graph could not see it, so I built a small daemon to make sure a contribution always lands.
Watching a real machine install itself over the network
InfrastructureDeploying the Eight.ly OS installer to a netboot server and validating a full PXE install end to end, on hardware I did not control.
The day the installer deleted my host's device nodes
GeneralA full day inside EOS-619, making a self-installing Eight.ly OS ISO actually boot and install over PXE, one buried failure at a time.
The installer meets real hardware
InfrastructureThe gap between a clean install VM and a stranger's ten-year-old box is where all the actual installer work lives.
The day the OS learned to install itself
GeneralEight.ly OS got a real branded installer that boots from its own ISO, and the LevelMinds storefront finished localizing into five languages with screenshot proof on prod.
Working an adversarial audit down to zero in a day
BuildingI turned yesterday's audit findings into twenty individual fixes on Eight.ly OS, then shipped a five-language storefront and started a URL shortener.
Giving LevelMinds a reason to be believed
GeneralA storefront day for LevelMinds: a Stories hub, press credibility, a real science page, a founder story, and a security and accessibility pass underneath.
Rebuilding a client's storefront in a day, and making its copy stop lying
BuildingA client CBD storefront rebuilt on a neutral zinc design in one day, plus the compliance pass that took the unfunded discounts out of the marketing.
I gave my AI a middle manager
GeneralA three-tier delegation chain I'm testing: a product-manager AI that owns outcomes, a project-manager AI peer that drives the work, and sub-agents that do it.
A storefront in a day, and the two times it said done too early
InfrastructureLevel Minds CBD went from an empty repo to a deployed storefront on June 30, along with the CI gates on the backup server that had never actually run.
The login screen did not know which edition it was
InfrastructureMaking one Eight.ly binary look like three products, the first deploy that rendered the wrong one, and a login screen that could not read its own edition.
The mitigations that weren't
GeneralA whole day of security fixes, and the two that mattered most were fixes to earlier security fixes that quietly did nothing. Plus a flaky-test sweep and a supply-chain cleanup across two products.
The doc said one registry. The container said another.
InfrastructureI split my one-page SDLC into a real docs section, shipped it to dev, and it looked done. It wasn't. A doc I wrote and trusted pointed the deploy at the wrong registry, and Watchtower quietly enforced the mistake.
The eight wastes of lean, applied to AI code
InfrastructureTIMWOODS is the standard lean mnemonic for the eight wastes. Here is how I point each one at AI-generated code and the infrastructure that ships it.
I told the reviewers another AI wrote it
BuildingA day after I got eight.ly OS to 80% coverage, I went looking for what the green was hiding. To get an honest review, I had to lie about who wrote the code.
The loose ends that were quietly broken for days
GeneralAfter a week of big pushes, the last day was four small things that had been wrong long enough that I had stopped noticing.
HomelabARR's turn in the gauntlet
BuildingThe open-source homelab app manager got the same security and coverage pass the paid products got. Here is what was hiding in the inherited code.
Making the code prove itself
BuildingChasing a coverage number on the eight.ly OS Go tree, and the six real bugs that fell out when never-run code finally ran.
Container Edition was one default away from root
InfrastructureA security audit of Eight.ly Container Edition turned up two criticals, and closing them in a day meant changing the defaults, not just patching the holes.
Eight runners across two hosts
GeneralGrowing my self-hosted GitHub Actions fleet from six runners to eight, and the unglamorous plumbing that kept it from stranding a release.
The NAS already passed a security audit
InfrastructureTwo days after the NAS OS cleared a full vulnerability audit, I ran a different kind of audit on it. It found an entire layer the first one never looked for.
A bucket is not a backup
GeneralI set up offsite backups for my whole estate, and the backup job started killing the services it was meant to protect. The fix was teaching the tooling to lose.
Prevention is not detection
InfrastructureI pointed a new security gauntlet at my backup server. It blocked every attack I threw at it, and that turned out to be half the work.
My access control was a suggestion
GeneralBefore making Eight.ly OS public I ran a real security audit on it. The headline finding: my role-based access control was wired up but never actually enforced.
The green checkmark was lying
InfrastructureI set out to clean up my Jira board and found three pieces of automation I trusted that were quietly broken. A passing test suite, a wiki that deployed by hand, and a rule erroring in silence.
Nova learned to remember, and I learned what Done actually means
AI & AgentsI shipped per-user memory for the local AI assistant in Eight.ly OS, then a branch cleanup turned into archaeology and found two finished tickets whose work never reached production.
The router picks the tool. The model just talks.
AI & AgentsI got my NAS assistant's router to 99 percent. Then I had to decide what the other models in the catalog were even for. The answer was to put the router in front of all of them and cut the ones that could not keep their mouths shut.
Why I refused to ship an ensemble
AI & AgentsMy NAS assistant's router was stuck at 98 percent. The easy way to 99 was to run three models and let them vote. That is not a product. So I did it the hard way.
The last two percent
AI & AgentsI tried to push my tool-routing model from 97 to 99 percent. It spent the day proving the wall was in my tooling and my benchmark, not the weights.
The token that was never trained
BuildingMy tool router shipped at 90 percent, and today I learned it could not actually say the one token that starts a tool call. Five rounds of data engineering had been bouncing off two frozen rows of an embedding matrix. Fixing them took eleven lines and beat everything I shipped this week by five points.
The control run
AI & AgentsI added 464 training examples to push my tool router from 77 to 90 percent. It collapsed twice, and the road back ran through an 80 cent control experiment, a quantization tax I didn't know I was paying, and two benchmark labels that were grading the model wrong for being right.
Four models, thirteen dollars, one number that finally moved
AI & AgentsThe full accounting of a model-training run: four fine-tunes of Qwen and Gemma, a license trap, a dead GPU budget, a rented H100, two honest failures, and a NAS assistant router that went from 38 to 77 percent.
Upstream took the patch: my Intel Arc port is in OmniVoice now
GeneralIn April I ported Xiaomi's OmniVoice TTS to Intel Arc and opened a pull request. The maintainers reviewed it, I went dark for a product launch, came back, fixed it, and they merged it. The whole messy middle included.
The number I couldn't ship: auditing my own NAS assistant
AI & AgentsI folded a frontier model into Nova, felt good about it, then built a real eval and found the shipped pipeline was scoring 38 percent. Here is how I measured it honestly and pulled it up to 62, with the walls I hit along the way.
I open-published a family of tiny NAS models, then benchmarked them against a frontier 12B
AI & AgentsWhy Eight.ly OS runs a small Qwen tool router plus a Gemma talker instead of one big model, how they ended up on Hugging Face, and what happened when I put the family against Google's Gemma 4 12B on my own hardware.
Engineering an agent control plane
AI & AgentsWhat happens when you can point any AI agent at any machine you own, from one browser tab, two different ways. The actual tools, how it works, the stale-tooling tax, and a from-source GPU build on a remote box that turned a belief into a measured number.
The night I lived in the control plane
AI & AgentsWhat it was actually like to spend a night working through the agent control plane: a model benchmark across three kinds of silicon, a GPU toolchain built from source on a box I wasn't sitting at, and a panel of rival AIs tearing apart my own writeup.
The control plane I didn't mean to build
AI & AgentsI set out to test whether one AI coding agent could drive another on a different machine. It can. The harder question was everything that came after.
The night I cleaned up my own operation
InfrastructureA blog post stuck in staging, a production box I took down with one flag, a pile of CLIs I'd forgotten I built, and an infrastructure doc that no longer matched reality. The cobbler's children needed shoes.
I stopped renting build minutes
BuildingGitHub Actions ran out of minutes and my builds started failing. The fix was four runners I already owned, and four bugs I did not know I had.
I shipped a backup agent, then one log line ate the rest of the night
AI & AgentsI finished the backup server epic and pushed it to prod. Then a deprecation warning in the deploy log turned into 16 repos, a regression, a full disk, and a budget scare.
Your paid plan doesn't follow your repos
GeneralI moved everything into a GitHub organization for succession. The migration looked clean. Then one of my best wikis quietly went dark, because a personal account and an org are two different billing worlds.
Deleting a file does not delete its history
GeneralGetting my products ready to open source meant auditing everything I had ever committed, not just the files in the repo today. git rm was the easy instinct and the wrong one.
One binary, three editions
InfrastructureI turned Eight.ly OS into a second product with one environment variable. Then I reversed myself on the backup server before the day was over.
My documentation was lying to me
InfrastructureMy infrastructure wiki had been deploying to the wrong server for weeks while printing success. Fixing that one bug exposed a bigger problem: every fact I wrote down lived in four places and they disagreed.
Six sprints, 21 dead tickets, and a color I didn't want
BuildingA Friday night spent auditing every Jira board, killing orphaned tickets, syncing diverged branches, and arguing about accent colors.
Nine commits to draw a line
GeneralI spent an entire session fixing four SVG sparkline charts. The lines were too thick, the data was wrong, the labels were invisible, and the Y-axis was upside down.
The purple purge
GeneralI ripped out every trace of purple from Eight.ly OS, rebuilt the parity check from scratch, and installed mechanical guardrails so my AI tooling follows the process.
Guacamole, guardrails, and a VM with 63 megabytes free
InfrastructureI tried to install Apache Guacamole on a VM that was already drowning. Then I fixed the Atlassian CLI I use for tickets.
I audited 28 Jira tickets and 19 of them were already done
BuildingA backlog full of ghost tickets, a CLI that was already built but never shipped, and the decision to open-source it instead of selling it.
Moving day
GeneralMy Mac Mini had 3.5 GB free. I had 45 repositories on the internal drive, no backups, and secrets that only existed locally. Time to fix that.
Nine apps that loaded forever
BuildingI shipped nine Jira Forge apps to production, told myself they worked, and learned the difference between rendering and functioning.
I built a 1,091-endpoint CLI and nine Jira apps in one day
BuildingFive Forge apps shipped before lunch, then a Go binary that turned six Atlassian API specs into one CLI covering every product, then a pile of CLIs installed across four machines.
Where was I working?
GeneralI could not find my own conversations. Two days of work spread across unknown repositories, and I had to search my entire machine to figure out where I left off.
22 security audit rounds on a 5,300-line Express app
BuildingI ran a continuous security audit on HomelabARR CE until the reviewer ran out of findings. 175 issues, a full refactor, and a login rate limiter that locked me out of my own demo.
The SDLC I wrote for an AI agent that kept skipping its own rules
AI & AgentsI codified an entire software development lifecycle into enforcement scripts because writing the rules down was not enough to make the agent follow them.
The day my homelab app got a threat model
AI & AgentsThe day HomelabARR CE's security work stopped being a bug list and became a program: a STRIDE threat model, ten incident playbooks, DAST in CI, and an nginx header that refused to inherit.
Twenty-two rounds of security hardening on a codebase I thought was done
BuildingHomelabARR CE got a proper security audit. Passwordless sudo, hardcoded JWT secrets, XSS vectors, and twenty-two rounds of fixes later, it actually passed.
Six security reviews, one release, zero shortcuts
AI & AgentsHow a Docker-proxy agent went from internal tool to a signed v1.0.0 open-source release in one session, driven by a loop of AI security reviews I couldn't argue with.
I spent a day cleaning dead projects off my hard drive
GeneralDead repos, misnamed folders, stale CLAUDE.md files, and the unglamorous reality of running fourteen products from one workspace.
183 files changed in one migration and I still had to do it twice
AI & AgentsMigrating mjashley.com from Next.js to Vite, matching the ImogenLabs design system, and learning that seven context-window sessions is the actual cost of a full site rebuild.
How I actually build software in 2026
AI & AgentsHuman-led construction, agent-led maintenance. The honest middle between AI hype and AI denial.
My AI agent deleted every container on my production server
AI & AgentsAn AI coding agent ran docker system prune -af on my production box, destroying 44 containers. Rebuilding them exposed DNS routing bugs, CPU overload crashes, and the limits of trusting an autonomous tool with infrastructure.
73 Dependabot alerts, a Next.js 16 upgrade, and scrubbing Portainer out of my codebase
AI & AgentsA security sweep across my repos: Dependabot and Snyk alerts cleared, a Next.js 16 upgrade, a Vaultwarden login scare, and replacing the Portainer-forked agent with a clean-room build.
Resurrecting a dead Docker plugin and auditing 509 LXC apps
InfrastructureI cut a dead Docker volume plugin out of my app catalog in favor of native bind mounts, kept the plugin alive as a standalone tool with real docs, and audited a 509-app LXC catalog for Eight.ly OS compatibility.
Building an OS ISO from a Go binary
BuildingEight.ly OS went from a web app you install on Ubuntu to its own bootable ISO. Plus pool-creation hardening, an external code audit, and splitting five oversized React files into 46.
Rebuilding imogenlabs.ai in twelve rounds
AI & AgentsA full marketing site rebuild from Astro to React 19, hammered into shape over 32 numbered review rounds, plus a production DNS chain reaction and a HealthKit rejection on an iPad Air.
81 compiler warnings and a Swift concurrency migration
BuildingNeuroHelper had 81 compiler warnings, deprecated onChange APIs, unused variables, unnecessary try/catch blocks, and Swift 6 concurrency violations. I fixed all of them in one afternoon.
The polish pass that took 36 messages and touched every page
AI & AgentsComparison table icons, lightbox dialogs, CSS cascade fixes, component extraction, Stripe checkout URLs, WCAG 2.2 AA accessibility, and the global border-color bug. A full-day polish round on imogenlabs.ai.
26 product pages in a day, built by 14 parallel agents
AI & AgentsOne day, a stack of tightly-scoped agent rounds, and imogenlabs.ai went from rough to a real marketing site: WebP images, aria-labelledby on every section, per-page JSON-LD, a three-column footer, and a homepage reordered by what a buyer reads first.
I audited my own wiki and found 19 pages that weren't enough
AI & AgentsA cross-repo consistency audit, 8 new wiki pages, a competitor comparison table, a full site rebuild from Astro to React, and a round of dependency vulnerability fixes. Documentation day at Imogen Labs.
Five evaluator passes and a SQLite migration before lunch
InfrastructureI pointed Claude Opus 4.7 at my NAS operating system and let it tear the product apart. Five review passes, ten bug fixes, an icon library consolidation, and a full persistence migration later, I had something I could actually sell.
I spent $25 on rented GPUs and trained three AI models for my NAS
AI & AgentsTraining custom AI models for Eight.ly OS on rented GPUs. Colab failed on a library regression, Unsloth had to go, Vast.ai worked, and the GGUF toolchain ate most of the day. The models shipped.
The ISO boots and twenty features ship
BuildingEBS integration, cache mover, pool hardening, GPU passthrough, a calendar page, and a bootable Linux ISO. All in one day.
Two products in one day
InfrastructureFunctionGemma ships as Nova's tool router while Eight.ly Backup Server gets a real backup engine, WebSocket shell, and 260 tests.
The FunctionGemma rabbit hole
AI & AgentsWiring a 270M function-calling model into Nova meant bypassing Ollama's broken renderer with a raw prompt, and fixing a training collator that never trained on the tool call.
4,684 training examples and the day I taught an AI to run a NAS
InfrastructureUI fixes, a tool-calling training dataset, and the SQLite migration that became Eight.ly OS's persistence layer.
Nova was born in a USB stick repo
AI & AgentsA long single-day session that started as a refactor of a forked USB LLM project and ended with Nova, the AI assistant, built into Eight.ly OS.
Auditing the product I built
GeneralI pointed parallel AI agents at my own product and asked them to tell me what actually works. The answer was uncomfortable.
Eight.ly Professional had no auth. Now it does.
BuildingSettings > Dashboard Users showed three accounts. None of them were real. Here is what it took to replace the theater with a real auth layer.
Twenty Attempts at a Gap
InfrastructureA 7-pixel layout gap in the App Store page took twenty deploys to close because I kept solving the wrong problem.
The CPU widget was stuck at zero for months
GeneralThe formula was load average over cores times 100. That is not CPU usage, and the chart was flat for months because of it.
Building a license-gated catalog for a paid NAS product
InfrastructureHow I protect 3,368 curated Docker templates behind a license check in Eight.ly Professional without breaking offline use, and what shipped versus what didn't.
I split five monolith files and built back cleaner
BuildingFive Jira tickets sat open for months. 6,463 lines of Go in five bloated files. One PR, twenty-eight focused modules, nothing over 636 lines.
Five Watchtowers, three ways to break
InfrastructureWatchtower was running on five VMs. Three were crash-looping, one had a mount pointing at a directory, and one was scanning zero containers while reporting success.
The templates repo never belonged to CE
InfrastructureHomelabARR CE's install instructions pointed at a private templates repo it never actually needed, and decoupling from that repo is what finally made the Eight.ly fork real.
users.json is not an array
GeneralLogin failed with TypeError: users.find is not a function. The file was {} instead of [], and the seeder that should have saved me silently skipped because users.length === 0 is false when users is an object.
OmniVoice just said hello from my Intel Arc
AI & AgentsI ported OmniVoice, a k2-fsa speech model, to run on an Intel Arc GPU. Six files changed, and an honest account of what I did and did not touch.
IPAC: Integrated Personality as Code
AI & AgentsI spent 8 months encoding myself into markdown files before I realized I was doing Infrastructure as Code for people. Here is what Integrated Personality as Code means.
185 commits, a wiki nuke, and one real customer
BuildingA week of HomelabARR CE shipping v2.0.0, nuking a 207-file dead wiki, and fixing a broken download link for my first paying customer.
How I built a 4-agent AI company in one day with Paperclip and OpenClaw
AI & AgentsSetting up Paperclip on top of my OpenClaw gateway to run four autonomous Claude agents as a company, including the device-auth problems that ate most of the day.
Session = sprint: why your AI agents need a Scrum master
AI & AgentsA project manager’s framework for running AI agent teams using Scrum principles. Every agent session is a sprint. Here’s why that changes everything.
The archaeology of a fork
InfrastructureHomelabARR CE is built on a codebase that runs back through Dockserver, PGBlitz, and PlexGuide. Each era left something behind, and removing it is its own kind of work.
I ripped out a Docker plugin nobody needed
Infrastructure77 Docker Compose templates. Every one had a third-party volume plugin dependency. I replaced them all with native Docker bind mounts in one afternoon.
I spent 16 hours in a terminal and rebuilt my entire company's infrastructure
InfrastructureIt started at 7 PM. I asked my AI coding partner to look at my setup. By 11 AM I had rebuilt everything. I planned an accordion.
All because I wanted an accordion
InfrastructureA scroll fix on my portfolio site cascaded into one sixteen-hour session that rebuilt three sites, stood up dev/staging/prod, and fixed five stacked bugs in one Docker image.
What 12 Scrum teams taught me about building alone
BuildingSprint planning, story points, and retrospectives work solo. The skills that got me hired are the same ones making me dangerous building on my own.
TrueNAS closed source and what it means for home NAS
InfrastructureiXsystems closed their build system and announced monetization for the community edition, and I lay out why Eight.ly's MergerFS plus SnapRAID stack is not exposed to that risk.
My AI assistant destroyed my website in 30 seconds
AI & AgentsOne vague task dispatch. No staging environment. Six hours of rebuilding. Everything I learned about production engineering with AI agents.
Why self-hosted AI agents are the future of automation
AI & AgentsForget cloud-locked AI. The real revolution in automation is happening with self-hosted AI agents. Here
Why I'm launching managed AI agent hosting
AI & AgentsHetzner bare metal, 90% margins, and an AI roundtable that told me to do it. The math behind managed OpenClaw hosting.
How I run a software company with an AI partner
AI & AgentsI built a 13-agent auto-router before most people were thinking about multi-agent orchestration. Then I built a company around it. Here is how AI-augmented development actually works.
The server rack that runs my entire company
InfrastructureA Mac Mini, a 96GB mini server, a 56TB NAS, two Dell servers, and an AI operator. The entire infrastructure behind every product I ship, every agent I run, and every late-night experiment.
Why I stopped using cloud AI and started running agents on my own hardware
AI & AgentsHow I run nine AI agents on a Mac Mini in my parents' house to monitor servers, write briefings, and audit configs for about thirty dollars a month.
The quiet weight of systems at scale
InfrastructureWhen Traefik goes down, 20 services follow. What running 34+ containers taught me about dependency awareness and infrastructure that fails gracefully.
What's in a name?
AI & AgentsPeople ask me why I named my AI assistant Imogen. The real story involves Shakespeare, Julia Stiles, a Plex server, and a British musician who made songs with homemade gloves.
The real timeline
BuildingI built like a maniac for six weeks. Then I stopped for five months. Then I came back and shipped everything in three weeks.
The gap wasn't nothing
GeneralFive months of a gray contribution graph that looked like I quit, and what the gap actually was: hardware, healing, and a DGX Spark I camped out for and returned.
AI agents need memory infrastructure
AI & AgentsMost AI agent memory is theater. Here is how we built real memory infrastructure: instincts, daily logs, and context engines that survive resets.
Why I built HomelabARR
BuildingFrom PGBlitz drama to a real software company. The complete story of how a project manager learned to code by rescuing a community project.
I got laid off and started building with AI agents
BuildingAfter nearly a decade in product management, I got laid off and started building HomelabARR with AI agents while I job hunt. Six months in, sales are trickling and the job search is still running.
I built an AI assistant that actually works (not a chatbot)
AI & AgentsNot a chatbot. An operator. Running in the background, handling operational work, surfacing only when it needs human judgment.
The NAS operating system that does everything Unraid does
InfrastructureUnraid costs $49 to $249 and bundles storage, Docker, and VMs behind a custom GUI. Eight.ly OS does the same four jobs on stock Linux, and I can point at the code for each one.
HomelabARR vs Unraid vs TrueNAS: which NAS platform should you pick?
InfrastructureChoosing a NAS platform is hard. An honest comparison of HomelabARR against Unraid and TrueNAS — where each wins, and where mine still has gaps.
My first dollar from something I built
General$36 from a stranger who bought something I built, no boss and no org chart in the way. The first dollar that was mine.
How to build a complete media server with Docker in 30 minutes
InfrastructurePlex, Sonarr, Radarr and the rest of the stack, deployed one container at a time from real HomelabARR templates, with the per-mode compose rewrite shown in full.
What do you build when you have nothing?
BuildingI could never answer the party question about what I do. The honest version: I ship real software, with AI doing the typing and me doing the judgment.
238 pages for a team of one
BuildingSprint retrospectives with two attendees: me and my AI. RCAs for bugs nobody else would ever hit. A master index for a Confluence space only I will ever read.
The 3 AM Linux deploy
InfrastructureI started at 3 AM. By 7:30 AM the binary was running on a real Linux VM. Cross-compilation failed, CORS was wrong, paths were hardcoded. But it worked.
6,345 lines in one file: the refactor that saved the project
InfrastructureMy main server file was 6,345 lines. 212% past the emergency refactor threshold. I broke it into 8 modules across two days. 56 story points. Zero breaking changes.
Why I built HomelabARR PE around MergerFS and SnapRAID
InfrastructureWhy I chose MergerFS and SnapRAID over ZFS, Btrfs, and bcachefs for the storage engine, and the real Go code that runs it.
The accidental audit that changed everything
BuildingI ran a project audit expecting bad news. Instead I discovered 20+ story points of completed work I forgot I had built. The Jira board was lying.
We migrated the entire frontend to React in one sprint
BuildingA full Bootstrap-to-React 19 rewrite of the HomelabARR frontend in one sprint, and the Tailwind plus shadcn/ui PostCSS gotcha that ate half a day.
I read Perfect Media Server and built the software I wished existed
InfrastructureAlex Kretzschmar documents the right way to do home NAS storage. Nobody had built a GUI for it. So I did.
I killed Windows support and saved 11 days
InfrastructureDropping Windows from a NAS operating system meant deleting code that worked. It returned a fake C: drive when no real storage was found, on a product that only ever ran on Linux.
How I made the app store 200-300x faster
InfrastructureMy own App Store took 20 to 30 seconds to load. I was calling docker ps once per template, 128 times per page, plus reading 128 YAML files off disk every request.
I rewrote the whole thing in Go over a weekend
BuildingThe JavaScript backend worked. It also couldn't ship as one binary. I threw it away and rebuilt it in Go, a language I had never used, over three days.
The day Claude deleted my project file
BuildingThree weeks into building HomelabARR, my AI coding partner ran a destructive git command without a dry run and wiped my project config. Here is what I lost, how I got it back, and the guardrails I put in afterward.
The container migration nobody saw
InfrastructureHomelabARR inherited its container names and build pipeline from the fork it grew out of. Cleaning that out was a day of work no user will ever notice, and that is exactly the point.
Everything broke on day four
InfrastructureCORS misconfigured. Docker incompatible. 65 YAML files corrupted. The system that worked at 11 AM was completely dead by 3 PM.
The first container that actually worked
InfrastructureTwo weeks after getting fired, I deployed a real Docker container from software I wrote. it-tools, port 8082. That was the moment.
Laid off. Time to build.
BuildingGetting laid off from a Senior TPM job in July 2025, moving back in with my parents, and deciding to run the job hunt and build HomelabARR in parallel.
Back at it
BuildingFive months. That is how long the repo sat untouched. I opened the project, read my own code like it was written by someone else, and started tearing apart the auth system.
The first sprint
BuildingI stayed up until 3:28 AM on a Thursday. Not for work. For a side project that did not exist eight days ago.
Eleven commits on a Monday afternoon
InfrastructureI had been running 34 containers on my homelab for over a year. I had a system, but the system was me.