Category Archives: Gadgets / Toys

The rack display running in the homelab rack

Homelab Rack Display: Giving a Prototype Panel a Second Life

So a couple months back there was an office clean-up at work, and sitting in a pile of stuff on its way to the trash was a long, skinny little USB display mounted in a 1RU chassis as a prototype front panel. Eventually the product went with a much simpler monochrome OLED, probably for very good reasons, but at least this one still worked once plugged into a PC.

In this blog post I’m going to walk through getting that panel lit up in my homelab rack, showing infographic-style status screens for the cluster, the internet connection, Jellyfin, the UPS and a handful of other things, and also how ridiculously fast it came together with Claude doing basically all of the heavy lifting. Like my last salvage project, this one started with a piece of hardware that was supposed to be garbage, and now it’s something I look at every time I walk past the rack.

For those impatient like me, the photo at the top of this post is how it turned out, sitting in the rack between my TP-Link switch and the two Turing Pi boards that make up most of my cluster.

A photo only shows one screen though, so here’s the full rotation captured from the live display:

Animated rotation of the rack display screens
The full rotation, sped up a bit (each screen normally sits for 8-15 seconds)

So What Is This Thing?

The panel is a 1440×240 pixel USB display. That’s a 6:1 aspect ratio, which is about as wide and short as displays get, and since it was already built into a 1RU chassis it dropped right into an open slot in the rack. It shows up on USB as a “UV01DA USB Display” with a DisplayLink chip in it, which sounded like great news at first since DisplayLink has Linux drivers (more on how that went in a minute).

The plan was to plug it into one of the nodes in my Kubernetes cluster and have something in the cluster drive it. My cluster is largely the two Turing Pi 2 boards you can see on the shelf in the photo above (Turing01 on the right, Turing02 on the left), each with four RK1 compute modules under those heatsinks. The Turing Pi’s USB 3 ports are wired to node 4 on the board, so the panel’s USB cable goes into Turing01, which means it physically belongs to a node called turing01-04 and whatever drives it has to run there.

“Look at What I Have in Grafana and Make Me Something Cool”

In one way it’s wild how little prompting is needed to make something cool, and in another way it’s even more wild to consider how much further we can go with stuff like this (and life in general) with such accessible tooling at our disposal. I already had a pretty good monitoring setup going in the cluster, with Prometheus scraping just about everything in the house and Grafana sitting on top of it with dashboards and about 40 alert rules, all managed in Terraform, but I didn’t have any way to see any of it without opening a browser and going looking (or getting a notification in Discord from a triggered alert).

So on a Sunday evening I opened up Claude Code in an empty folder and typed out what was basically “look at what I have in Grafana and make me some cool rack display”. In all truth it was a little longer than that, but not by much. Here’s the distilled down version of that prompt:

I’d like to get a designer sub-agent to consider the dashboards and alerts in Grafana, and design some useful but ‘infographic’ / stylish screens that we can iterate through to give us some info like network/internet bandwidth, key application health and stats like Jellyfin. The kicker here is that the screen display we have is 1440×240 so it’s very wide aspect ratio […] Ideally we can build a framework that will allow us to add configuration as this evolves, sort of like terraform but could probably be a JSON doc.

I also mentioned I was thinking Go and maybe HTMX, and that I hadn’t actually seen a single pixel on the display since I plugged it in (that last bit turned out to be important). Here’s roughly how the timeline played out, straight from the session history:

  • 8:30 PM: the prompt above
  • 8:50 PM: a test pattern and a moving progress bar on the panel for the first time ever (after a detour through kernel drivers, below)
  • 9:15 PM: a sub-agent had read my Grafana dashboards and alert rules and come back with 7 designed screens, rendered to PNG at exactly 1440×240, after three rounds of critiquing and fixing its own renders
  • Tuesday 6:30 AM: the whole thing was running in the cluster, deployed by ArgoCD, with live data on the panel and an internal web page at rackdisplay.int.knightware.net

The designs were good enough on the first try that I mostly just said “these look amazing” and kept going, and I didn’t pick the colors, the fonts or the layouts. Claude even made all the colors ‘565-exact’ (every value lands perfectly in the 16-bit color space the DisplayLink chip actually uses) so there’s no banding on the real panel, which I wouldn’t have thought to ask for. The one real design change I made was pointing out that I already had a speed test service running in the cluster, so we didn’t need to build our own.

Full disclosure here, this wasn’t a magic trick and I did have to do things. I ran commands on the node, I created the GitHub and Docker Hub repos, I applied the Terraform for a Grafana service account, and I told it when something looked wrong. But the ratio of my effort to the result on this one is unlike anything I’ve built before, and honestly it looks better than what I would have come up with on my own.

Getting Pixels on the Screen

Before any of the pretty stuff mattered we had to get the panel to show anything at all. I had already installed DisplayLink’s official Linux driver package (DisplayLinkManager plus the evdi kernel module) on the node, and the display just sat there dark.

It turns out the UV01DA is built on an older DisplayLink DL-1×5 chip, and DisplayLink’s modern driver only supports their newer DL-3xxx and up, so its udev rule never even matched the device and the service never started. The older chips are handled by a driver called udl that’s been in the mainline Linux kernel for years… except the Ubuntu Rockchip kernel on the RK1 is built with CONFIG_DRM_UDL turned off, so it just wasn’t there.

The fix was to build udl out of tree, pulling just the driver’s source files from kernel.org for the exact kernel version the node runs (6.1.75) and compiling them against the installed kernel headers. The first build failed on a missing header file, and once we pulled that down too it built and loaded fine. Then I ran a little Go test program that Claude had written to draw straight to the display through DRM/KMS (the Linux kernel’s display API, so no X or Wayland desktop needed), and up came a test pattern with a moving bar along the bottom, sized perfectly on the first try.

There was one more quirk. udl insists on a minimum framebuffer size of 640×480, and since this panel is only 240 pixels tall a 1440×240 buffer gets rejected. The workaround (read: hack) is to allocate a 1440×480 buffer, show only the top 240 rows, and tell the driver to flush just that region. Later I set the module up with DKMS so it rebuilds itself on kernel updates and loads on boot, and the steps for all of this are in the repo’s panel driver doc for anyone else with one of these panels.

How it Works

With pixels on the screen, the rest is a Go app running as a single pod in the cluster, and the image below is the whole picture:

Rack display architecture diagram
How the rack display works, from Prometheus to the panel

All the data comes out of the Prometheus instance I already had, with node-exporter and kube-state-metrics for the cluster, OPNsense, blackbox and a speed test exporter for the internet connection, SNMP for the APC UPS, Synology NAS and TP-Link switch, plus exporters for Technitium DNS, Jellyfin and my LiteLLM proxy. Alert state comes from the Grafana API using a read-only service account.

Every 15 seconds the app runs every query in a single JSON config file and builds one data document out of the results. This is the ‘sort of like Terraform’ piece I asked for. Each value on a screen is a named binding in config/display.json along with its warning/critical thresholds, and the rotation order lives in there too, so adding a stat is a config change rather than a code change. The screens themselves are just HTML/CSS/SVG pages, and the server injects the latest data document into the page where a small script binds it to the elements on screen.

The panel part is the bit I like the most. Rather than writing a separate renderer, the pod runs a headless Chromium browser pointed at the exact same rotating web page a person would see, takes a screenshot once a second, and pushes any changed frames straight to the panel through DRM/KMS and udl. So the panel and the web view are literally the same page, and they always match.

For Jellyfin, the exporter publishes now-playing sessions, transcodes and library counts to Prometheus, and the media screen just binds to those. The first time I started a movie (Hackers, obviously) and saw it pop up on the display, I may have giggled a little.

Only one process can own the panel at a time, so it’s a single privileged pod pinned to turing01-04 with access to the node’s display devices, and it gets fully stopped before a new one starts during a deploy. If the panel isn’t plugged in, the app just keeps serving the web view and checks back every 30 seconds.

The Chromium thing

The one real bug so far was memory. Headless Chromium taking a screenshot every second while swapping iframes around grows by roughly 150 MB a day and never gives any of it back, and about three and a half days in it ran the pod into its 1 GiB memory limit and got killed. The ‘fix’ was to restart the browser every 6 hours (with tini in the container to clean up the leftover Chromium processes), which isn’t elegant but keeps the memory in check and works great. Here’s the Grafana graph from before the fix went in:

Rack display memory usage climbing to the 1 GiB limit
A nice straight line up to the 1 GiB limit, an OOM kill, and then right back at it

The Screens

There are currently 9 screens in the rotation (11 counting alternate versions), each with a rail on the left showing the clock, overall alert status and where you are in the rotation. Here they are, captured live:

Overview screen
The overview: one tile per area, each with a status dot
Internet screen
Internet: live WAN bandwidth, the last 6 hours, the latest speed test and ping to a few DNS providers
Cluster screen
Cluster: CPU, memory and SoC temperature for all 9 nodes, plus any crash-looping pods
Media screen while streaming
Jellyfin while something’s playing (this one is the design mock with made-up data)
Media screen while idle
When nothing is playing, it swaps to an idle version with library stats
Power and storage screen
Power and storage: UPS runtime and load, NAS volume and disk temps, and the last successful database backups
DNS and network screen
DNS and LAN: Technitium query and block stats, and traffic on the TP-Link switch sitting right above the panel
LLM usage screen
LLM usage through my LiteLLM proxy over the last 7 days, local models vs. Claude (there’s also a 24 hour version)

The LLM screens were added a week later, and adding them was really easy. There’s a little ‘new screen’ skill checked into the repo that walks Claude through finding the metrics, designing the screen, wiring up the bindings and checking it against live Prometheus, so a new screen is now pretty much a one-sentence request.

The Web View (and why it’s not embedded here)

Since the panel is just a browser screenshotting a web page, the same page is available at rackdisplay.int.knightware.net on my home network. It scales to fit whatever window you open it in, there’s a page per screen, and there’s even a /frame.png that shows exactly what the panel is showing at that moment. It’s handy for checking on things from the couch, and I could see it ending up as a widget in a bigger dashboard someday.

I did think about embedding it live right here in this post, but while this blog runs on the same cluster, the rack display is only exposed internally, and I don’t really want my UPS runtime and backup job names on the public internet (or a stranger’s browser hitting my Prometheus every few seconds). So you get the GIF above, which is honestly 95% of the experience.

The Alerts Screen Shamed Me Into Fixing Things

If nothing is firing the alerts screen is pretty boring, which is exactly what you want:

Alerts screen, all clear
All clear, 41 rules watching

It didn’t start that way. When the display first went live it showed one critical and three warnings, and the critical was a “Monitoring target down” alert for an exporter that had been firing for 21 days. Twenty one days! The alerts had been going to Discord the whole time and I’d just tuned them out. The rotation is also set up so that while anything critical is firing, the alerts screen shows up after every other screen, so half the time I looked at the rack it was flashing red at me. It looked something like this:

Alerts screen with alerts firing
What it looks like when things are on fire (design mock, but the real thing looked just as alarming)

Turns out a big red box in the rack is a lot harder for me to ignore than a Discord channel. So I went and fixed (or had Claude and OpenClaw fix) everything that was firing, and it’s stayed green since. I honestly didn’t expect a status display to get me to clean up my alerts, but it did, and that might be the most useful thing about it.

So where are we, and what’s next?

The display has been running in the rack for about a week now and it survives reboots, and releases go out as pinned, tagged versions through ArgoCD, so nothing changes in the cluster unless I bump a version on purpose. It’s also really easy to extend, so I expect the rotation to keep growing. Next on the list is probably per-application health checks for the apps that don’t have their own exporters yet (Immich, the *arr apps, Home Assistant), and maybe that dashboard widget idea for the web view.

If you happen to have one of these DisplayLink bar panels (or any odd-sized display) and want to do something similar, all of the code, screen designs and driver notes are up on GitHub. If you run into any snags or have ideas, the easiest way to reach me is to file an issue on the repo, since I have a bad habit of not checking blog comments for months at a time 🙂

GitHub Project Link: https://github.com/dsmithson/homelab-rack-display

Thanks for reading, and until next time!

The finished panel, showing a green build

Building an Azure DevOps Build Pipeline Monitor from Salvage

How this one got started

Every so often a project starts with a piece of hardware that somebody else has already given up on. This one started in a pile headed for the trash.

During an office clean-out of old stuff, I found and held onto the front panel of an old Phoenix Quad-T, a media encoder that has been end-of-support for a while now. The device was missing its main processing card, but it still had its front-panel in-tact — a half-rack wide and one rack unit tall textured plate with curved corners, a little window for a 128×32 OLED, a strip of holes for indicator LEDs, and a frosted hexagon dead center.

For a while it just sat on the bench while I tried to decide what it should be. The OLED and the LED board were still intact, which meant whatever I built could show both text and color. The answer that I kept coming back around to was a build monitor. At work we’ve got a handful of Azure DevOps pipelines that I care about, and “is anything red right now?” is a question I answer by opening a browser tab and looking at a dashboard. A physical thing on a shelf that just tells me seemed like a better answer.

A couple of decisions fell out of that pretty quickly:

  • The center hexagon was too good to waste on a single LED, so I added a 7-pixel NeoPixel Jewel behind it. Seven pixels, seven pipelines. To diffuse it I cut two layers out of a gallon milk jug and stacked them behind the hexagon window — I fully intended to replace that with something proper later, and of course I never did, because it works perfectly.
  • It had to be wireless. A device that needs an Ethernet run is a device that lives wherever the Ethernet run is, and I wanted this thing to be able to sit anywhere.
  • I have a bunch of small embedded boards lying around, and landed on using a Raspberry Pi Pico W running MicroPython. It’s cheap, tiny, has WiFi built in, and is quick to iterate on.
Early bring-up on the bench
Early bring-up on the dining room table

That’s the first time it really looked like something. I had it cycling through some colors for testing ahead of wiring up the real application, but everything worked and looked pretty sweet (albeit crazy bright which I ended up dialing down in software later).

Building the hardware

Here’s the whole thing on paper:

Wiring diagram
Wiring diagram

Parts in the panel

  • Raspberry Pi Pico W — the brains, and about $6.
  • Adafruit SSD1306 128×32 SPI OLED (#661) — this is the stock panel used in the panel’s window. It now runs off the Pico’s 3.3V output.
  • Adafruit NeoPixel Stick — 8 WS2812 pixels, though only 4 of them line up with holes in the bezel. The other 4 are wired, powered, and permanently invisible (more on that later).
  • An Adafruit NeoPixel Jewel — 7 pixels, and exactly the right size to fill the center hexagon.
  • Two silicon diodes — cheap, and the entire reason this works. More below.
  • A 220µF (or larger) electrolytic cap — across 5V/GND at the first pixel, to soak up power-on inrush.
  • A 5V / 2A external supply — the Pico W pulls 100-150mA with WiFi spikes on top, and 11 NeoPixels at full white can hit ~660mA. 2A leaves comfortable headroom.

The OLED is straightforward SPI — clock, data, chip select, D/C, reset, plus 3.3V and ground straight off the Pico. The NeoPixels are a single data line off GPIO15, daisy chained from the panel’s original board into the Jewel.

Power is where it gets slightly more interesting. Rather than run the Pico off USB and the LEDs off a separate supply, I run one 5V supply and “T” it: one leg to the LED chain, one leg to the Pico’s VSYS pin. That’s pin 39, and it’s the input the onboard regulator is designed to accept external power on. Not VBUS (pin 40) — that one’s tied to the USB rail, and I learned that feeding it can damage the Pi board. Splitting a single supply this way also hands you a shared ground for free, which the NeoPixels very much require. And because there’s a diode between VBUS and VSYS on the Pico, you can leave USB plugged in at the same time while you’re developing without the two supplies fighting which ended up being massively helpful while tweaking the software on this thing.

The diode trick

Here’s the one that I spent a lot of time on and almost gave up over. WS2812 pixels want a logic-high somewhere north of 0.7 × VDD. The Pico’s GPIO is 3.3V. On a true 5V rail, that’s under spec, and the symptom is maddening: the pixels get data — I could see it on my oscilloscope — but they mostly ignore it. Sometimes the first pixel lights, usually nothing, and sometimes it worked fine for a few minutes and then stopped.

The fix that finally worked was two silicon diodes in series in the LED’s +5V feed — not in the data line — dropping the pixel rail to roughly 4.2-4.4V and giving that 3.3V signal actual margin. I tried one diode first, and it almost worked, which is the worst outcome. Forward voltage sags at low current, and this chain spends most of its life dim or idle, so a single ~0.6-0.7V drop kept wandering back out of spec. Two diodes, rock solid ever since. This is apparently common knowledge for smart people, and I saw a lot of posts online about it (after circling in on it by chatting with ChatGPT).

Worth checking your supply too, incidentally — some USB bricks like the one I’m using measure 5.4-5.5V under light load, which makes the margin problem worse than the initial math would suggest.

About that soldering

The board on the bench, mid-assembly
The board on the bench, mid-assembly

The pictures above and below show the wired-up mess at the back of the panel, and admittedly it’s messy. Every joint on that Pico is a small monument to enthusiasm exceeding technique — blobby, over-tinned, and held in position by optimism and hot glue. In my defense: the panel’s backing plate is the mounting surface, everything has to sit flat enough to still fit behind a curved bezel, and there was not a lot of room in there to be elegant.

The back of the assembled panel
The back of the assembled panel

It doesn’t look like much, but it hasn’t dropped a connection yet, so I’ve decided it’s a feature.

Breadboard testing with a spare display
Breadboard testing with a spare (unrelated) display

I did most of the ugly experimentation using a breadboard — which is where I dialed in that diode business — before committing anything to the panel itself. This picture above was the last part where I was figuring out the diode thing.

3D printing a chassis

The panel is a curved face with nothing behind it, so it needed a body. I designed one in Tinkercad and printed it on my Prusa Core One, which I will take any excuse to mention because I love that printer. Big enclosed build volume, quiet, and it just prints — I’ve stopped thinking about first layers entirely, which is the highest compliment I can pay a 3D printer.

The case design in Tinkercad
The case design in Tinkercad

The design is deliberately dumb: an open-backed tray that the panel drops onto face-out, with all the electronics hanging down inside it. The panel is held on with velcro rather than screws, which means getting back to the Pico’s USB port is a matter of peeling the face off.

It took three revisions to get right, which I think is about par:

Three printed revisions with notes
Three printed revisions with notes
  • Revision 1 was the naive version — a tray, a hole in the bottom for the power connector, and two corner tabs to hold velcro. It worked, and immediately told me what was wrong with it.
  • Revision 2 added “diamond” vent holes along the walls so the Pico and the LED chain aren’t sealed in a black plastic box (I noticed it getting warm with the first rev), mounting brackets on the back wall so it can hang, and a lot more flat surface area for the velcro to actually grab.
  • Revision 3 was one specific annoyance: the back-wall brackets only worked with the power connector oriented one way. Reworked them so it hangs correctly with the cable exiting either direction.

The STL is in the repo under case-stl/ if you happen to have a Quad-T front panel lying around, which, statistically, you do not.

Building the software

This is the part I actually enjoyed, and the part where the design decisions mattered.

The firmware should be as dumb as possible

The single most important constraint came from the case: updating this thing is a pain. To reflash it I have to peel the panel off the case, find the USB cable, plug into the Pico, push files with mpremote, and put it all back together. That’s a five minute round trip for even a simple change.

In addition to the physical update process, the reality is that this is a simple IoT device that I might repurpose for anything, and instead of writing tedious code specific for one use case on the target, making it a wireless display and allowing external code to control it feels like the more natural fit. In my case I have a homelab Kubernetes cluster providing more than enough infrastructure to provide messaging and application hosting for the Azure build monitor today and whatever else I might want tomorrow.

So the firmware knows nothing about builds, and it doesn’t know what Azure DevOps is. It stores no credentials beyond the WiFi password, makes no outbound calls to anything but the local MQTT broker, and listens on no inbound ports. It is a generic display device: it connects to WiFi, subscribes to one MQTT topic, and renders whatever JSON shows up there. All the interesting logic lives somewhere I can redeploy in thirty seconds.

That MQTT message contract is the whole design, and it looks roughly like this:

{
  "display": {
    "drawMode": "text2Line",
    "textLine1": "EchoCommon",
    "textLine2": "Success: 2 Days ago",
    "textLine1Align": "left",
    "textLine2AutoScroll": true,
    "oledBurnInProtectionMode": "invertDisplay",
    "oledBurnInProtectionInterval": 60
  },
  "leds": [
    { "r": 0, "g": 255, "b": 0 },
    { "r": 255, "g": 0, "b": 0, "displayMode": "blinkThruBlack", "blinkDuration": 0.75 }
  ],
  "transitionDuration": 2.5,
  "transitionType": "smooth",
  "pixelBrightness": 1.0
}

The goal was high-level but rich — one message should be able to express an entire visual state, and the device should own everything about how that state gets rendered over time. Some of what it can do:

  • Text, one or two lines, each independently left-aligned or centered, and each able to ping-pong scroll if it’s wider than the screen (with a dwell at each end, because scrolling that instantly reverses looks nervous).
  • Raw pixels — you can also just hand it 512 bytes of hex and paint the 128×32 framebuffer yourself.
  • Per-pixel color as a simple array. Position 0 is the first externally visible LED.
  • Transitions — immediate, smooth (blend from current to target), or thruBlack (fade down, then up into the new color), with a duration.
  • Per-pixel blinking — blink for hard on/off, or blinkThruBlack for a gentle pulse, which is what “this pipeline is running right now” uses.
  • Global brightness, as a single multiplier applied at the moment of writing to hardware.
  • OLED burn-in mitigation, because this display is on 24/7 and may showing nearly the same thing which can get burned into the display permanently. The protocol supports configuration to automatically can invert the panel colors, nudge the text around DVD-logo style, or do nothing.

A few details are worth calling out that evolved organically while I was working on this project. First, the burn-in inversion is motion aware — it only flips once the screen has held genuinely static content for a while, and any new content immediately un-inverts. Since the build monitor rotates through builds anyway, this just looked annoying and served no purpose. The one caveat is that if for some reason the application driving display messages stops updating the panel, the panel will protect itself automatically by using the invertion technique. Second call out is for the hidden LEDs: the panel’s board has 8 pixels but only 4 are visible, so the firmware keeps a map from logical position to physical position and never exposes the hidden ones over MQTT at all. Everything upstream sees a clean chain of 11 addressable pixels and has no idea there’s anything else on the wire. Had I thought about this before hot glueing everything together I would have popped the NeoPixel strip out and drilled 4 extra holes, but it’s simply not worth it to me anymore (but since it’s software configurable if I make another one down the road it’s a simple code change to re-enable it).

The device also publishes its last applied state back to a retained state topic, plus an online/offline status topic wired up as an MQTT Last Will — so anything that cares can tell at a glance whether the panel is actually alive.

Getting real build data to it

How a build result reaches the panel
How a build result reaches the panel

With the panel reduced to “renders JSON from a topic”, the rest is a plumbing problem:

  1. Azure DevOps fires a run.statechanged service hook whenever a build changes state.
  2. n8n catches that webhook and republishes it onto my MQTT broker. n8n is already running in my homelab and is very good at exactly this sort of glue, so it saved me from writing and deploying my own webhook receiver just to turn one POST into one MQTT message.
  3. A small Go service (azureBuildMonitor, containerized and running in my k8s cluster) subscribes to that topic, keeps track of the world, and publishes display commands to the panel.

The Go service is where all the real behavior lives, and it works on two timescales at once. It does a full refresh every hour — one API call that fetches the latest 4 builds for every pipeline and overwrites its in-memory view with the authoritative answer. That’s the drift-correction backstop. Then, for responsiveness, every webhook event patches that view in place immediately.

The webhook payload alone turned out not to be trustworthy enough to use directly, which was a fun little discovery: it carries no branch name, and its pipelineId field is actually the run ID rather than the build definition ID that everything else is keyed on. So each event triggers exactly one “get build by ID” call to resolve the truth, and that result is what lands in the store. Events make it fast; the hourly sweep makes it correct.

What you actually see

Every 5 seconds the service publishes a fresh command, and that’s what drives the little show:

  • The 7 hexagon pixels are the pipelines — one each, in a fixed order. Each one always shows its own true status color, so the hexagon as a whole is an at-a-glance health check of everything at once. Green is good, red is failed, blue means running (and pulses through black so it’s obviously live), gray is cancelled.
  • The pipeline being highlighted right now is shown at full brightness while the other six are scaled down to about a quarter. It cycles.
  • The OLED shows the highlighted pipeline’s name on line 1, and cycles line 2 between its status (“Success: 2 Days ago”) and its branch. Once a pipeline has shown all its messages, the highlight moves to the next pipeline and the hexagon follows along.
  • The 4-pixel strip on the right is the highlighted pipeline’s last four builds, newest first. Three green and a red tells you something quite different from four reds, and it’s the fastest possible answer to “is this a blip or a trend?”

Colors and blink timings are all configurable per status, and — this detail matters more than it sounds like it should — configurable separately for the hexagon and the strip. The hexagon sits behind two layers of milk jug and the strip pixels don’t, so the same RGB triple reads as two completely different colors depending on which group it lands on.

Two bugs that weren’t where I thought they were

The panel would connect to MQTT and then get its socket killed immediately, with the broker logging Bad socket read/write and the client reporting a CONNACK “identifier rejected”. I spent an embarrassing amount of time on client IDs. The actual cause: umqtt’s default keepalive is 0, and this particular mosquitto build accepts the CONNECT and then hangs up. Send a real keepalive and ping on it, and everything’s fine.

Worse — and later — well-formed clients started getting rejected with strange, inconsistent errors again. This time it genuinely wasn’t my code. My broker’s Kubernetes Service had externalTrafficPolicy: Cluster, which let MetalLB’s L2 speaker answer ARP from a node that wasn’t the one running the mosquitto pod, forcing an extra kube-proxy SNAT hop that corrupted connections from at least one client. One word changed to Local and it’s been quiet ever since. I’d already rewritten a working MQTT library twice by then, chasing a bug that lived three layers below it.

MQTT Connected
MQTT Connected

So where are we, and what’s next?

The panel in its natural habitat
The panel in its natural habitat

This one landed a lot better than I expected. Most of my projects are fun first and useful second, if at all — this one is genuinely, boringly useful. It sits on a shelf above my little cluster of boards, and every morning I glance at it on my way past and know instantly whether anything needs my attention. No tab, no dashboard, no login.

The unexpected part is what that’s done to my behavior. When build health is a thing sitting in my peripheral vision, a red pixel is a small irritation that I want to make go away, and I fix things the same day instead of discovering them on Thursday. I did not build this expecting to become a more disciplined engineer, and yet.

Next up, most likely: moving off my local broker onto a cloud-hosted message queue, so the panel doesn’t need to live on the same network as the service driving it. Then I can hang it on my cubicle wall at the office, which was sort of the point all along. Or I’ll just build a second one — assuming I can find another junk front panel, which is honestly the hard part.

Everything’s up on GitHub if you want a closer look — firmware, the Go service, the tester CLI, and the case STL:

github.com/dsmithson/quad-t-mqtt-display

If you run into problems or have ideas, filing an issue on the repo is by far the best way to reach me — I have a bad habit of not checking blog comments for months at a time.

ScreenMaster III 3216 – Restoring a Classic – Part 2

In the first post in this series, we walked through upgrading the internal PC hardware within a ScreenMaster III 3216 hardware controller. In this post, we’re going to start creating the software to communicate between the internal computer and the hardware buttons, joystick, rotary encoders, and T-bar. All of the work discussed in this post lives in a GitHub repository that you can look through and use.

Before we dive in, let’s quickly agree on some terminology. I’m going to refer to the buttons, joystick, rotary encoders, and the T-bar collectively as the ‘keyboard’. Additionally, I’m going to refer to the Windows PC-based computer inside the console as the ‘host’. When we want to turn on or of a lamp for a button, for example, we’ll consider that to be a ‘host’ to ‘keyboard’ communication. Alternatively, when a user presses a key or moves the joystick a ‘keyboard’ to ‘host’ message will be generated. Hopefully this makes sense. Luckily the GitHub repository mentioned earlier in this post uses the same terminology in it’s documentation.

So how do we interact with the console hardware?

Communication between the host and keyboard is all performed using an internal Ethernet connection. This proved to be extremely helpful when upgrading the hardware, since we didn’t need esoteric connections to interface with the hardware. The keyboard has a hardcoded IP address of 192.168.1.3, and is hardcoded to send event messages to our host computer at the IP address of 192.168.1.2. This is quite limiting, but since it was initially designed to be an internal Ethernet connection we simply added a second USB network adapter during our PC upgrade dedicated to this interface. The image below shows the high-level Ethernet connectivity within the controller:

Over Ethernet, the messages between the host and keyboard are all simple UDP packets, with a very basic structure. I won’t detail the protocol here, but if you’re interested I’ve written up the protocol and published it within the GitHub repo here. Effectively, here’s the main list of things that you can do:

Host to Keyboard:
– Set lamp lights on or off for each button
– Set the LCD button background to Off, Red, Green, or some alternating combination
– Set the LCD button text – 3 lines, 6 characters per line

Keyboard to Host:
– Get event when button is pressed or released
– Get event when the joystick x, y, or z axis value changes
– Get event when a rotary encoder is turned (+/- change value)
– Get event when the T-Bar position changes

Time to write some software…

I’ve written and published a .Net Standard 2.0 library to implement the keyboard protocol and make it easy to consume from an application. The repository (listed at the beginning of this post) makes it simple to subscribe to events and control the simple pushbutton lights and LCD display buttons.

The repository contains a console test application that subscribes to events and writes these events to the console output. It’s a good example of how to use the library, and I’ve included a snippet of the test program’s Main function below for reference:

using Spyder.Controllers.ScreenMaster3;

static async Task Main(string[] args)
{
    // Initialize connection to keyboard
    var keyboard = new KeyboardInterface();
    await keyboard.StartupAsync();

    // Subscribe to events from the keyboard
    keyboard.TBarValueChanged += Keyboard_TBarValueChanged;
    keyboard.KeyPressed += Keyboard_KeyPressed;
    keyboard.KeyReleased += Keyboard_KeyReleased;
    keyboard.RotaryValueChanged += Keyboard_RotaryValueChanged;
    keyboard.JoystickValueChanged += Keyboard_JoystickValueChanged;
    keyboard.KeyAction += Keyboard_KeyAction;
    Console.WriteLine("Listening for events");

    Console.WriteLine("Press enter to exit");
    Console.ReadLine();

    Console.WriteLine("Shutting down...");
    await keyboard.ShutdownAsync(); 
}

If you’re playing along at home, you can pull down the library and build it from source, or for convenience I’ve also published it as a Nuget package for easier consumption.

Mapping out the buttons

With a software driver library and a test application created, we can now go through the process of pressing and releasing every button on the controller, and watching the event data to determine the logical button ID associated with each physical button and rotary encoder. The console has well over 200 buttons, so it took a bit. My GitHub repo has a Visio document with some different pages that include the index map, and below is an image of this as well as a keycap map for the ScreenMaster III 3216.

Now where are we, and where do we go next?

At this point we have a controller with a modern computer inside, and we have a software library that allows us to interact with the hardware keyboard. With these in place, the real technical challenges are out of the way and we can focus on building an application to control something similarly modern. With my history and existing software libraries and software applications to control Spyder video processors, it’s probably not a spoiler alert to say this is where I’m heading.

I’m not sure yet if the next blog post will be software heavy (yet) – I’m planning to spend some time planning a good console application before diving too far into code. That being said, I couldn’t resist getting some simple app(s) started. As such, I’ll leave you with a quick video showing an app framework that works with the keyboard interface and is setting the selected page based from menu button presses.

If you’re still with me reading through this blog series, thanks! I don’t expect this project to amount to much more than a love letter to an old product that holds nostalgic value to me, but I hope it might resonate with a couple people out there. And if you’re lucky enough to have one of your own ScreenMaster III or a Montage controller lying around, let me know about it!

ScreenMaster III 3216 Console

ScreenMaster III 3216 – Restoring a Classic – Part 1

The ScreenMaster III 3216 and it’s half-sized little sibling the 1608 were some of my favorite consoles every made at Vista Control Systems (later renamed to Vista Systems and later acquired by Christie Digital). These were the first consoles to incorporate 15″ touch screen displays to compliment the massive button boards from prior console generations. This console, like others before it, were purpose built to control a combination of Folsom (now Barco) VFC-2200 scalers and either Extron, Pesa, or Sierra routing switchers. We’re talking analog video only here – this was the nineties after all, when 1080p video was king.

It’s worth a note that this console would be appear again in the future as the control surface for the Vista Montage video processor, painted in blue and re-dubbed as the Montage SC-3200 console. This stuff is all but lost in time, even on the internet these days, and so I’ve included a picture below for full historic reference.

Let’s talk about this got started

Let’s talk about eBay. eBay is a great place to get great deals, and is a great place to troll late nights when you can’t sleep to find stuff you didn’t know you needed (and probably don’t need). One night I happen to be aimlessly looking around the internet, and somehow I made it to this posting on eBay:

This picture of the console immediately gave me ‘the feels’, as it has solid sentimental value. I remember shipping the first of these consoles, sometime back in the nineties. We had a problem with a driver for an internal USB network adapter, and it was causing the console to bluescreen. We weren’t sure it was fixed, but I ‘thought’ I had resolved the issue with an updated driver. We got the console in it’s case and brought it to the back door. After talking a bit about this bluescreen, Jeff Wilson and I opened the console up and made sure it booted successfully five times. As long as it could boot five times in a row, then we’d ship it. Good news – that was the day unit #1 shipped.

<rant>Just to touch briefly on that last point – this was a time when Windows was considered to be generally unstable, and we caught a lot of flack from rental and staging customers (and competitors). Here we were though, shipping a device with Windows (NT Embedded I vaguely recall) and asking customers to entrust their high-dollar live events on it. Software bugs aside, I think this ask proved to be a reasonable one and these consoles were way ahead of their time (my opinion). </rant>

So this console on eBay had an asking price of $599, and I sent an offer for $299 which was accepted. I don’t remember the real figure, but I think this console cost around $45,000 new. From the pictures I could see some button-board activity and a boot failure, and so I knew that (1) the embedded part of the console was alive and (2) the PC was ‘alive’ and the Compact Flash card we used for the hard drive had likely failed. After some back-and-forth on the shipping, we decided to drop the shipping case to avoid going freight, and all-in I think I was down $500.

The arrival

It took about a week for my console to arrive, and it showed up packed tight with instant-foam in a box that must have had a roll of fiber tape for additional support. It took a solid hour to practically chisel the console out of this foam tomb, but it shipped in pristine condition so I can’t complain.

After getting the console freed from it’s shipping cocoon, I gave it a good look over and powered it up. The screen showed the same hard drive failure message that I saw on the eBay post. The quick-keys (the top-right green button array) showed diagnostic counters that I was able to use to verify that the buttons, joystick, T-bar, and rotary encoders were all functional (minus a few burnt-out incandescent switch lamps). So far so good. Next up is to take inventory of the computer hardware and try to get it working again, but before we move on go ahead and take a hard look at that good-looking piece of hardware 🙂 .

Console internals

When you open the bottom of this console, you immediately see that there is a LOT going on. Luckily, when we look closely at the internals of the console there are two main subsystems – the embedded computer (PC) and the physical control interface (buttons / T-Bar / Joystick) subsystem. The picture below shows the console on it’s side with the bottom removed.

The application board is the main embedded component which houses the programming required to read and write the physical hardware. Luckily for us, the interface between the embedded computer and this application board is Ethernet. It’s difficult to see in the pictures, but there is an Ethernet crossover cable between a second Ethernet port on the SBC and the application board. This is great, because it means that we can upgrade the embedded computer without worrying about obscure hardware interface dependencies.

Before we move on to replacing the embedding computer, let’s take a second to talk about the embedded computer that came with the console. This was an EBX form factor computer called the ‘Olympus’ and was manufactured by Arcom Controls (datasheet here). This was a 1.0 Ghz Celeron processor and 128MB of RAM. The hard drive was provided with a 128MB Compact Flash card, which has limited read/write cycles and almost certainly explains why the console eventually stopped working. I did try jump-starting the console with a new Compact Flash module, but the SBC was so old that it felt that even if I could run a modern operating system on it (unlikely) that it would be terribly slow. It was also sporting a 3.5″ floppy disk drive, for those of you who remember what that even is, and I knew that needed to be brought into the modern age as well. With that back-story out of the way, let’s get to upgrading.

Upgrading the Embedded Computer Hardware

Full disclosure here – this console sat on it’s side in my living room for several weeks, and I’d give it occasional blocks of evening and weekend time. This wasn’t particularly hard, but there were some time-consuming parts of the effort that I’ll bullet point rather than writing a book about:

  • Removed the prior EBX form-factor PC board and graphics adapter
  • Removed the existing floppy disk drive
  • Drilled holes in the chassis to fit a new Mini-ITX form factor PC
  • Bought a +12V to ATX power adapter module to power the new SBC
  • Added two USB to RS-232 cables to drive the touch screens
  • Added a USB to Analog adapter to drive the second touch screen (the SBC had one VGA port I was able to use as well)
  • Replaced the Neutrik Ethernet connector running to the back of the console with a new one
  • Modeled and 3D Printed a custom bracket to hold a new Solid State hard disk and expose 2x USB 3.0 ports to fit in the old floppy disk drive spot.

The mounting and retro-fitting of the new SBC and parts was a pain, but I’m pretty happy with how it came out. This thing feels like it could travel and be as reliable as the original console was. I am particularly proud of the 3D printed module that I made to hold the hard drive and expose a couple USB ports, and you can see this along with a bunch of other build pictures below

After the aforementioned evenings and weekends spanning may weeks, the hardware for the console was done. It was now sporting an Intel i7 processor with 16GB RAM and a 512GB solid state disk, and was running quite nicely. I did sneak in a picture of installing Windows on the replacement Embedded PC / Single Board Computer (SBC), but I’ve saved this for the last section of this post.

Let’s put Windows 10 on this thing

A modern console needs a modern operating system. I actually considered installing Ubuntu Linux for the fun factor, however ultimately I decided to install Windows 10 Pro. It’s less ’embedded’, but it allows for wider flexibility for software to run (including Vista Advanced / Spyder Studio / Spyder Client software).

One wrinkle that I did run into was around the touch screens. They have RS-232 serial controllers, and the native 3M driver didn’t work. I remember running into this with my Car computer touch screen (same hardware), and so I knew I needed to pony up and buy a license for a software called touch-base that makes the touch controllers work with Windows 10. Related note – the screens for the console are each 1024×768 resolution and VGA, and if I were braver I would crack open the top and try to replace them with some newer displays. Maybe someday.

Big reveal time

I don’t need a lot of words for this – I’ll let the pictures of the console working speak for themselves. For those curious, the pretty looking Spyder control software is the Spyder Client application that is available on the Windows 8/10 Store.

So where are we, and what’s next?

So we have a large powerful hardware console alive and well, and we have a working PC operating system on it. Probably a way over-powered PC in there hardware spec-wise, but we still don’t know what we’ll do with it so hard to say. By the time I write this, the console has actually been sitting in my dining room for the past two months. I’ve already done some development and created a software interface to integrate the hardware with the PC, but I’ll save that for a future blog post.

In the next post, I’ll talk about how the PC-to-embedded hardware interface works, and we’ll probably have a simple app to show take a look at. I expect the next post will be a bit shorter – this one was a monster and it feels like it just skims the surface. With that, thank you for reading and see you next time!

The Useless Box

You know, I’d argue that a box that makes you smile hardly qualifies as useless.  Nevertheless, today I finished assembly on the self named ‘Useless Box’ by solarbotics.com.  This black acrylic box has a single switch on it’s top, which when flipped, turns on a motor connected to a ‘finger’ which pops up and flips the switch back to the off position.  I’ve found myself flipping it on at my desk almost instinctively throughout the day, and it makes me smile every single time.  It’s just simply great.  Take a look at the video to get a better idea of what we’re talking about here.

 Assembly

Assembling the useless box is not hard, but it is a little time consuming, and does require you to solder a small amount of stuff together.  Honestly I think I spent half as much time peeling the protective brown paper off all the parts as I did doing the rest of the assembly.  Here is how your useless box arrives:

Raw materials for a Useless Box

Raw materials for a Useless Box

The assembly breaks into four or five steps, each taking about an equal amount of time.  The instructions estimated 1.5 hours for the full build, but it took me more like 2.5 hours (yes I’m slow).

Build Steps:

  1. Remove all the damn brown protective paper from the acrylic parts.
  2. Solder together the electronics.
  3. Assemble the main box.
  4. Disassemble and then re-assemble the box after you figure out that the piece holding the motor needs to be mounted as the rest of the box is being put together.
  5. Mount the electronics in the box, adjust for fit, and then test!

The electronics harness would have been much easier if I still had a vice to hold the parts in place while trying to solder, but a little scotch tape did the trick in a pinch.  Below is the finished electronic harness and the box parts all laid out:

Assembled electronics harness and box parts

Assembled electronics harness and box parts

And here is the finished product:

Useless box assembled product

Useless box assembled product

Call to Action: Go Buy One

This thing is fun, and is worth plunking down some cash for.  I’m sure you can get it at a number of places, but mine came from ThinkGeek, which is a site I believe you should be giving your money to anyway.  Below is the link; you’ve seen it in all it’s glory, now take one home for yourself.  It’s great for your desk at work and your home, so maybe get two…

Go buy the Useless Box at ThinkGeek