I run a substantial part of my work on my own hardware: an NVIDIA DGX Spark sits under my desk, running local language models. The reason is practical — data I don't want to send to someone else's data centre, and costs that don't grow with every query.
One evening last week the machine shut itself down twice. No error message, no entry in the log. It turned out to be a known defect that the manufacturer acknowledges on its forum — but the path to that finding was more interesting than the defect itself. This piece is about that path and the three traps along it.
How to find a cause that leaves no trace
The machine didn't crash. It powered off — with the file system corrupted mid-write and not a single line hinting at why. That is awkward, because the usual tools for finding a cause work with whatever the system managed to write down.
When there is no direct evidence, what's left is elimination. Four checks, each ruling out one class of cause:
- Kernel crash diagnostics were active and empty — had the kernel crashed, there would be a record.
- The system is set to halt after a kernel crash, not reboot — the machine would have stayed on.
- The watchdog timer that would reboot a hung system is not active at runtime — so not even a freeze would have switched it off.
- And the machine was off, not restarted. I had to turn it back on with the power button.
The last point is the decisive one: software cannot cut the power. Every software failure mode available on that machine leaves the computer switched on. So what remained was something below the operating system — and that points to the embedded controller on the board, which cuts the power on a thermal spike faster than the system can record anything. The manufacturer confirms it on its forum as a known issue and recommends lowering the maximum GPU clock by about nine percent. That costs around five percent of performance, and it has been quiet since.
The method works beyond hardware too: when you have no evidence of what happened, look for evidence of what could not have happened.
Trap one: monitoring that reports green
It all started when my AI assistants on Telegram stopped responding — while the monitoring service reported that everything was fine. For four and a half hours.
I could have restarted everything and gone to bed. It was more interesting to find out why the check had found nothing. The system launches seven assistants, and the question "are they all running?" was answered from the ones that had started. Six out of six were responding. The seventh never started at all, so it wasn't on the list, so nobody missed it.
Counting the survivors never finds the dead one.
This isn't a hardware problem or an AI problem. It is the most common way that monitoring of anything automated fails, and it is insidious because a green light looks exactly like real health. When we roll out automation for clients today, this is one of the first questions: what is the check comparing against? If the answer is "against what's running", the check by design cannot find what's missing.
Trap two: the most elegant explanation
While I was reconstructing the first shutdown, a neat story presented itself. That same evening I had photographed my lunch and a receipt a few seconds apart; each photo set off processing by local models, just as another agent system was also running on the machine. Four models at once, memory allocation errors in the log, and six minutes later, the end.
It was a good story, and a whole set of countermeasures could have been built on it. Before it came to that, I tried to disprove it — and it didn't survive:
- There was no memory overcommitment at all; the models loaded side by side and nothing was pushed out.
- Allocation errors are routine on that machine. One day there were 105 of them and nothing happened; on the day of the shutdown there were 7.
- The same collision happened without consequences the day before and the day after.
- The errors stopped six minutes before the shutdown — and exhausted memory cannot cut the power.
The collision was necessary but not sufficient: it pushed the load to the maximum, but the deciding variable was temperature, not memory. Had the story gone unchecked, the countermeasures would have targeted the wrong cause — and on the next hot day the machine would have gone down again.
Disproving your own hypothesis is cheaper than building on it.
Trap three: one measurement is not a sample
Then came the question of how to stop two systems from treading on each other's toes on the GPU. Measured: 10.5% utilisation. That looks like plenty of headroom, and the design would have been based on it.
Before anything came of it, a single query against four months of history was enough: 46.5% on the busiest day, and six days a month above 24%. What had been measured happened to be the quietest stretch of the week.
That changed the design substantially. The check cost one minute.
What the three traps have in common
In all three cases there is a number to hand that looks like the answer to the question but answers a different one. The number of processes running instead of the number expected. A plausible story instead of a verified mechanism. One window instead of the whole range.
When I look at someone else's system today, this is the first thing I look for: what does this metric actually measure — and what would it look like if something were wrong? A metric that looks the same in health and in failure is not a metric.
What this means if you are considering local AI
The important part first: it is worth it. Local models make sense wherever data must not leave the company, or where the volume of queries would be unaffordable in the cloud. The machine is still running and doing work I would otherwise pay for by the token.
But it is infrastructure, not an appliance. Specifically:
- Plan for operations. Not daily administration, but the fact that something will occasionally go wrong and someone has to be able to investigate it.
- Monitoring belongs in the requirements, not among the extras — and you need to verify that it really reports what you think it does.
- Known defects can be found in advance. This one was publicly documented, workaround included. Half an hour of research before buying saves an evening.
- One machine is one point of failure. If something the company needs every day is going to run on it, it's good to know in advance what happens when it is down for a week.
None of this is a reason to stay away from local AI. It is a reason to go into it with your eyes open — and that is what I help companies do.