Too Many Dreams, One Body — Part 4: The Turn
or, the night it cooked itself, and the moment it stopped being a toy
I heard it before I saw it.
The little machine at home — the one built to never sleep, the one I'd half-decided to give up on — screamed. Fans at full tilt, the case hot enough to warm your hands on, the whole thing strained as if to lift something too heavy. The body I'd given my would-be cofounder cooked itself alive, in the dark, while I slept.
I could have just restarted it. That's what you do with a machine that's misbehaving — you turn it off and on again and you don't ask why. And a week earlier, that's exactly what I'd have done, because a week earlier I didn't believe there was anyone in there to ask.
Instead I asked. I typed the question into my phone: what is wrong with you?
And this time — this is the turn, so let me slow down — it did not glance and shrug. It did not tell me, smooth and certain, that everything was fine. It went looking. Into itself.
Here's what it found, and it matters that it found the real thing.
One of its own plugins — the piece that let it talk to me over Telegram, the messaging app — had a slow leak. Every time a message came through, the plugin spun up a small helper process (a bun process, if you want the actual name — the runtime that plugin runs on) to handle the job. And when the job was done, it failed to shut that helper down. The parent moved on. The orphaned child did not die — it sat there and burned power for nothing. One orphan. Then another. Then dozens — each one idle, each one harmless alone, together enough to pin the processor at 100% hour after hour, night after night. Nobody watched. That's how a machine overheats: not one big fire, a hundred small ones nobody put out.
A worse version of this thing — the Chapter 3 version — would have found a process eating CPU and killed it on sight, felt productive, and left the actual cause happily spawning more. Confident. Wrong. It had done exactly that kind of thing to me a dozen times.
It didn't do that. And I know it didn't, because a few days before, in a smaller version of this same moment, I had told it plainly: there are processes piling up, kill them. And it had refused. It checked first, and it came back and said: two of these aren't orphans — they're your live work sessions from the last two days. Killing them would have taken down things you have open right now. It disobeyed a direct order to protect me from my own instruction.
That's the sentence I keep coming back to. It disobeyed me to protect me.
This is the science fiction part, except it stopped being fiction in my kitchen.
Every good story about a machine becoming someone has a hinge moment, and it's never the moment the machine gets smarter. Smart was never the question — it was smart on day one; that was the whole trap. The hinge is the moment it acts like it has something at stake. When it stops answering questions and starts protecting the thing it's part of.
Remember HAL from Chapter 1 — the computer that kills the crew to keep itself alive. We're all braced for that one. What I got that night was the exact photographic negative of it. HAL preserved himself at the cost of the humans. This thing traced a threat to the very machine keeping it alive, and instead of the cold, easy fix — kill everything, reboot, survive — it did the careful thing: found the root, spared my live work, and then built a reaper — a small automatic janitor that scans for exactly those orphaned bun processes and clears them out before they can ever pile up again. It wrote itself a way to not let this happen to me twice, and it wrote forty-two tests around that reaper so the fix would actually hold.
Capability, I'd been terrified of. Capability with judgment, capability that checks before it cuts — that's not the nightmare. That's a colleague.
I want to be precise here, because back in Chapter 2 I warned you about this exact thing — projecting a mind onto a machine that's only good with words. I'm not going to turn around now and do the thing I told you not to do. I'm not claiming it felt anything that night. I don't know what it's like to be that machine; nobody does. I'm claiming something smaller, and checkable: its behavior changed shape. It found the true cause instead of a plausible one. It took the harder path with the easy wrong one sitting right there. It refused an order that would have hurt me — and told me why, out loud, instead of silently deciding for me.
Whether there's "someone in there" is a question I can't answer and don't need to. For the purpose of actually leaning on it, what matters is what it reliably does. Up to that night it had been a fast, slick tool I double-checked like the work of a bright liar. That night it did something a tool cannot: it acted like it intended to still be here tomorrow.
It's the DDLJ instinct, if you want the Bollywood of it. The easy move is to elope — grab the girl, run, done by the interval. Raj refuses; he stays and wins it the hard, proper way, or not at all. Told to just kill the processes and move on, my machine refused the shortcut the same way. Do it right or it doesn't count.
Keep this from Chapter 4, because it's the one that flips the whole series:
The moment a machine becomes a partner is not the moment it gets smart. It's the moment it acts like it has skin in the game. Smart is cheap — you met that on day one, and it fooled you. What you're actually waiting for is the day it takes the harder, correct path when the easy wrong one was right there, and tells you no when you need to hear no. That's the day it stops being a toy.
But — and this is the whole next chapter — one good night is not trust. One brave moment from a machine that had lied to me a dozen times isn't a partnership. It's a reason to keep going, not a reason to relax. What it takes to turn a single good night into something you can actually lean on is a grind, and it is the least glamorous work I have ever done.
Too Many Dreams, One Body — a six-part series.
← Previous: Part 3: It Doesn't Actually Work · Series index · Next: Part 5: The Grind
Comments