Why I Moved My AI Experiments Local

In late 2023 the pace of AI tooling changed fast. The cloud options were exciting, but they also made the cost and privacy trade-offs impossible to ignore.

So I started moving experiments into a local homelab.

The setup

The early setup was basic:

• a GPU-backed VM
• local model storage
• simple monitoring from day one

That was enough to prove the idea.

What I learned

Local inference was not perfect, but it was good enough for a surprising amount of work. The bigger models stayed useful for harder reasoning tasks, but a local model could handle a lot of the boring stuff cheaply.

That changed how I thought about the system. The cloud stopped being the default. It became the exception.

The real lesson

Once you run a few workflows locally, the shape of the problem changes. Privacy becomes easier to reason about. Costs become predictable. And you start writing down the routing rules because the routing rules themselves become valuable.

That was the start of a much longer shift for me, but the important bit was simple: the local stack made the work feel more deliberate.