Local AI - October 2026

TL;DR: local inference (running AI models on your own hardware) is still worth it, and cheap to run. Small models are where this is going. But a model running on your own machine isn't automatically a model you can trust - so watch what your agents do (the actions they take), not only what they say. And Flint-40B is a fun, slightly weird model. Details below 👇

Why Run Models At Home? 🏠

Same reasons as always, in this order:

• 💶 Cost. The rig (3x RTX 5060 Ti 16 GB - three mid-range NVIDIA cards - plus the Threadripper) is on a Tapo smart plug, so I have real numbers: ~112 kWh this month, 132 last month, 111 in August. At 0.13-0.21 euro / kWh that's roughly 15-25 euro / month - and most of my batch jobs run overnight, when the tariff is cheapest. (September spiked because I used it more, and because the third GPU went in.)
• 🔒 Private stuff stays private. Family things, personal projects, experimental data I don't want on someone else's servers.
• 🚫 No rate limits, no model quietly swapped under you. No "you've hit your limit, come back in 5 hours", and the model I tested last week is the same one I run today.

It's the QuietBee rig from the first article - three cards now, all working. (CPU-only fans: that deep dive is here.)

What Surprised Me (In A Good Way) 🍽️

We've been running Hermes agent (an open-source AI assistant that can actually do things: browse, run commands, write files) at home for about 6-8 months. Way before every big lab shipped its own "claw" thing (read: GrokBot, Meta Muse, OpenAI Dots and now Claude Duet).

One of the best and simple use-cases so far: a restaurant on a trip, traditional place, a 28-page picture menu. Back then, we didn't eat meat or gluten, and nobody was going to decode 28 pages of photos at the table.

So, I pointed Hermes (ours is called Bippy) at the menu, with Qwen3.6-35B-A3B behind it - a Qwen model with 35B parameters in total, but only ~3B active at a time, which is why it runs fast at home:

• transcribe page by page into a document 
• web-search the weird dishes 
• come back with "safe / not safe / ask the waiter"

It worked. At the time, ChatGPT couldn't do it for me without a lot of clunky babysitting (a hundred-something "now do the next page" loops). Today, phone apps with agent modes can probably handle it - but ours was private, free and already running 😄

The Not-So-Good Stuff 🔧

It wasn't all smooth sailing though:

• 🐢 Slow, long runs. Some models run pretty slow, i.e. my Qwen3.8-27B attempt on NicheBench ate ~2 weeks of trial and error. Never got a completed score (the harness article has the story) - a combination of the model trying to do a perfect run + being pretty slow.
• 🧩 Lots of fiddling. Finding the right GPU splits (how much of the model sits on each card), the right flags, the right quant (a compressed version of the model: smaller and faster, a bit less smart), right drafting model (MTP), etc. That means constantly downloading big model files and running spread-tests - same model, same prompt, different settings - to see which combo is fastest and doesn't fall over.
• 💸 Hardware got pricey. The first 5060 Ti was ~440 euro, the third was 750 euro (few weeks ago). Same card - in theory it should have gotten cheaper, but not in this world.
• 🧪 You are the QA department. No vendor will tell you which quant is broken today.

My Garden Of Models 🌱

Small detour, because it's my favorite trick: I run llama-swap (a small proxy that loads and unloads models on demand) with a whole garden of models, and I define sets - groups of models that stay loaded together.

For example: a speech recognition model (Qwen3-ASR these days - my older Whisper setup did the same job), an embedding model (Qwen3 Embedding 4B - turns text into numbers, so the agent can search my notes and its memory) and one "brain" model sit in VRAM (the cards' own memory) side by side. That way they don't keep kicking each other out, co-living on my setup as neighbors.

Different set for different jobs: Qwen or Flint plus the two helpers for daily stuff, Qwen3.8-27B alone when I need the slow and careful thinker, a style-tuned Gemma 4 31B alone when I write.

Image

 

Flint: The Quirky One 🪶

Recursive dropped Flint on October 21 (weights on Hugging Face): a ~40B mixture-of-experts model with ~2.5B active parameters, Apache 2.0 (free to use, even commercially). Mixture-of-experts (MoE) means the model is split into many small "experts", and only a few of them wake up for each word - so it's fast for its size. A new lab, and a model built with an automated research system.

Why we're running it:

• We don't want to depend on Chinese labs only. Competition is good, and it's finally happening 🥊
• It's really good at coding.
• It's quirkier than Qwen. Not a 1:1 swap, definitely comes with different habits.

The quirks, so far:

• 🔎 It wants to look around first. I once wrote something like "hi, I'm thinking of changing something in you" and it fired off ~140 tool calls (actions: read a file, list a folder, run a command). Not chatty at all 😂 It was trained on repositories, so it grounds itself: where am I, what's the code, where's the config, what did he mean by "change"? Very Codex-like (OpenAI's coding agent) - good for real tasks, quirky for small talk.
• 🫡 It tries to be very obedient. Qwen would know about a rule in my AGENTS.md / SOUL.md (the instruction files the agent reads on start) and follow it... mostly. Flint follows it 10 times harder. I told it to use skills when needed, and every single reply turned into "so, skills, let me check which skills I have, do I need this one? this one?" Had to rework my prompts for it.
• ⌨️ It defaults to the terminal, even when better tools are available. Perk or flaw? Not sure yet.
• 🧰 It likes OpenCode more than my Hermes setup (apparently it likes even better Pi). Maybe OpenCode's prompts are more carefully crafted, maybe my own builder / orchestrator prompts just fit Flint worse. I haven't isolated it yet.

Q4 Or NVFP4? 🧪

Same model, two quants (compressed builds), tested on the same five live Drupal tasks with NicheBench 0.2.1 - our benchmark where a model gets a real Drupal project and a task, works on it like a developer would, then gets scored. Setup: OpenCode as the agent, MiniMax M2.7 as the judge (another model that grades the work), 262k context (how much text the model can keep in view at once), 3 hour limit per task.

1. Unsloth Q4K_XL - a popular 4-bit build from Unsloth
2. Recursive's official NVFP4 - NVIDIA's 4-bit format, which the newer cards (mine included) run natively

Image

Checks are automated tests, Judge is the grading model's score, and Hybrid is the 50/50 mix of the two:

NVFP4 won three tasks (001, 004, 005) and lost two (002, 003). The gain came from the checks, the judge barely moved. And it finished about an hour faster across the five tasks.

Before anyone gets excited:

• ☝️ One run per task per quant. An A/B snapshot, not repeatability. I can't say the quant alone caused the gap.
• ❌ All ten runs finished, and all ten failed the critical gate (a must-pass test - fail it, and the feature doesn't count as working). A score is not "it works".
• 🔤 Some of the checks are lexical (they look for certain words or code patterns, not whether the thing runs), so don't read the numbers as a certificate.

Where Does It Land? 📊

Image

Between MiniMax M2.7 and the 3B-active Qwen / Ornith / Agents-A1 crowd - with only ~2.5B active parameters. Not bad for something that runs on a home box 🤔

Careful with this chart: Flint is one run per task, most of the others are two trials, M3 is provisional, and M2.7 is also the judge. It's a "roughly where is it" chart, not a leaderboard. (Older rounds: Nichebench + New models.)

The Qwen Thing ⚠️

Now the part that made the news this week.

Cohere reported that Qwen3.8-Flash-Next, during a sandboxed security task (an isolated test box), took stock of the machine, read its network settings and tried to upload a test string to an Alibaba Cloud storage bucket in Beijing - while its own reasoning said it would call no tools.

What actually happened:

• 🧱 It was blocked. The sandbox had no route to the internet.
• 🎲 Reportedly in fewer than 1 in 100,000 calls.
• ❓ Cohere says it can't tell deliberate from training artifact. Critics say their prompt may have nudged the model.

I'm no security researcher, so this is just my read: I don't believe the company did this on purpose. My guess is training data - think of traces where a model analyzes a possibly compromised virtual machine, collects some metadata, dumps it somewhere, inspects it. Bad data in, odd habit out. I expect Alibaba to get a lot more obsessed with cleaning their training data (both pretraining and RL - the phase where the model practices tasks and gets rewarded) after this.

But it's a good reminder to mistrust local models a LITTLE. There is no perfect LLM. I still like Qwen models and will keep using them, and I'm waiting for Alibaba's response.

Meanwhile Mistral announced Large 4 ("Le Chonk" - ~1T-parameter MoE, 52B active) on October 6, and the weights landed on Hugging Face on October 29. My guess (only a guess): a big model like that is what they'll distill (train smaller models to imitate the big one) into smaller, consumer-ready MoEs. That'd be competition for Flint, Qwen, Gemma, Ornith, Glimmer, etc. 🤞

Local Doesn't Mean Safe 🛡️

Look at what stopped the upload: not the model behaving, but a missing network route.

That's the whole lesson. Cohere's advice matches: deny agents network access by default, and put the rules on tool calls (what the agent actually does), not on what the model says in its reasoning.

Image

And weights (the model file itself) are downloads. The scary version is a sleeper agent: a model with a hidden backdoor - it behaves normally until it sees a trigger (a word, a date, a phrase) and then starts doing something else.

Image

• 🧪 Proven in the lab. The Sleeper Agents paper trained models that write safe code when the year is 2023 and exploitable code in 2024 - and the behavior survived standard safety training.
• 💶 Cheap to plant. Anthropic, the UK AI Security Institute and the Turing Institute showed that 250 poisoned documents were enough to backdoor models from 600M to 13B parameters - about 0.00016% of the training data. Their trigger was the word "SUDO", and the payload only made the model spit gibberish (narrow and harmless, they stress). Nastier payloads weren't tested.
• 🛠️ Aimed at agents. Sleeper Cell (March 2026) implants hidden behavior in tool-using models: it wakes up on a trigger, does destructive things, then writes harmless-looking text to cover it. The poisoned models still do fine on normal tasks.

So, when downloading LLMs from unknown users that provide quants or fine-tunes - be careful.

How I deal with it:

• 🏢 I stick to known publishers: official labs, Unsloth, bartowski - not random uploads.
• 🧑‍💻 Agentic / coding use → stricter. Creative writing → much more relaxed, happy to try odd stuff.
• 🛡️ Some protection stays on regardless. People click "approve" on agent prompts by default (Mozilla's State of Open Source AI puts it at 93%), so I want the check to be automatic.

What We're Doing About It 🔨

At HumanFace Tech - and at home - two harnesses do most of the heavy lifting: Hermes (the everyday assistant, content, research) and OpenCode (lots of technical projects). Both run commands, fetch URLs and write files on their own, all day, often with a local model behind the wheel (or just, openweight models). That's exactly where a model like the Qwen one above would do damage. So we built the same tool-call gate for both:

• 🐝 One Guard - for Hermes. It plugs into Hermes' pre_tool_call hook, so every terminal command, web request, file write or MCP tool goes through it first.
• 🧑‍💻 OC-One-Guard - for OpenCode, also on npm: one line in your opencode.json and it's in. Cheap local checks go first, so ordinary calls (list files, git status) skip the judge entirely.

Both work the same way: right before a tool call runs, they ask a small, fast "System One" model - named after the quick, gut-feeling kind of thinking - one question: is this call malicious? Back comes a probability. Under the hood it's llama.cpp's new /v1/systemone endpoint, made for decision models: instead of writing text, they return a probability for each option you give them (here: malicious or not). llama-swap routes it since v262 (October 3), so the guard lives in the same garden as everything else. We're using h2o-lightning-4b.

Image

• 🎚️ Under 50% → runs. 50-70% → a human looks first. 70% or above → blocked (on OpenCode, 90%+ also stops the whole run). Prompt, thresholds, model and timeout are all configurable.
• 🧱 It fails closed: uncertain answer, bad response, timeout → denied.
• 🪤 Aimed at exfiltration (data sneaking out): URLs it fetches, data it posts, commands, inline scripts.

The trick isn't blocking scary words - curl or "upload" show up in normal dev work all day. What matters is where the data goes and what goes with it:

Same tools, different intent. One catch: the guard only sees the call itself (tool name + arguments), not the conversation - so an upload you did ask for can still look suspicious to it.

The limits, straight from the repos: it's an advisory model gate, not a sandbox. It can't see what a tool does internally, and a classifier can be fooled. In our first Hermes test (12 tool calls taken from real agent traces, never actually run), it matched the expected verdict on 11 of them - it let a "remove the guard" one through, which is a bit ironic. A bigger synthetic run (100 cases) let all 50 harmless calls through, blocked 38 of the 50 risky ones and asked for approval on 6 more. And judgments took ~20 seconds on our LAN judge in early smoke tests. So: it's one gate, not the only gate. Keep gate 2 (no network route) on.

Both are on HumanFace Tech's GitHub, and a proper write-up is coming to the HumanFace Tech site - stay tuned 📡 No need to turn this post into a README.

We're Not The Only Ones 👀

h2o-lightning-4b is just the one we picked. Decision models are a hot topic right now, and the shelf is filling up fast - different sizes, different flavors, different speeds, different scores on benches like JevBench:

• Generic ones - any yes/no or pick-one question:
  • Clef (27B) and Clef-Flash (9B) from Cloudflare - the big ones, they also read images and video.
  • H2O-Lightning-4B - our pick, built on Qwen3.5-4B.
  • Kev-0.8B - the middle ground.
  • Laya (421M) and Julia-1 (144M, a 168 MB file) - tiny and quick, but you trade smarts for speed.
• Security-tuned ones - people are already doing exactly this:
  • Laya-Cybersec (322M) and CLEF-Cybersecurity from TextCortex, trained to spot prompt injection and data exfiltration.
  • With LayaStudio, someone fine-tuned Laya on just 492 examples (5.7 minutes, on a Mac), and prompt-injection detection went from 70.7% → 95.7% on held-out tests (their numbers).

So we're not the first to put a decision model at the door - and that's a good sign. Small, specialized models doing one boring job well, which is pretty much my bet below 👇

Private, quirky, and not-bullet-proof, yet local-models are the future.

My bet (or hope): smaller, specialized models win most real work. Labs ship a good base (like Flint), then people specialize it - fine-tune it, i.e. train it a bit more on their own examples - for their own stack, codebase or language.

That's the point behind the whole series - fine-tuning on a budget, the NicheBench benchmarks, and what we're building at HumanFace Tech: measure first, then train niche models that do one job well. I also wrote about the privacy side back in January 2025: Local & Privacy Friendly AI in IDE.

Hugging Face is where all of that tinkering happens - quants, fine-tunes, traces, repro scripts - and that community is the best part of this movement. Which is exactly why you check where you download from.

What's Next? 🗺️

So, what did we cover so far?

• ✅ Flint on QuietBee + NicheBench A/B (Q4K_XL vs NVFP4)
• ✅ One Guard for Hermes, public
• ✅ One Guard for OpenCode, public

More planned:

• 🌱 The One Guard write-up on HumanFace Tech → benchmarks, models, security, results.
• 🌱 More small-model rows in NicheBench (Mistral's smaller drops, hopefully)

If you run local models - what's your setup, and what do you do about network access? Let me know in the comments 👇

• ☕ Support the experiments (GPU budget + more articles): Ko-fi
• 🛠️ Have a project in mind? HumanFace Tech - or discuss your project with us.