Nichebench: the harness

So, Where Have I Been? 🧭

It's been about 10 months (!) since the last article. Most of that time went into my work at Dropsolid as an AI Product Architect. I've participated in the design, building, and maintaining the AI infrastructure: inference, models, guardrails, tracing, vector stores, and their integrations with Drupal. Quite a few moving parts - plus, some participation in the Drupal AI community, which was a fun process but definitely took some time ⏳.

Meanwhile, we secured €2.5 million in financing at Dropsolid AI. There was a lot happening on that side, and moving NicheBench forward alongside it was harder than I expected 💢.

The hardware was another limit. Progress with one GPU was difficult, and even after adding a second one, running a model through the new benchmark could turn into days of trial and error - i.e. I'd start a run midnight and would wake-up to a failed attempt 😥. And, keeping up with the AI news takes a ton of time as well: new models, new harnesses, and experiments to understand what they could actually do. Useful work, but it competed with both NicheBench and writing about it.

So, this is where things stand. The goal is still to experiment with fine-tuned models for Drupal coding. NicheBench now has a runtime mode, QuietBee has more hardware, and I have some results to share - along with a few runs that went nowhere.

First, A Thank You ✨

Dropsolid AI became the first sponsor of our Drupal-focused AI experiments and this series of articles. They chipped in toward QuietBee's second GPU, which let me run better models and test them on NicheBench's runtime tasks. That was a practical help with the experiments - thank you, Dropsolid!

Image

By the way, if you or your company would like to help fund this work, there are sponsorship opportunities too. The goal is to work toward fine-tuned Drupal coding models (initially) and document what works along the way - including what doesn't. I can't promise a finished model, but that's the direction I'm working toward ⭐.

Reach out to me via consultation page or via my LinkedIn.

NicheBench v2: Can A Model Do The Drupal Work?

I've been calling this NicheBench v2, though the actual package version is 0.2.x (idk when I'll be ready to call it >=1.0), the runtime-capable release. The earlier quiz and code-generation evaluations are still there. The new part is testing what an agent actually builds inside a realistic running Drupal environment. The harness is in the NicheBench repository.

A model can answer Drupal questions or generate a convincing implementation without delivering a working feature. I wanted to test that next step: give it a project and a task, let it work through OpenCode, then check what it actually did.

I created a dummy project: NicheJobs - a simple jobs based website, similar to LinkedIn. Then, I created 5 separate tasks (each task has its own branch) - each task is both specific and vague - something a real client would ask 😉

How Does The Runtime Test Work?

To answer the question "can a model work as a coding agent inside a known harness?" - we use our NicheJobs project, we create a full working environment around it (DDEV), and then we let the LLM work through OpenCode (the coding harness). Our 5 task branches provide pinned starting states; NicheBench puts the selected task into its own workspace, makes sure it's reset to the beginning (clean workspace), no extra files, no artifacts and kicks off the process by providing the Task to the LLM (in the Opencode). We wait for a few minutes (or hours, OR DAYS) and the model does its magic: it can run commands, query the database, run tests, etc - this gives the model a huge freedom & flexibility (but in the cage, just to be safe).

Once the job's ready, the LLM will raise a flag 🚩.

Image

When the flag's raised, NicheBench collects everything into an artifact: code changes, traces, logs, etc. And finally, we run deterministic checks, and sends the artifacts to a separate AI judge (an LLM itself, with hints about what's supposed to look "right").

Here's an overview of how the pieces come together and talk to each-other - a high level architectural overview of sorts:

Image

I've grouped OpenCode and DDEV together in the diagram because they work on the same task - the DDEV containers actually run alongside the cage, not inside it. Each task gets its own workspace and Drupal setup → we clean up afterwards and start fresh for the next one.

🛠️ Here's what I gave the agent to build:

1. Job Application wizard
2. A company verification process
3. Saved job searches and job alerts
4. Company-profile completeness gate
5. Super advanced job search

These are feature-sized tasks, not one-function exercises. Runtime evaluations ran sequentially on this host. I also tried to force the agents to touch some API layers that were recently introduced (so it plays on either their knowledge, or their ability to discover it from the Drupal's Core itself).

What Do The Scores Mean? ⚖️

For this task pack, the runtime hybrid score is 50% deterministic checks and 50% judge score. Deterministic checks test defined outcomes; the judge reviews the implementation and its artifacts. The separate acceptance gate fails if a critical check fails, even when the score is high.

The score combines deterministic checks and the judge’s assessment - useful for comparing what the models built, but not proof that every feature works. If a run stops before scoring, I leave it unscored, rather than calling it a zero.

Early Runtime Results - And Their Limits 📊

These results answer the runtime question better than a quiz or code-generation score can. They are still a snapshot of particular models and runs, not a current leaderboard.

In the table above - you can see the hybrid (final) score of the models when it comes to run-time tasks (I also included old "quiz" and "code-generation" score alongside to compare the models). I was NOT very consistent in my runs, some models had 2 runs (and an average between those), some took too long so I settled just on a single run.

Here's a view of the results graphically, sorted by runtime score, highest first. Laguna and local Qwen3.8 have no scored runtime result → I've left them out of the graphs.

Image

The score blends deterministic checks and judge assessment - call it hybrid or final score. Here's a break-down PER TASK. Each model performs differently on different tasks - some tasks can touch certain APIs that the model might not be aware of.

Image

Quick heads-up before comparing the bars 👇

• Main cohort: Two trials across five tasks. 
• GPT: One trial across all five tasks. Useful as a single-trial reference, not a two-trial comparison. It's expensive so I only ran it as a reference.
• M3: One trial across five tasks. I ran this one quickly few days ago - consumed all my weekly token allowance on my token-plan.

I mainly focused on the Qwen models, and the 'custom' fine-tuned flavors of the Qwen like: Ornith or Agents-A1 - that's because I actually eye those models for further fine-tuning myself.

The runtime speed is also an interesting story - but at times also misleading. Some models believed they were done - stopped early and got an average or bad score, others tried hard to polish their work, built a test-suite, etc and in result took way more hours per 1 task.

Here's what I learned from these experiments:

• Harness + Freedom results in bigger spreads, lots of noise. The model can take a sudden turn and randomly either decide that it's done, or resolve the same problem wildly different. But you still see smaller models performing worse than bigger models (or, better trained).
• Qwen 3.6 35B A3B model performed close to Minimax M2.7, I really hoped that Qwen3.8-27B (currently my favorite local model) - would show results closer to GPT5.6-Terra or Minimax M3, but I still failed to get any meaningful results in my tests(will update this article when I do).
• Spin-off models like Agents A1 or Ornith - didn't show dramatic changes here. In some cases the classic Qwen 3.6 35B performed even better.
• There's still room for improvement - again we're not aiming to build best generic coding model - we just want to make sure it performs BEST in a specialized given domain. These series of articles are still relevant and so is the effort, or so I hope 🫡

BTW, the previous NicheBench vs Drupal 10-11 article covered the quiz and one-shot code-generation results. In NicheBench + New models, I was already looking at tool use, long contexts, and multi-turn behavior before Drupal knowledge. This round puts that agentic part to the test - an OpenCode agent that can inspect and edit a project, run commands, and check its work in DDEV.

Runtime Failures

• Qwen3.8-27B, local: I spent roughly two weeks trying to get it through the benchmark. Token speed looked fine, but I never got a completed score. I tried different budgets and local setups; it seems like the model is afraid to fail and spends 10x time just to get everything perfectly right - which means that 1 task = more than 9 hour run.
• Qwen3.8-27B, GROQ: By my recollection, hosted inference cost about $15 in an hour or two, then I stopped without a completed score. I was worried that it will repeat the same pattern, spending 4-5 hours (and, thus $30-$50) per just 1 task.
• Provider / judge calls: Not every interrupted run points to bad code. Ornith task 003 could not be scored after a judge/network failure. M3's first task had interrupted streams and retries; the retained capture doesn't establish where the interruptions came from.
• Laguna-S-2.1: Runtime attempts ended through watchdog or hard timeouts, with no completed runtime result.

The model, harness, provider, and budget all play a part here 🤷‍♂️.

UPDATE: QuietBee Is Getting Its Third GPU 🐝

The original QuietBee build article was published almost a year ago. Back then, I had the Threadripper base with 96 GB of DDR4, and buying three RTX 5060 Ti cards was still the plan. Now two RTX 5060 Ti 16 GB cards are running, I've bought the third, and RAM is up to 128 GB of DDR4.

Yes, 128 GB of DDR4 in this RAM-apocalypse. TrendForce has been reporting reduced DDR4 allocations as suppliers prioritize HBM and server memory. Building a budget lab hasn't got easier.

The first 5060 Ti cost me around €440, second ~€580. The third cost €750, and that was the cheapest I could find. World went crazy 🤯.

I also came across Kai's updated local-AI hardware comparison. Almost a year after my build article, he landed on 2 x 5060 Ti @ 16GB cards too. Nice to see someone else arrive at the same choice - though his comparison also makes the point that the software and how you split a model matter a lot.

More GPUs don't automatically become one big pool of memory. What we can run or fine-tune still depends on the model and how the software uses the cards. Having that said, I still have room for 4th GPU (my PCIe lanes allow it) - so, looking forward for anyone who wants to help out!

Image

What Comes After The Benchmark? 🚀

The actual next step is data collection and synthetic data generation. There is quite a lot of complexity in that process, so I want to start with some basics, fine-tune a model, and check whether we can make meaningful progress on NicheBench.

I don't need to build the entire data pipeline before testing that idea. For this first experiment, start with a small, checked set of useful examples, run a fine-tuning experiment, and compare the model against its original baseline under the same evaluation conditions. Then use what we learn to decide what data to collect or generate next.

The evaluation tasks also need to stay out of the training set. Teaching a model the benchmark answers and then reporting a higher score wouldn't tell us much about whether it got better at Drupal.

• 📚 Collect Drupal examples and generate synthetic data.
• 🔎 Start with a small, checked training set.
• 🧪 Fine-tune a candidate model.
• 📊 Compare it with the original model on held-out tasks.
• 🔁 Inspect what changed and improve the data.

That's the next experiment. Can a focused training set make a model better at this niche, and does that improvement show up beyond the quiz? I don't know yet. But we now have a way to measure more than whether it can explain a Drupal API.

If you work on Drupal AI, I'd be interested in what kinds of coding tasks you'd want to see tested - leave a comment 👇.

Now that you reached the end 😆 - another thank-you to Dropsolid, the first sponsor of this Drupal-focused series. Their support helped get QuietBee's second GPU into the rig.

• ☕ If you'd like to support the experiments, there's Ko-fi. Donations go toward the GPU budget and more articles. I'm already eyeing a fourth 16 GB GPU → 64 GB of total VRAM across four cards, giving me more room to continue the research.
• 🐝 Company sponsorship is welcome too. If you'd like to support the Drupal-focused work, get in touch - we'll discuss the details privately.
• 🛠️ Have a project in mind? Check out HumanFace Tech and book a meeting.