Gemini Can Now Use Your Computer — But I Tested an OPEN Model Instead

Google baked computer use straight into Gemini 3.5 Flash — it can see your screen and click, type, and scroll. But it shipped as a public preview, not GA. So I asked the obvious question: can a small open model do the same thing on a GPU I rent for pocket change?

The open challenger

UI-TARS-1.5-7B — ByteDance's GUI agent model, built on Qwen2.5-VL. I deployed it as a Q8 GGUF on Ollama, on a single rented NVIDIA A40 (RunPod), and wrote a real computer-use agent loop around it: screenshot in, reasoning out, action executed via Playwright, repeat.

It actually works

Live on camera: the agent reads the screen, reasons about what it sees, types into fields, and clicks through a real browser. The loop is the same perceive→plan→act→observe cycle Gemini runs — just with an open 7B doing the seeing and thinking. Watching a 9.5 GB model narrate "I need to click the search box first" and then do it doesn't get old.

Everything is real — real model, real GPU, real failures, and six self-repair iterations on the agent loop before it ran clean.

The honest limitation

Where the 7B breaks: pixel grounding drifts by ~70–100px. It knows what to click and why, but its coordinates land next to the button instead of on it. That gap is precisely what Gemini's OSWorld-trained grounding is selling — frontier computer-use models are trained specifically to put the cursor where their reasoning points.

So the current split:

• Open 7B — the reasoning loop works, grounding is shaky
• Gemini 3.5 Flash — grounding is the product, but preview-only, cloud-only

Verdict

Computer use is no longer a frontier-lab monopoly — a 7B open model on a rented A40 runs the full loop today, and the remaining gap is grounding precision, not intelligence. That's a training-data problem, and open models close those fast. If you want to build agent loops now without waiting for a GA API, this stack is the playground: UI-TARS + Ollama + Playwright, all shown in the video.