Autonomous agents operating in real operating systems cannot rely on HTML DOM selectors. Desktop apps — Spotify, Photoshop, VS Code, Slack — are raw bitmaps.
To power LuxurAI Cowork, we built Neo v0.1: our dedicated vision-language model cluster specialized in real-time visual perception.
Technical Challenges Solved
1. High-DPI Scaling & Coordinate Drift
On Windows 11 with 150% or 200% scaling, pixel coordinates drift between OS display APIs and DirectX screen grab buffers. We implemented an affine transformation matrix that recalibrates bounding box centers with sub-pixel precision.
2. Dual-GPU Inference & Low-Latency OCR
Screen perception must return action vectors in under 300ms. By caching static UI anchors and running parallelized OCR token embeddings, Neo v0.1 spots buttons, inputs, and text elements in near real-time.
3. Fail-Closed Safety Dial
Destructive operating system calls (file deletion, terminal execution, credential input) are intercepted by a strict human-in-the-loop safety gate, requiring user confirmation before execution.