Target sources
One action surface, four ways to find it.
Detection is a source, not the product boundary. Every source becomes the same immutable target scene, so hints, safe points, actions, and stale-work rejection behave consistently. Read the source contract.
Pixel
The local model finds visible controls directly from captured pixels.
Semantic
Native accessibility providers expose controls the operating system already understands.
Fused
Frame-matched visual and semantic targets merge into one deduplicated scene.
Grid
A deterministic fallback reaches any point when no detector owns the right answer.
Action surface
More than a faster click.
Single, Dual, Multi, and Path selection feed one native action planner. Physical input, focus changes, permission loss, and stale scenes can cancel the plan before it reaches the operating system. Read the planner.
Move and hover
Target a safe point or place the pointer directly.
Click and repeat
Left, right, middle, double, modifiers, Multi, and Path.
Hold and release
Physical action state guarantees bounded cleanup.
Drag and drop
Two targets define source and destination.
Scroll
Vertical or horizontal, stepped or continuous.
Select text
Two safe points become a native range selection.
Navigate windows
Geometry, overlap, z-order, and visible-window hints.
Type
Bounded Unicode text through the validated input path.
Native path
Pixels stay on the accelerator until targets are compact.
ScreenCaptureKit or Windows Graphics Capture supplies native surfaces. Owned Metal and D3D12 kernels prepare the tensor and reduce model output before a bounded target packet crosses into CPU-visible scene state. Follow the buffer ownership.
Native capture
Leased platform surface plus frame and topology epochs.
GPU prepare
Crop, resize, channel map, normalize.
Local model
Core ML or Windows ML on available compute.
GPU reduce
Decode, rank, suppress, pack targets.
Target scene
Accessibility fusion and immutable publication.
Act
Hint overlay or validated native input plan.
No screenshot history. No cloud inference. No per-frame heap protocol. Superseded work is replaced by the newest unstarted frame.
Local target protocol
Capture, model, accessibility, and grid providers publish bounded target records with geometry, identity, capability bits, safe point, source, and frame, focus, transform, topology, permission, and session generations. Read the C11 records.
Compact hybrid merge
Visual and semantic records are matched by compatible generation context. The merge keeps the compact visual geometry, adds fresh semantic identity when available, and lets safety restrictions remove capabilities. Read the merge.
Resolve v0
Resolve v0 selects a candidate, preserves the planner coordinate, or abstains. A selected target reaches native input only after the action path checks its current generations. Optional local visual text must echo eight frame fields, then pass a byte-identical screenshot, app, process, display, and scope bracket.
Implementation status
Running code and open measurements.
Implemented, tested, measured, and open describe different states. The attended ordinary-application workflow has not run yet.
macOS neural path
measured29.973 Hz full-scope cadence and 18.643 ms p95 on M5 Max, with no measured deadline misses.
benchmarkWindows neural path
testedCurrent source passes a 300-frame D3D12 synthetic soak at 29.954 Hz. Live Windows Graphics Capture and high-refresh presentation remain unqualified.
benchmarkMetal 4 presentation
measured119.810 Hz for 1,198 of 1,198 display-backed frames, with zero deadline or busy misses.
benchmarkAccessibility fusion
implementedThe bounded frame-matched merge is implemented. The 90% intended-target gate has not been measured.
sourceNative actions
implementedPlanner, epoch preflight, cleanup, CGEvent, and SendInput paths exist. The attended cross-platform action matrix has not run.
sourceLive fused coverage
targetCompact hybrid merge is implemented. No live intended-target result exists yet.
Agent task lift
targetThe local target protocol is implemented. Paired agent task results are still open.
Release signing
targetArtifact fields are versioned. Production signing has not run.
Two clocks
Seeing and responding do not share a deadline.
Neural work is asynchronous and bounded by the selected screen scope. Interaction and overlay state continue independently, so a model pass cannot stall key reduction or pointer feedback.
Capture, preprocessing, inference, postprocessing, and scene publication run on bounded newest-frame work. A passing live Windows cadence run is still open.
Metal presentation reached 119.810 Hz in a display-backed component run. Attended interaction at that rate has not run.
Boundaries
The fast path is useful because its ownership is explicit.
Saccade is C++20 behind versioned C11 headers. Capture, inference, accessibility, overlay, input, and agent services keep separate provider contracts and bounded handoff points. Read the ABI.
Thread-owned state
Static storage, fixed buffers, newest-frame mailboxes, and rare cold-path coordination keep hot loops away from shared ownership. Read the ownership model.
Generation-safe actions
Frame, focus, transform, topology, permission, and session epochs travel with targets and are checked again before input. Read the preflight.
Local agent surface
Structured target generations and bounded action batches reuse the same validation path without putting screenshots into the protocol. Read the protocol.
Engineering notes
Follow the mechanisms, not the feature list.
Each article follows one constraint through the representation and ownership decisions it forced. Measured results and open work stay separate.
Teaching a smaller model to see actionable controls
How offline teachers, human truth, a stride-four student, and deployment qualification fit together.
Read the model path → SYSTEM TOURA request through Saccade
One activation from native capture to a safe-point action, with every epoch and ownership boundary intact.
Read the tour → SCHEDULINGThirty frames of vision, one hundred and twenty frames of interaction
Why two clocks, newest-frame replacement, and fixed in-flight storage are one design decision.
Read the scheduler → GPU RENDERINGTen thousand targets without ten thousand draw calls
Compact target records, GPU expansion, epoch-only static work, and one indirect draw.
Read the renderer →