Target sources
One action surface, four ways to find it.
Detection is a source, not the product boundary. Every source becomes the same immutable target scene, so hints, safe points, actions, and stale-work rejection behave consistently.
Pixel
The local model finds visible controls directly from captured pixels.
Semantic
Native accessibility providers expose controls the operating system already understands.
Fused
Frame-matched visual and semantic targets merge into one deduplicated scene.
Grid
A deterministic fallback reaches any point when no detector owns the right answer.
Action surface
More than a faster click.
Single, Dual, Multi, and Path selection feed one native action planner. Physical input, focus changes, permission loss, and stale scenes can cancel the plan before it reaches the operating system.
Move and hover
Target a safe point or place the pointer directly.
Click and repeat
Left, right, middle, double, modifiers, Multi, and Path.
Hold and release
Physical action state guarantees bounded cleanup.
Drag and drop
Two targets define source and destination.
Scroll
Vertical or horizontal, stepped or continuous.
Select text
Two safe points become a native range selection.
Navigate windows
Geometry, overlap, z-order, and visible-window hints.
Type
Bounded Unicode text through the validated input path.
Native path
Pixels stay on the accelerator until targets are compact.
ScreenCaptureKit or Windows Graphics Capture supplies native surfaces. Owned Metal and D3D12 kernels prepare the tensor and reduce model output before a bounded target packet crosses into CPU-visible scene state.
Native capture
Leased platform surface plus frame and topology epochs.
GPU prepare
Crop, resize, channel map, normalize.
Local model
Core ML or Windows ML on available compute.
GPU reduce
Decode, rank, suppress, pack targets.
Target scene
Accessibility fusion and immutable publication.
Act
Hint overlay or validated native input plan.
No screenshot history. No cloud inference. No per-frame heap protocol. Superseded work is replaced rather than queued behind the present.
Two clocks
Seeing and responding do not share a deadline.
Neural work is asynchronous and bounded by the selected screen scope. Interaction and overlay state continue independently, so a model pass cannot stall key reduction or pointer feedback.
Capture, preprocessing, inference, postprocessing, and scene publication run on bounded newest-frame work.
Key reduction, prefix resolution, active-state updates, and display submission stay responsive between model results.
Boundaries
The fast path is useful because its ownership is explicit.
Saccade is C++20 behind versioned C11 headers. Capture, inference, accessibility, overlay, input, and agent services keep separate provider contracts and bounded handoff points.
Thread-owned state
Static storage, fixed buffers, newest-frame mailboxes, and rare cold-path coordination keep hot loops away from shared ownership.
Generation-safe actions
Frame, focus, transform, topology, permission, and session epochs travel with targets and are checked again before input.
Local agent surface
Structured target generations and bounded action batches reuse the same validation path without putting screenshots into the protocol.
Engineering notes
Follow the mechanisms, not the feature list.
Each article takes one constraint through the representation and ownership decisions it forced, including the parts that remain targets rather than published performance claims.
Teaching a smaller model to see actionable controls
How private teachers, human truth, a stride-four student, and deployment qualification fit together.
Read the model path → SYSTEM TOURA request through Saccade
One activation from native capture to a safe-point action, with every epoch and ownership boundary intact.
Read the tour → SCHEDULINGThirty frames of vision, one hundred and twenty frames of interaction
Why two clocks, newest-frame replacement, and fixed in-flight storage are one design decision.
Read the scheduler → GPU RENDERINGTen thousand targets without ten thousand draw calls
Compact target records, GPU expansion, epoch-only static work, and one indirect draw.
Read the renderer →