Aisso

Saturday

ἀΐσσω, to dart · my Mac app, in research preview

Tell your Mac what to do

Aisso is my Mac app for running tasks across your apps from a typed command. Press Option-Space and ask for a checklist from an email or a batch file rename; it reads your window, aims at each button, and clicks through the task.

Swift on top, Python underneath

The menu-bar UI and overlays are 18,700 lines of SwiftUI and AppKit over a 29,500-line Python step engine. One tokens.json file generates both the Swift styles and a 14-component React kit, with drift tests that fail if the two diverge.

Send

0 ms

See every click before it lands

Before Aisso presses anything, a terracotta ring flies across the window and settles on the button it chose, so you see where it is about to click a split second before the click goes down.

Timed to the mouse-down pause

The Python engine pauses 40 ms between moving the pointer and pressing the mouse button. I gave the SwiftUI ring a 40 ms cubic ease-out flight so it arrives at the target the moment the click goes down, then holds for 250 ms and fades over 200 ms.

A bubble that never clicks a gap

Aisso aims with a bubble cursor that swells until it reaches the nearest button, so even a point in empty space lands inside one control.

One bubble rule for three jobs

Grossman and Balakrishnan’s CHI 2005 bubble cursor is implemented once as a pure function. Aisso uses that same math to aim its clicks at least 3 pt inside a control, to label which button you meant in Ichnos recordings, and to draw the live overlay.

Delete 12 files

84% sure

It stops when it isn’t sure

Every decision Aisso makes carries a confidence. When it drops below the line, Aisso refuses to guess and hands the task to a larger planner with what it already did.

Code extracts facts before Kev

Each step reads the accessibility tree and screen text first, keeping the pixels as ground truth. My Python code works out the active app, focused field, and open menus so Kev answers a bounded multiple-choice question, stopping below a 0.40 floor.

Plan once, then click right away

Asking a model before every click takes a median of 67 seconds across 16 Mac apps. Plan mode cuts a four-step command to about 18 seconds by planning once and clicking controls directly, on the way to my under-10-second target.

Two-part asks and plan mode

Asking the model everything in one call took 23 seconds for a single Calculator click. Splitting it into action type and target cut that step to 8 seconds, and plan mode plans once so it can click named controls with no model calls in between.

Recording on this Mac

It learns how you use your apps

Ichnos (ἴχνος, footprint) is Aisso’s built-in recorder. It logs your clicks, keys, and scrolls on your Mac while leaving password fields blank, building a local dataset to train a small model on the apps you use every day.

Three rejected Kev-0.8B fine-tunes

Training labels come from verified runs, with Safari, Mail, and Xcode held out for testing. I fine-tuned Kev-0.8B, the smallest of Jared Palmer’s open Kev models, three times and rejected all three after v6 collapsed to predicting “done” and v8 matched base.

Write to me

quinn@upaway.dev

Notes

What it is: I built Aisso to explore computer use that runs at the speed of the UI instead of waiting on a model before every click. Press Option-Space in any Mac app and type the task; Aisso reads the active window, aims with a bubble cursor, shows a terracotta ring before every click, and hands the task to a planner when it isn’t sure.
The name: From the Greek ἀΐσσω (aïssō), meaning to dart or shoot forward, the word Homer uses for birds and gods flashing through the air. Its recorder is Ichnos (ἴχνος), Greek for a footprint.
My part: I designed and built the Mac app (Swift, SwiftUI and AppKit, ~18,700 lines), the Python engine, the Core ML icon detector, the Ichnos recorder, plan mode, and the training pipeline, backed by 2,358 offline tests.
Credits: Kev, the decision model, is Jared Palmer’s open model. The bubble cursor is Grossman and Balakrishnan’s, from CHI 2005.
Where it is: All 16 everyday Mac apps in my suite pass at a 67-second median, and plan mode runs a four-step command in about 18 seconds on the way to my under-10-second goal. Ichnos records locally today, and I’m working on training a small Kev model after rejecting my first three fine-tunes (v6 to v8).
On this page: The windows above are mock-ups drawn for this page. The ring, its timing and the bubble cursor run Aisso’s own math, the task times were measured on my Mac, and the confidence numbers are examples.

Write to mequinn@upaway.dev

Next EOS AI