GrokEye started with a direct question: could an assistant help with something in your hands without making you put it down? Pointing at the right connector felt more useful than another paragraph describing where the connector might be.
I built it with Derek and Dan at xAI's August 2026 Grokathon, where we placed in the top six. Most of the implementation landed on August 8; the demo and README followed the next day. The result was a browser app that could watch a camera or a prerecorded clip, answer aloud, highlight objects, show instructions, and review a step.
The hard part was agreeing about what was happening now. A good answer delivered after the object moves is a bad interaction. My work concentrated on intent routing, grounding, tracking, and connecting a voice turn to the video.
What we built
A video player holds object boxes, connection arrows, a manual, and a tools panel. You can ask where a part connects or how you did on a step. Webcam mode provides a live source; the presentation mostly used a small catalog of known clips.
React and Vite run the client. An Express server calls the model and speech services. The checked-in configuration uses grok-4.5 for answers, visual grounding, and several review paths, with xAI's Carina voice for speech output. Speech recognition comes from the browser's Web Speech API. There is no separate speech-recognition API key in the app; that does not mean recognition is necessarily performed locally.
The components divide the problem along practical boundaries:
| Component | Responsibility | Why it is separate |
|---|---|---|
| Voice listener and router | Turn speech into an action | A request for the next page does not need visual reasoning |
| Answer endpoint | Produce a short spoken explanation | Speech should not wait for every overlay |
| Grounding cascade | Locate the requested object or endpoints | Coordinates need a different output contract from prose |
| Browser tracker | Move the last box with the video | Calling a large model for every frame would be impractical |
| Manual and review paths | Supply instructions and assess a step | Describing an action and checking it are different jobs |
One voice turn, several clocks
Ordinary speech starts a request without a wake word. At speech onset, the client captures a frame and its media timestamp. After about 1.6 seconds of silence, it submits the utterance. Capturing at the beginning matters: by the time someone finishes saying “where does this go?”, their hand may have left the frame.
The router first checks whether the utterance means a local action, a manual request, a visual question, a web query, or a review. It is mostly rules and regular expressions. The command space was small, and mistakes were easy to reproduce. “Next step” should advance the current instructions, not initiate another open-ended conversation.
For a typical highlight question, the spoken answer and the label request run in parallel. The label path returns structured coordinates; the answer path returns prose for speech synthesis. Whichever finishes first can begin contributing to the interface. We were trying to overlap useful work instead of waiting for one large response containing everything.
Retrospective questions get different visual context. The frame selector can add the frame nearest roughly two seconds before speech onset and the highest-motion frame in the recent history. It keeps at most three distinct views, ordered in time, and excludes frames after the question began. A normal “where is it?” question can stay with one image. “Did I just do that correctly?” needs evidence of the action, not only the resting pose afterward.
Interrupting speech was another small but important detail. While the assistant speaks, a wake-phrase listener can stop it. The recognition code accepts several spellings of “Grok,” including “grock,” “greg,” and “brook.” The ordinary listener and interrupt listener have to hand control back correctly; simply mounting both does not give reliable simultaneous recognition.
Making visual grounding fast enough
The first direct vision calls were too slow for the interaction. The August 8 commit notes record roughly 15–40 seconds at default reasoning effort, with 600–1,500 reasoning tokens spent on a box request. Lowering the effort often produced a response around two seconds, but slow calls still took roughly 9–17 seconds. These are development observations from that day, not a controlled latency benchmark.
We addressed this in two ways. First, the client tries cheaper methods before asking Grok. A small HSV color detector can find salmon-colored regions in the sushi clip. An optional local Python detector supports YOLO and YOLO-World packs. The multimodal model handles requests the earlier tiers cannot satisfy. The color detector is tuned to that situation; it is not a general detector for arbitrary objects.
Second, the server sends three identical label requests concurrently. It accepts successfully parsed results, waits at most another 450 milliseconds for agreement, and aborts the remaining requests. When two results have compatible structure and overlapping boxes, their coordinates can be averaged. The implementation requires matching label counts, link structure, box kinds, and an intersection-over-union of at least 0.25 for a pair of boxes to be fused. Disagreement leaves the earlier box alone.
The crucial correction was making a parse failure lose the race. The fastest response is not useful if it contains unusable JSON. The label-arm tests cover that distinction, along with the grace period and fusion behavior. The change and its measurements are in the grounding commit.
| Change | Benefit | Cost or remaining limitation |
|---|---|---|
| Low reasoning effort | Reduced the observed wait for simple localization | Response time still varied substantially |
| Color and local detector tiers | Avoided a remote grounding request when a cheap method matched | Narrow coverage; a wrong early match could pre-empt a better answer |
| Three concurrent label requests | Reduced reliance on a single slow response | More upstream work and potentially higher cost |
| Short agreement window | Allowed some coordinate noise to be averaged | Added a bounded wait; agreement does not prove correctness |
We did not measure a production cost curve, and aborting a request does not establish what the provider already billed. Hedging was a hackathon tradeoff.
Keeping a box attached to the video
Grounding tells us where something is in one frame. Tracking tries to keep that answer useful while the video moves.
The tracker is custom TypeScript using canvas pixels, without OpenCV in the browser. It works on frames downsampled to 320 pixels wide. Each object keeps a 40-by-40 grayscale template, an edge template, and an RGB histogram. A small grid of block matches estimates recent motion, and its median displacement predicts where to search next.
A strong grayscale normalized cross-correlation score is the fast path. Above 0.62, the tracker can skip the more expensive comparison. Otherwise, it combines grayscale similarity, edge similarity, and color-histogram similarity with weights of one-half, one-quarter, and one-quarter. A small scale search handles an object moving toward or away from the camera.
Texture, edges, and color each fail under different conditions. Combining them helped on our clips, but did not give the tracker an understanding of object identity.
The display smooths position more quickly than size: each accepted update moves 60% toward the new position and 15% toward the new dimensions. The processing loop targets roughly 24 frames per second when confident and 16 when struggling. These are implementation targets, not measured throughput guarantees across devices.
We also tighten overly generous model boxes using edge energy before initializing a track. A box containing mostly background gives the tracker the wrong template. And there is a specific salmon assist: after misses, a color centroid can nudge a fish track toward the pink region. It helped the sushi demo; it should not be mistaken for a general recovery strategy.
What changed during the day
The repository history captures decisions that the final demo hides:
| Stage on August 8 | Change | What prompted it |
|---|---|---|
| Morning | Scaffold, salmon detection, early tracking | Establish the basic see-and-point loop |
| Around midday | Parallel labels, typed annotations, connection arrows | Separate geometry from the spoken answer |
| Early afternoon | Remove freezing from ordinary highlight turns | A paused frame interrupted the task and could catch motion blur |
| Afternoon | Tightening, routing tests, temporal context, verification | Improve what the system was pointing at and remembering |
| Evening | Three-arm grounding, authored demo refinements, voice watchdogs | Make the interaction more predictable under presentation conditions |
The freeze change is worth separating from a later exception. Ordinary highlights keep the video moving and seed the tracker from the captured frame. Authored motion cues deliberately pause and snap to a traced frame. Those are different paths serving different purposes.
The last voice fixes were less glamorous. Backgrounded tabs could delay animation-frame work, and stalled audio could strand a turn. A timeout watchdog and playback completion independent of foreground rendering helped return the interface to listening.
Checking work is harder than describing it
Step review exposed a problem that faster responses alone could not fix. An early implementation confused the espresso knock box with the portafilter and described an empty basket that the supplied images did not support.
Dan tried a two-stage observe-then-judge pass. The follow-up commit records that it took 20–40 seconds, then describes collapsing the process into one call. The output schema still asks for a brief observation for each frame before the verdict. It also separates the footage's recorded action from the correct procedure. A script saying someone locked in an untamped portafilter should not make that action the grading standard.
The commit reports an average of 3.6 seconds and a maximum of 5.0 seconds for the revised checks. The repository does not supply a broad evaluation set or enough measurement detail to treat those as general performance numbers. The frame-first format is useful discipline, but a model can still write a wrong observation before a wrong verdict. The step-review revision documents the tradeoff.
There are checks before inference too: a review with fewer than three distinct frame images returns an insufficient-visibility result. This catches a capture pipeline that failed to seek, rather than asking the model to invent a temporal sequence from repeated images. The next-day change added spoken narration to the review panel, so the final behavior differs from the evening commit's panel-only version.
What was live and what was authored
Some of the most polished moments were rehearsed. The repository documents this, and it belongs in the write-up too.
Known clip-and-time windows can select hand-traced silhouettes and destination shapes. The player snaps to the traced frame, then animates the motion. Several windows were widened or retimed around the rehearsed question. That geometry is authored scene knowledge, not a model inferring a precise movement from an unseen video.
The espresso “check my work” shortcut is even more explicit: it matches the playhead to a known timeline and returns the skipped-tamp or corrected-state answer without a vision call. The separate step-review endpoint does inspect frames against the procedure. The manual system is also mixed: it can retrieve web instructions, while known clips have baked scripts and the IKEA guide uses a hosted PDF.
The demo demonstrates a combination of live inference, local tracking, and authored presentation. It does not establish reliable coaching for arbitrary physical work. A convincing overlay also cannot establish hidden properties such as torque, temperature, or whether a connection is electrically safe.
What I would carry forward
I would keep the separation between answering, grounding, and tracking. Each has its own latency and failure modes. I would also keep the small routing layer and the willingness to return no box when the detector has no good match. A confidently wrong cheap tier makes the entire cascade worse.
The next useful experiment would be on unfamiliar clips: measure the time to the first usable cue, how often the requested object is correctly grounded, how long the track stays attached, and how often review declines when evidence is missing. I would run that with authored cues disabled and report it separately from the rehearsed experience.
For a day-long project, I am happy with what we made. Handling time, state, and failure was as important as the model call.
Sources and scope
This account follows the public repository at commit b4a9066, its August 8–9 history, and my original account of the hackathon. Team context and placement are personal recollections; implementation claims are grounded in the code. Latency figures are the team's contemporaneous observations, not new measurements.
The most useful source files are the grounding implementation, browser tracker, frame selection, choreography notes, and catalog verification. A later tracking branch exists, but its changes are not presented here as part of the merged hackathon build.