Pointer Engineering: Deepmind wants to turn the mouse cursor into the key variable in context engineering.
"Because a typical AI tool lives in its own window, users need to drag their world into it. We want the opposite," write Deepmind researchers Adrien Baranes and Rob Marchant. Instead of relying on precise prompts, a Gemini-powered pointer is meant to capture the visual and semantic context around the cursor. Users could then work with shorthand commands like "Fix this" or "Move that here," combined with speech and gestures.
According to Deepmind, pixels turn into "structured entities" such as places, dates, or objects. A handwritten note becomes an interactive to-do list, a paused video frame a booking link. The principles are already being woven into Gemini in Chrome, where users can select parts of a webpage and ask questions about them directly. On the upcoming Googlebook, the feature will launch as "Magic Pointer."
This won't replace prompt engineering, since more complex tasks still require precise descriptions. Other tools rely on screenshots or hand-drawn markers, like red arrows in image and video editors, that serve as visual anchors for detailed prompts. Deepmind's approach mostly streamlines short, chat-like everyday interactions by reducing the effort needed to hand context over to the model.