I use Talon to control my computer some of the time. It's mostly voice control, but it has so many other controls built in! One lets you use an eye tracker as a mouse. I thought this sounded like a neat interaction for other situations too. When I mentioned this to a friend, he suggested building a game with it.

First, though, I had to do some setup for us: get the eye tracker connected to a Rust game library.

Normal ways to use the eye tracker

There are a few ways that you can integrate eye tracker that are legitimate.

The Pro SDK would let us do what we want, but it doesn't the Tobii 5, and never did. This SDK only supports the more expensive research models (which are thousands of dollars instead of hundreds)¹. Since I have a Tobii 5, this is out.

Instead, maybe we can use the game integration which they provide! Unfortunately, I also use Linux, and the game integrations are for Windows only. These are built on top of the stream engine, though!

It looks like we can use their Stream Engine to get the data we need. This is nice. But it also means I'll be integrating C++ into my Rust project, and that doesn't sound like a good time. Maybe it will by the time I'm done here².

Can I copy from Talon?

Instead of figuring out how to use Stream Engine, I thought about trying to see how Talon does it. However, since Talon is closed source, there's not a lot of poking around I can do, so that's a dead end...

But is it? I can't seem to find which library they're integrating, though it seems likely to be Stream Engine itself. Instead, I looked to do some reversing to see if there's a way to see anything deeper. Talon is written in Python but the source is obfuscated, so it's hard to see.

It does seem to call into some C code, and that's where the investigation ended for me. I don't think it's a great idea for me to try to poke any deeper, since this is commercial software that's intended to be closed, and we should honor the intentions of the author.

Instead of copying from them, let's just make a sidecar and have Talon do things it was never intended for!

Talon gets a sidecar

One of the beautiful things about Talon is that it's extremely configurable and customizable. Out of the box, it is quite bare, which is why using the community command set is highly recommended to start. In many ways, it's more a workshop for making your own tools, than it is a tool itself. This gives us a lot of room to hook into Talon!

The first thing is to add a folder into our talon user directory, which for me is at ~/.talon/user/. I added one called eyedaemon. Then, inside that folder, you can write Python code, and Talon will load it!

Talon supports hot reloads for plugins, which makes it very easy to iterate on this code. In my code, I just imported Talon's eye tracker module and made an instance. Then I iterate over the attached USB devices until I find an eye tracker, which I stash into a global variable to use later on.

from talon import usb
from talon.track import tobii

tracker = None

for dev in usb.devices():
    if isinstance(dev, tobii.TobiiEC):
        if not tracker:
            tracker = dev

After that, we write a handler to take the on_gaze event. (I told you Talon gives us a lot of convenient ways to hook in.) This will just take the frame and store it. Specifically, we'll store this into a single-element list for multi-threading reasons that we'll see later.

gaze = [None]

def on_gaze(frame: tobii.GazeFrame) -> None:
    global gaze
    gaze[0] = frame.gaze

tracker.register('gaze', on_gaze)

After this, we just need to expose this on a web server! We can use Python's built-in http.server library for this—we don't really need to harden it, since this is just a local prototype. The server is really basic: it just returns the gaze point (0 to 1 for each point, from top-left to bottom-right of your display).

class RequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()

        data = (gaze[0].x, gaze[0].y)
        body = json.dumps(data)
        self.wfile.write(body.encode("utf-8"))

def run_server(port=8080):
    server_address = ("localhost", port)
    httpd = HTTPServer(server_address, RequestHandler)
    httpd.serve_forever()

When we run this via run_server(), we run into a snag: since it never terminates, Talon thinks the module stalled. We want this, but Talon might get grumpy. So let's put it in a thread.

t = threading.Thread(target=run_server)
t.gaze = gaze
t.start()

And then in our handler, where we get gaze[0], we'll have to get it from the thread instead since this module variable will be different on each load.

        gaze = threading.current_thread().gaze[0]
        data = (gaze.x, gaze.y)

Now we fixed that problem but we effectively lost hot reloads, since the thread lives on past the reload and our port is already in use! We need to stop the http server and its thread on each reload. I got tired of restarting Talon, so I figured out a delightfully hacky feeling solution.

First, we make sure that we name the thread so we can identify it later. Then we iterate through all the threads before starting a new one, and when we find it, we can do our stopping! To do that shutdown, you have to have access to the http server that's running, so we'll update a couple of other places.

for t in threading.enumerate():
    print(t, t.name)
    if t.name == GAZE_THREAD_NAME:
        t.httpd.shutdown()
        t.httpd.server_close()
        t.join()

Then after we create our thread, we make sure to create the httpd field.

t = threading.Thread(target=run_server)
t.name = GAZE_THREAD_NAME
t.httpd = None
t.gaze = gaze
t.start()

And inside run_server, we also add a couple of lines to save httpd to the current thread:

    t = threading.current_thread()
    t.httpd = httpd

And after that, we put it all together and we have a hot-reloading http server running inside Talon!

Sneak peek

I got it integrated into a Rust program on the read side. Here's what that looks like right now.

The red dot follows my gaze to wherever I look! There's some jitter, but it does follow my eyes pretty accurately, and works best in full screen. Recording an eye tracker, or debugging it, is pretty funny because you can't look at the controls to trigger things or stop them. And when you try to read print statements to see where the current measured point is, it changes to be exactly the area you're looking at!

Hopefully there'll be more updates on this project after we do our one-day game jam. And hopefully this hack is a solid enough foundation for that.