I use Talon to control my computer some of the time. It's mostly voice control, but it has so many other controls built in! One lets you use an eye tracker as a mouse. I thought this sounded like a neat interaction for other situations too. When I mentioned this to a friend, he suggested building a game concept around controlling the game with the eye tracker.

First, though, I had to do some setup for us: get the eye tracker connected to a Rust game library. Ultimately, we wound up using pure mouse control—then you don't need an eye tracker, and if you have one, you can just use Talon to control it—but along the way there were some interesting shenanigans with threads!

How do I use an eye tracker?

It was really tough to figure out how to use my eye tracker, because I'm on Linux! The SDKs available mostly don't support Linux, or they're C++ and gnarly and I didn't have time to figure out how to do calibration. I do have Talon setup for eye tracking, so I thought it could be a neat way to interact with things.

Talon seems to call into another library, but it wasn't obvious what it was. I was hoping it would be obvious and I could go use that library, too, but alas. That's where the investigation ended for me. I don't think it's a great idea for me to try to poke any deeper, since this is commercial software that's intended to be closed, and we should honor the intentions of the author.

Instead, I tried using a sidecar for Talon to use the eye tracker! This was an interesting diversion, though ultimately one that we shouldn't really indulge in: it's probably against Talon's license, and it's certainly against the spirit of both Talon and the eye trackers' licenses.

That said, adding modules to Talon is a really valuable thing I tried doing for this! I've abandoned that approach here, but it was useful to explore in a throwaway project so I can use it when I want to actually expand my tooling.

Talon user modules

One of the beautiful things about Talon is that it's extremely configurable and customizable. Out of the box, it is quite bare, which is why using the community command set is highly recommended to start. In many ways, it's more a workshop for making your own tools, than it is a tool itself. This gives us a lot of room to build tools within Talon!

The first thing I did is created a folder in my talon user directory (for me, at ~/.talon/user). Then if you add Python code inside that folder, Talon loads it! This is really neat for building useful accessibility tooling.

Talon hot reloads plugins, which makes it very easy to iterate on this code. When I was exploring, my code created a thread. This thread was problematic, because it wouldn't be stopped when the module was reloaded, so if it held a resource (for example, write access on a file), you'd end up with conflicts. So I needed a way to stop the thread when we reload it!

Stopping an old thread

To stop our thread, we really ultimately needed a way to access the previous thread after we reload. We can't do that by just keeping it in a global in the module, since those are new on each reload.

I found there's a neat way to do that. We name the thread when we make it! Then we can find it by its name later on.

So I did something like this:

t = threading.Thread(target=myfunction)
t.name = "my-thread"
t.data = None

And then inside the function, we can assign to our data:

def myfunction():
    t = threading.current_thread()
    t.data = # some object we're using

Now the next time we reload the module, we can do some cleanup at the top!

for t in threading.enumerate():
    print(t, t.name)
    if t.name == "my-thread":
        t.data.shutdown()
        t.join()

And there we have it, we've stopped the previous thread.

Sneak peek

I got it integrated into a Rust program on the read side. Ultimately, we did this by just... using everything as intended, and using the mouse as input. Since Talon gives us our eye tracker as a mouse, this works for folks with that setup—and it also works if you don't!

It works, and I can render a red dot wherever I look! There's some jitter, but it does follow my eyes pretty accurately, and works best in full screen. Trying to screen record when you're using an eye tracker as your mouse, or debugging it, is pretty funny because you can't look at the controls to trigger things or stop them. And good luck clicking on the screen recorder.

Hopefully there'll be more updates on this project after we do our one-day game jam. And hopefully eventually we'll have a demo that folks can try out, with Talon or a mouse!