I thought it would be fun to see if an LLM could play Baldurs Gate 3, so I made this script to run on a computer that can SSH into a Steamdeck where the game is installed (and stream via OBS to twitch). I noticed that the LLM has to be pretty big to be able to even play at all – anything below Claude Haiku is pretty much shit at this. And the bigger ones usually makes it too expensive to run (even though the screenshot is scaled down to save on tokens) – so lots of optimizations have to be made for that to be worth it. It’s an ongoing project. You can see the results on my Twitch or Youtube