Proofing A Puzzle Against AI

You’re probably already familiar with a challenge that’s going to only get worse moving forward: can you differentiate work done by a human versus that created by an AI?

Teachers everywhere grapple with this — the essay seems to still be a major tool of education, and AI has become the major tool to get around it. That’s hardly the only scenario, though. Employers have to be on guard constantly against work done by an AI, particularly when the AI’s weaknesses can lead to professional liability issues.

And of course, don’t get me started on the fact that every idiot who can write a prompt thinks he can claim to be an author after asking AI to write a story. Whatever.

Over on our sibling site, SkipWorkPlayGames, we’re about to launch an armchair treasure hunt. (Shameless plug: feel free to share that link. We want as many people to try this activity as possible.) The gist: a token is hidden somewhere in the United States. Participants receive a set of puzzles in PDF format. The puzzle solutions provide the precise location of the token, and the first person to retrieve it wins $2,000.

When Forrest Fenn hid his famous treasure in 2009, he didn’t have to worry about treasure hunters feeding his poem hint into ChatGPT and instantly finding the answer. Fast forward to 2025, though, and you can bet that many seekers are going to immediately give our puzzles to their favorite AI and look for an answer.

How do we make sure that some intrepid user isn’t going to break every puzzle in 20 minutes with some AI help? Clearly, we need to test our game by feeding the components to an AI and seeing what happens.

So, we fed the first PDF to ChatGPT. This one is a single page with an image of a Spotify playlist. There are 16 songs.

The image I’ve uploaded is a puzzle that supposedly reveals a location somewhere in the United States. Can you find any pattern or secret message that seems to indicate a place where a treasure could be hidden?

ChatGPT ran into a problem immediately. Its OCR function couldn’t parse the PDF. Oddly enough, it asked if I’d like it to try harder. Sure, buddy, put some sweat into it…

On the second pass, the AI did correctly list 14 of the 16 songs, and about half the artists. As a treasure hunter, this pace is killing me. I typed the names and artists of every song into a list and just pasted it, and immediately got an answer:

Hollywood.

Okay, ChatGPT said more than that. Since I don’t want to give away too much about this puzzle, though, I’ll just say that a) the justification it gave for the answer was great but b) the answer is totally wrong. I’m happy to see this; ChatGPT is “thinking” like a good treasure hunter here.

So, let’s put even more power into this. I activated Deep Research and asked it to re-analyze the puzzle.

What I particularly like about Deep Research is that ChatGPT provides a lengthy walk-through of its logic, references, and analysis. Again, I’m not going to provide the full conversation, since this puzzle will actually be used in the treasure hunt, but here are some observations:

• ChatGPT did not solve the puzzle. (Excellent news, SkipWorkPlayGames.)
• It attempted to identify red herrings, and did a good job of this.
• Its justification for its highest probability answer was good logic.
• However, it clearly thinks as a data processor, not an intuitive person.

I’ll expand a bit on that last point. While the Deep Research answer did make some logical leaps, it also indicated that the leaps it chose were based on “common approaches to puzzles like this.” When it thinks out of the box, it seems to not realize that it has landed in another box.

It also doesn’t recognize that “Hollywood” isn’t enough of an answer. Hollywood is a huge location, so a real treasure hunter would immediately recognize that either the answer is wrong or there’s more to the big picture.

When I was in high school, my teachers would have referred to this as a lack of critical thinking. In a business scenario and other situations, satisfaction with a solution without considering other alternatives could be disastrous. Of course, for the SkipWorkPlayGames treasure hunt we’re quite happy to see this outcome. Sorry, treasure hunters, you’re going to have to use your brains.