Hit Me With Your Best Prompt

Let’s start out with an opinion: the idea that AI might replace human creative endeavors such as writing novels or movies is pretty dismaying. I’ve seen quite a few posts about this on the web in the past year, much of it clickbait, but there are certainly some serious attempts out there to trade human creators for AI, and justifiable concern from people in many industries.

In the future I’ll probably talk about my mostly negative reaction to this concept, but today I’m going to share a practical experiment, and hopefully a bit of reassurance. In a nutshell, I’ll contend that unless your literary tastes have a significantly low bar for quality, you don’t have to worry too much about AI taking over the bookshelves or Hollywood anytime soon.

What prompted this experiment? First, a post I read on MSN. Some guy claimed to be writing hundreds of novels per year using generative AI. He described his “work” as incredibly voluminous and world-rich; apparently his favorite AI was just churning out massive amounts of characters, scenes, plot lines, place descriptions, and writing entire stories, which the human agent then edited and published at an amazing rate.

Like I said, I certainly have some opinions on that, but they can come later. And you can probably read between the lines to figure out where I generally stand.

Second factor: I published a novel myself last year. (Go read it; you could probably use a break from LinkedIn posts and business articles, and I could use the Amazon reviews.) I made my first novel and short story sales around 2003, and I’ve been a semi-pro dabbler ever since. Last year I decided to pursue a ghost novel, and it hit the virtual shelves in November.

So, time for an experiment. First step, I gave ChatGPT the entire novel to read. That took about thirty seconds. Hopefully most of my readers will spend a little more time with it.

Next, I asked ChatGPT a number of very factual questions. What does this character do for a living? Where did these two characters first meet? How does the main character finally solve the major conflict? My AI fan did an excellent job of answering. Apparently it’s a very detail oriented reader.

From there I gave it another round of trickier questions. These required understanding of theme and some conjecture. For example, the novel has a very strong and persistent motif of something that’s factually incorrect in the setting. Of the human readers I’ve talked with, only about 30% noticed this anomaly. ChatGPT identified it right away, and was able to give six or seven really good guesses about why the anomaly exists.

At this point I was convinced that ChatGPT understands the novel I wrote, so let’s see if it can write another for me. Or at least a chapter.

Continuing the session, I gave ChatGPT a three paragraph summary of my intended sequel, and an additional, detailed paragraph of what I want to have happen in the first chapter of the sequel. Then I simply asked it to write the first draft of the chapter. Keep in mind, ChatGPT has a pretty good amount of source material to train on here; 120,000 words of the original novel, plus a bit of guidance.

The result was underwhelming, but not a complete disaster. The chapter my AI scribe came up with was actually a coherent story, of suitable length, and it incorporated the elements I laid out. Furthermore, it did sound a lot like my writing. It’s easy to see from this example why people would be concerned about AI replacing authors, screenwriters, etc.

However, the chapter simply wasn’t very good.

To really evaluate, I gave the chapter to three people who read the original novel and asked them what they thought of “my” start for the sequel. Some of their comments:

“This seems like your voice and style, but not really.”
“It’s not very good, honestly, it kinda feels like you phoned it in.”
“Did someone else write this while you were doing a puzzle?” (Ironically, yes.)

This is gratifying. At least a few of my readers believe I’m a better writer than one of the most expensive bits of software in the world. Could I edit ChatGPT’s attempt and saved time? Yes, if I want to publish some really low-quality work and hope for the best. Because here’s one last observation: the best snippets of Chapterish One turned out to be items that I explicitly told ChatGPT to include, not any of the material the AI originated. In this experiment, at least, the creative quality is still coming from the human brain, and this human brain is quite happy with that outcome.