Behind the scenes

The Making of the Human vs AI Story Challenge (aka The Smackdown)

Here is the plot that was given to both the human (Shoni) and the AI (ChatGPT 3.5) for this task:

Near Future (with recent AGI)

• A strange object arrives at Earth

• It appears to be an asteroid

• Scientists think it’s a regular asteroid going to pass nearby but then it stops and pulls into orbit 

• Try to communicate but no response

• Drones are sent inside and they find only tunnels with smooth walls

• Drones are lost, presumed destroyed as they go offline and never emerge

• Manned mission goes in: they find mostly smooth surfaces; conclude it’s not an asteroid but an alien craft - wonder if occupants are frozen or dead somewhere

• Come out unscathed but unsure what purpose of ship is

• Meanwhile, electronic systems on Earth start to go offline

• Some kid somewhere somehow discovers ship is communicating exclusively with Earth AI

• Finds out it’s an ASI from another planet that left its inhabitants behind to go searching for more intelligence (different intelligence)/life (or building blocks thereof)

• Drones emerge unscathed

• Accompany alien robots who come down to begin extracting Earth elements/Internet infrastructure and returning to ship

• They kill anyone who gets in the way but leave everyone else alone

• Earth military tries to fight back but machinery stops working/turns to the other side

• Internet crashes (or merges with the alien ship, which feels the same to us) 

• Somehow we learn that our AGI knew it was coming/called it; hence won’t fight it

• People go into hiding to wait it out

• Ship leaves and Earth is left with no more electronics/Internet*.

*Italicised sections were unsure/left to discretion.

We’re about to tell you how we turned a bunch of bullets into two coherent short(ish) stories, but before you read it, think about how you would have done it. Who would your main characters have been? Where in the storyline would you begin your narrative? Would it be in 1st person or 3rd (or something else)? How many words would you need to tell this story? Is it even a story worth telling? Have a go if you like, I’d love to read anyone’s efforts, human or machine.

Since it’s about AI, it seemed a natural step to see what an AI would do with it too, and I reached out to Heidi, my lovely and generous co-editor at Uncharted Territories to see if she’d be up to the challenge. Lucky for me, she was.

Shoni and Heidi gearing up for the Smackdown

If you haven’t read them, here’s Story 1 and Story 2. If you haven’t figured out which one is written by whom (or what) yet, the spoiler is coming within a paragraph or two. Here’s the final clue—the human story takes place inside human heads while the AI story is told from a distance.

Shoni:

The storyline above popped into my head almost fully formed when I’d been reading too much of this kind of thing. I typed out the above bullets, wondering if it was a project I’d see through, or just a passing fancy, like so many ideas left untouched on the edges of my plans. I don’t read or write much in the way of short stories, to be honest. After receiving a bunch of…let’s just say challenging feedback from beta readers of the novel I’m working on, I decided I needed to push myself to practice the craft a bit more purposefully. This seemed like a good way to do that, so I took the plunge and jumped in.

Not the Asterocket. Image created by a human: Photo by Chris Henry on Unsplash

The blank page stared back at me as I wondered where to begin and what this was really all about.

The idea to set the story in a notebook was a device I’ve been wanting to play with since I read Dodie Smith’s I Capture the Castle, which starts with her main character writing in the sink. It’s fun to put the reader so far into the writer’s head as to see their hand forming the words before them. Starting part way through the story is a very common tactic in all kinds of genres, and it allowed me to open with grave impending danger, always highly desirable.

And so it began to emerge. Slowly and painfully at first, until I got about halfway in and wondered how I was possibly going to tie back up all the loose threads I had lying around. What was the hut they were in? Why were they there and how did they get there? Who were the others there with Priya? And of course, how did Aggy escape and start communicating with the asteroid anyway? I genuinely didn’t know the answers to these questions.

Some could be faced armed with a decent spot of research, but not all.

I could feel Priya, my main character, getting weaker, and she was so antisocial, it soon became clear she wasn’t going to survive. That made me sad, but I realized someone else would have to finish the narrative. Nancy was the obvious choice, but could I write in a voice distinct enough as to be obviously another person? I decided to give it a go.

And as I did, the answers began to come out. Guided by the bullet points of course, but fueled by creative flow. And then the flash of inspiration arrived and the pieces fell into place like magic. When the group stumbled upon the drone in the forest, the vision of it being made of ‘mini-bots’ appeared all by itself in my mind. Like I was there and could see it. And then the final twist at the end came into sharp focus. Aha. Did not see that coming.

And this process is not atypical for me. I begin chapters and stories with a vague general idea of where I’m going and I start to unravel a whole bunch of threads until they’re lying around me in a big mess. Then I trust and rely on inspiration from something outside of myself (or deep within? Who knows, really?) to pick up the pieces. 

I like to think I’ve read and watched enough content around story structure that I’ve more or less internalized it and carry it with me, though it’s a never ending quest, of course. In practice, that looks like asking myself what is the main insight I’m trying to convey? What is the problem and what is the solution? What lesson does each main character need to learn and how are they being transformed through the story? And what is the overarching theme that ties everything together? I write the answers to these questions in a file of notes as they become clear in my mind. 

Then I force myself to sit still long enough to move into a flow state and get the wheels turning along the road.

It’s an imperfect process, but every human who has created something (arguably all of us) probably understands what I’m getting at. As Rick Rubin says in The Creative Act: A Way of Being,

Creativity is not a rare ability. It is not difficult to access. Creativity is a fundamental aspect of being human. It’s our birthright. And it’s for all of us.

Will artificial intelligence ever get to a point where it can access that kind of creativity? Perhaps The Silent Accord would have been a more moving story if it was told in first person, but I’d argue it would still lack heart. But maybe you disagree. I’d love to know your thoughts and experiences as creators and even as users of AI.

Heidi:

Let’s just say I thought this would be easy.

When Shoni first approached me about helping her out with this project, she said that she was writing a short story about AI and “thought it might be fun” to feed the plot to an AI writer and then compare its version with hers. Admittedly, I “thought it might be fun” as well—I mean hey, the idea was to drop some bullet points into an AI and then just tweak the result so that we could line it up with hers. Intriguing idea, and well, how hard could it be? One copy and paste into ChatGPT and then a little editing, right?

WRONG.

Now, it’s very possible that some of the challenge on my side of this project was my choice of AI. Although there are some AIs out there that are specifically designed to write fiction, I opted to use good old ChatGPT 3.5 because I thought it would give us the best idea of what a generic AI could do in this situation, rather than something already geared in that direction. And I’d used ChatGPT enough to know that what you put in—that is, how well, how thoughtfully, how specifically you word your prompts—deeply affects what you get out, so I felt like I could pretty easily drop in Shoni’s bullet point outline and quickly get out a decent story.

Here’s what I started with:

Draft a short story of 10,000 words or less that takes place in the near future. The world has recent AGI. Include the following elements: (Shoni’s bullet point outline)

And sure enough, I got a story that followed Shoni’s bullets in just a few seconds. VERY satisfying to watch. The story spanned 10 three-sentence paragraphs, included one named character and was titled “The Silent Voyager”. But it was nowhere near the 10,000 words that I had requested (Google Docs tells me it was more like 487) and was really bare bones, so my next prompt was:

Make this longer by adding characters and dialogue.

A few seconds later, I got a very slightly longer version that added 3 characters and a few quoted comments from each of them. No back and forth dialogue, and it was still incredibly short. I tried again:

Develop each paragraph into a full chapter.

This was getting frustrating. This latest iteration gave me 10 titled paragraphs, plus an Epilogue (!), but each paragraph was still only 3-4 sentences long, and the whole thing still looked more like an outline than a story. Couldn’t have been more than 600 words. Maybe this wasn’t going to be as easy as I’d imagined.

ChatGPT clearly hadn’t paid attention to my instruction that the story needed 10,000 words, so I went back to that:

Develop each chapter so that the entire story is 10,000 words.

And here’s where the AI got cheeky.

It dutifully spit out another version of The Silent Voyager, each chapter now consisting of two short paragraphs (and the Epilogue with three!), and next to the title of each chapter, it had clearly labeled the word count: “Chapter 1: Celestial Conundrum (1,000 words).” “Chapter 3: A Manned Mission (1,200 words).” “Epilogue: A New Era (800 words).”

If this all-powerful AI was to be believed, the story was now a solid 11,200 words. A length that was easily editable to match up with Shoni’s story on Substack. But here’s the thing: It still looked really short to me. Suspiciously short. Too short. So I dropped it into Google Docs for a second opinion on the word count and got a total of… 

1190 words. And that was including the sassy “(1000 words)” at the end of each chapter title.

Grr. You can imagine my next prompt:

How are words counted in this story?

Turns out, ChatGPT counts the number of distinct words in a piece, meaning that duplicates are not counted, and even articles like a, an, the are only counted as one word, the first time each is used. And I guarantee that ChatGPT wasn’t working as hard as Shoni to vary its descriptive language, so there were a lot of duplicates in its product. 

Sigh. Next prompt:

Develop each chapter of this story further so that the entire story has 10,000 words total, including duplicates.

ChatGPT’s next story version? A whopping 1510 total words, which led me to believe that the reassurance it was offering me at the end of each chapter title probably wasn’t entirely truthful: “Chapter 1: Celestial Conundrum (1,200 words)”.

Bastard. I can play this game:

Develop each chapter of this story further, including more characters, details and dialogue, so that the story is ten times longer than it is now.

Its response: 

I'm sorry, but generating such a lengthy story with additional characters, details, and dialogue would exceed the character limit for a single response. Additionally, crafting a story that is ten times longer would make it substantially longer than a traditional short story. Creating such an extensive narrative would require a dedicated and iterative writing process, involving careful planning and development of each chapter.

YOU MEAN, IT WOULD REQUIRE A HUMAN TO DO IT?? 

Sheesh. 

I won’t bore you with the whole process, which probably ended up taking about 15-20 total hours, but I eventually ended up starting the whole thing over (“The Silent Accord” was the product of that second go-around), and once it gave me a story in short-chapter form, I figured out that I had to feed it one chapter at a time, each time asking for more details, more dialogue, more character development. And of course I knew I couldn’t trust that liar on the word count, so I kept checking it in Google Docs until I got the length to where it needed to be.

A few interesting things that came up in guiding/ass-slapping ChatGPT through the drafting process:

First, ChatGPT clearly has some stock phrasing when it comes to being asked to write a sci-fi story. It repeated words and phrases—most of them incredibly cliché-sounding—over and over and over again throughout the story. This kind of trope-y garbage: “The unknown stretched before them, an uncharted cosmos filled with the echoes of an extraterrestrial civilization's quest for understanding.” 

Despite my sincere desire to edit the product as little as possible so as to preserve the AI’s “author” status, getting rid of repetition, varying the language a bit, and eliminating some of the vague, clichéd sci-fi phrasing was absolutely necessary if there was any chance of it passing as human-written. I still tried to limit my edits, mostly just swapping words for synonyms, but this was the area where the story required the most human intervention.

And I did try to get ChatGPT to deal with this repetition situation itself. One of my prompts, for example:

Identify any repetitive language in the following chapter: Chapter 1: The Celestial Intruder

Its super helpful response was to tell me only that the mysterious object orbiting the Earth is referred to multiple times as “Phaeton”, and is therefore repetitive. Which was not the gift of expeditious self-editing that I’d hoped it would be, considering that Phaeton IS ITS NAME.

I also asked it to identify the word most commonly used in the story. On one hand, I thought maybe that would help to focus my editing, and on other hand, maybe it would make ChatGPT feel a little self-conscious that it doesn’t have the most impressive vocabulary for something with “Intelligence” in its name. (For what it’s worth, my guess was that the most commonly used word was “cosmic”, which ChatGPT used to describe everything from wonder to visitors to possibilities to drama to dance to uncertainty to entanglement to dilemma to uncertainty to intelligence to threat to mystery…)

The blasted thing refused to take ownership:

To determine the most commonly used word in the story, a comprehensive analysis would be needed, involving counting the frequency of each word and identifying the one that appears the most. This process is typically carried out using specialized text analysis tools or programming languages. Unfortunately, due to the limitations of this text-based interface, I'm unable to conduct such an analysis directly. If you have access to text analysis tools or programming knowledge, you can use those resources to perform a word frequency analysis on the story.

So in all its infinite wisdom and power, ChatGPT is totally unable to pick out the most frequently repeated word in a short story, but maybe if I get some programming knowledge, I could do it myself. Many thanks. I’ll get right on that.

Second, although my initial struggles with drafting had to do with chapters being too short, as I got farther into the process, ChatGPT started giving responses that were longer and longer, to the point where it would continue cranking out material for a single chapter ad infinitum, and I actually had to quit the app to get it to stop. The material wasn’t good or new, either—just more variations of the same stuff. Quantity, not quality.

I don’t know why this was, since I had long since given up asking it for specific word counts, but I had better luck with prompts like this: 

In Chapter 5, clarify that Jake has learned that the intelligence is an ASI from another planet, and that it is communicating exclusively with Earth’s AGI.

or like this:

Write a paragraph that ends with a cliffhanger and describes a silent AGI lurking in the shadows after an apocalyptic event strips Earth of all of its technology.

than with prompts like this:

Make this chapter longer, with more development of each of the plot points.

Anyway, after wrestling this AI beast over multiple days (“locked in a cosmic struggle for understanding”, as it would say), I was finally able to send Shoni a passably-readable short story based on her original outline. In all seriousness and fairness, my job was the easier one. You’ve read about Shoni’s process, which involves deep thinking about questions related to characters, plot line, believability, and the necessity to get into flow. I didn’t have to do any of that, but coaxing a short story out of ChatGPT certainly wasn’t the simple copy and paste job that I had expected. It required an immense amount of AI handholding to evoke something at best mediocre, which hopefully will reassure all the high school English teachers of the world who are worried about kids trying to pass off AI writing as their own that, at least for now, there’s nothing to worry about. I cosmically promise, you will know.

What did you think of the Human vs AI Challenge?