John August

GPTZero is a new service created by Edward Tian designed to sniff out AI-generated text.

I’m dubious, in part because I’m not sure I could reliably spot the differences between sentences assembled by humans versus machines. But maybe that’s just my human bias.

As an experiment, I decided to run a quick, not-at-all-scientific test. I asked ChatGPT to “write three paragraphs explaining what a manager does for screenwriters.” I also looked up the answer for “What does a manager do?” on screenwriting.io.

Here are the two responses, one written by a human¹, the other by an AI. See if you can spot which is which.

SAMPLE A

In brief, a manager manages your career.

This includes tangible acts, like setting up meetings, giving notes and feedback, and assisting with deal negotiation, as well as the more abstract, like helping steer your career’s direction.

Managers, unlike agents, are not required to be licensed by the state. They usually have fewer clients than agents, and often take more personal and less business-focused roles in their clients’ careers. Managers, are allowed to take producer credits on their clients movies, which agents are not.

Like agents, a manager typically receives 10% of a client’s earnings. A manager should never ask for an upfront or flat fee.

SAMPLE B

A manager for a screenwriter plays a vital role in helping the writer navigate the film and television industry. They can assist with finding and developing new ideas, as well as helping the writer shape their existing ideas into marketable concepts. This often involves providing feedback and suggestions to help improve the story and make it more appealing to potential buyers.

Managers also help screenwriters with the business side of things, such as identifying potential buyers and pitching the writer’s work to them. They can also help negotiate deals and contracts, as well as manage any legal issues that may arise. In addition, they can help the writer navigate the many different players in the industry, such as agents, producers, and studio executives.

Finally, a manager can also help screenwriters with their long-term career goals by helping them identify opportunities for growth and advancement. This might include working on larger projects, transitioning to other forms of media, or developing a personal brand to increase their visibility and credibility in the industry. Overall, a manager plays a critical role in helping a screenwriter achieve success in the competitive film and television industry.

For each answer, GPTZero measured the text’s Perplexity — the randomness of the text — across several ranges.

	Total	Average	Highest
Sample A	41	136	219
Sample B	11	36	49

That 219 is an example of “Burstiness,” which GPTZero defines as “spikes in perplexity.” For Sample A, the opening sentence has a perplexity of 219, whereas highest perplexity in Sample B is just 49.

That’s the tell, apparently.

Some human written sentences can have low perplexities, but there is bound to be spikes in perplexity as the human continues writing. Contrastingly, perplexity is uniformly distributed and constantly low for machine generated texts.

After crunching the numbers, GPTZero renders its verdict.

SAMPLE A (score 66.44): Your text is likely human generated!

SAMPLE B (score 9.93): Your text is most likely to be AI generated!

On both counts, GPTZero got it correct.

But how hard would it be to fool the system? I spent two minutes editing the AI-generated version into the following:

Sample C

Managers can play a vital role for screenwriters. Basically, they help the writer navigate the film and television industry. They can assist with finding and developing new ideas, or shaping their existing ideas. They provide feedback and suggestions.

Managers also help screenwriters with the business side of things: who the buyers are and how to pitch to them. They can help negotiate deals and contracts, as well as manage any legal issues that may arise. They help the writer navigate the many different players in the industry, including as agents, producers, and studio executives.

Good managers think about screenwriters’ long-term career goals. This might include working on bigger (or smaller!) projects, transitioning to other forms of media, or developing a personal brand. Overall, a manager can play an important role in helping a screenwriter achieve success in the competitive film and television industry.

	Total	Average	Highest
Sample A	41	136	219
Sample B	11	36	49
Sample C	19	106	439

The verdict:

SAMPLE C (score 118.77): Your text is likely human generated!

Some takeaways

• The ChatGPT answer was well-written and potentially helpful for someone who didn’t know anything about managers.
• GPTZero is looking for patterns a human likely wouldn’t notice, which makes sense. But an AI model trained to provide responses with high perplexity and burstiness would likely evade detection. It’s interesting to see this arms race play out, because adversarial networks are already a fundamental tool in AI.
• It’s currently very easy to fool GPTZero with just minimal rewriting. If this tool meant to detect AI-generated term papers, students will quickly learn how to evade it.

1. Stuart Friedel wrote most of the early screenwriting.io answers. ↩