How to EVALUATE what ChatGPT’s writing
LLMs hallucinate. We know this. So, to go along with my guide on writing PROMPTS for ChatGPT, here is a first take on how to EVALUATE the output.
EVALUATE:
Explore, Verify, Analyze, Look up, Understand, Ask, Teach
Explore the output. Read the generated content with an open mind.
Verify all facts because hallucination is real. LLMs are starting to add search and sourcing, but don’t be fooled. That’s not their strong suit.
Analyze the internal logic because ChatGPT is stochastic not structured. In other words, ChatGPT is not designed to create writing that follows, say, a thesis. It is not designed to match the output to a main idea or to follow some plan. Any unity you perceive is the result of your input and patterns drawn from the writing in its model.
Look up any references because they may not exist
Understand how the system works (and how that system leads to errors). If you understand the basic ideas of predictive text generation, that the system is putting the next likely phrase based on the prompt and what it has been trained on, you will know better what to expect from output.
Adjust your prompt. Using LLMs is an iterative process. Perhaps you need to adjust your rubric or add additional constraints or context.
Teach it to produce better. Give it feedback on what it has produced. Note: you may have to chain your prompts, meaning when you give it feedback also give it the output back again.
Enhance the text with your own writing. At the end of the day, an LLM is a tool that can be part of your writing process not a replacement for it. Treat the content like starting material rather than the last word.
What would you change or tweak on this?
For further adventures with ChatGPT, check out Hallucinate This! an authoritized autobotography of ChatGPT.
