What do college students know about ChatGPT: a classroom activity
Can ChatGPT pass for a student? In “Computing Machinery and Intelligence,” Alan Turing asked this question. While Turing was proposing this challenge as a proof of concept, these days, especially for those teaching writing or using it for assessment, it is more a kind of fearful bleat of a fallen species.
Those headed into the 2023 academic year may wonder if computers will be generating the writing they evaluate. Of course, they could always use a computer to evaluate it (see CoachTutor).
But the bigger question is what do students know about LLMs and how might discussions of these tools help classes in writing and critical thinking.
This past summer, I found this exercise helpful. I call it Turing Tests.
Turing Tests:
- Ask students to answer a question in a limited number of sentences. I used a question about AI, but it is not required.
- Have the students answer the question and then have an LLM answer it. I wouldn’t ask the LLM to use sources. I might limit the exercise to one LLM, and even one stage of an LLM, like ChatGPT 3.5.
- Ask students to post one of the two answers in an online forum.
- As a class, read through the posts and have people try to determine which was bot written and which was hunan written.
You’d think this exercise would just show how good the LLM is at writing like a student, but I found the exercise revealed so much about what students know about:
- prompting LLMs
- voice and style
- generic writing
- unique writing
These ideas emerge not just from the writing but from the way the students read the writing.
I also found these exercises good jumping off points for discussions about LLMs and, more importantly, writing.
Anyway, this might not be an interesting exercise forever, but this semester, it works. This week, at least.
For more on ChatGPT, see my posts here and my new book Hallucinate This! an authoritized autobotography of ChatGPT.
Special thanks to Jeremy Douglass for helping me brainstorm this activity.
