Category Archives: media and AI

assessing ChatGPT as student

I have started organizing my blog posts in view of a possible publication. This is involving re-reading and re-indexing. In this process I found a series of posts published in 2011 “on assessment” following a small conference at Teachers College that a preparatory gathering that was part of the large “Commission on the Future of Assessment” led by Ed Gordon. I do not have the full list of participants but they included Ray McDermott, Robbie McClintock, Aaron Hung, Gus Andrews. At the time, I wrote 5 posts and one article (Varenne 2013) that developed our perennial themes reflected in this blog: a continuing exploration of the tension between State and the individual in their local assemblies. In brief, one the one hand, in the world of schooling, the State determines what is the “right” answer to a question to which the teacher knows the answer where “right” means that the teacher authorizes the student to move on to the next step. In brief, local assemblies look for, and find, ways to bypass the State as they inscribe an alternate “next” as a plausible, if not always consequential, new “right-for-current-purposes.”

In the last post of the 2011 series, I played at imagining something that, then, appeared to be a flight of “science fiction.” I imagined a world where “Google” (standing of any kind of search “engine”) would keep in its memory our searches and would base next searches on the basis of these earlier ones thereby helping one find faster some obscure matter one was searching for. This “imagined” technology would analyze our interests and, perhaps, even tell us not to search further into disallowed topics as a kind of mentor.

Fourteen years later, much of this is not science fiction anymore. It is our everyday life. I would date the social facting of this brave new world with the explosion of ChatGPT in the Fall 2024—when students mentioned it, I explored it, and was told by my grandson that he had already used it to flesh out a story of his. What may still be missing in this world is a mentor.

Given ChatGPT’s claim to authoritative, “intelligent,” knowledge, it is fair to assess it as a professor would any student claiming the status. So, as stereotypical teacher in a stereotypical lesson, asking students question to which I know the answer and checking how close is the answer to being “right,” I asked ChatGPT: what is emile durkheim best known for? ChatGPT answered (excerpt, full answer):

Émile Durkheim (1858–1917) is best known as one of the founders of modern sociology and for establishing sociology as an academic discipline distinct from philosophy and psychology. His major contributions include:
1. Theory of Social Facts …
2. Division of Labor and Solidarity …
3. Study of Suicide …
4. Religion and Collective Consciousness …
5. Education and Morality…
In short: Durkheim is best known for demonstrating that society shapes individual behavior, for introducing systematic methods to study social phenomena, and for foundational works on social facts, solidarity, suicide, and religion.

This answer passes as “right” (B+ if given by a doctoral student in anthropology) though not particularly imaginative, or indicative that everyone of these statements was controversial. So, to probe whether my exam question was too vague, and knowing the answer, I asked: Which aspects of Durkhem did Talcott Parsons emphasize? ChatGPT answered (excerpt, full answer):

Talcott Parsons emphasized Émile Durkheim primarily as a theorist of normative order, social integration, and moral regulation, selectively reading Durkheim in ways that supported Parsons’s own project of structural functionalism. Here are the main aspects Parsons highlighted:
1. Social Order and Norms …
2. Action Governed by Values (Not Just Interests) …
3. Social Facts as Moral Constraints …
4. Integration as the Core Social Problem …
5. Religion as a Moral System …
Parsons tended to soften or neglect:
Durkheim’s attention to conflict and coercion
The emotional, ritual, and ecstatic dimensions of religion
Durkheim’s methodological radicalism (“treat social facts as things”)
In One Sentence:
Parsons emphasized Durkheim as a theorist of normative integration and moral order, transforming him into a cornerstone of consensus-oriented functionalism, while muting Durkheim’s sharper edges around power, conflict, and crisis.

This answer also passes as “right” (A- if given by a doctoral student in anthropology), though, again somewhat lacking in caveats. The answer may be deemed overly “declarative” by some assessors. So, to probe further into weeds most social scientists would not consider useful, I then asked a question I did not expect ChatGPT to answer as expertly (for the full answer go to): What are the main aspects of Garfinkel’s critique of Parsons’ take on Durkheim? ChatGPT answered, in 612 words of which I only quote the last (excerpt, full answer):

III. The fundamental divergence (in one contrast)
Parsons Garfinkel
Norms explain action Action produces norms
Values are internalized Accountability is displayed
Order is systemic Order is local and practical
Actor is norm-following Actor is methodically competent
Durkheim = moral consensus Durkheim = practical moral workBottom line: Garfinkel’s critique is not that Parsons misunderstood Durkheim entirely, but that he domesticated him. Garfinkel’s Durkheim: is not a theorist of value systems, but a theorist of how moral order is accomplished in practice.

This answer if fully right in my world and I would rate it a doctoral student “A.”

Except that I suddenly had a doubt: did ChatGPT know that I am the one who was asking the question? This came to me as I noticed that it exactly summarizes what I teach, publish, and blog. Much of this is open to everyone… These concluding sentences are what makes me wonder:

This reinterpretation profoundly reshaped sociology, influencing conversation analysis, studies of work, education (including McDermott & Varenne), and the micro-foundations of social order.
If you’d like, I can:
trace this argument directly through Studies in Ethnomethodology
connect this to McDermott’s classroom analyses as Durkheimian ethnomethodology

I felt spooked: CG has read our work and is quoting it back to us! However, and I did not dig further, it is, also, an interpretation: After all, for many years of his careers, and possibly onto now, McDermott would not have claimed Durkheim (about whom he may have learned through secondary discussion of Parsons’s take–the most common in the 1960s and 70s when we were doctoral students at Stanford and Chicago)—or might have considered him irrelevant.

In any event, the sequence of answer makes ChatGPT appear almost “intelligent” and altogther compliant. I do not have to worry that it will discuss with itself (as I know students do among themselves) about Varenne’s odd interests.

That leaves us with at least two set of further questions. Among the faculty at TC, as it appears is the case in all schools around the world, there is much emphasis on one type of “danger”: that our students, when asked an exam question will put this question to ChatGPT who will give an unimpeachable answer the students can then pass as their own. ChatGPT can now be used as glorified Cliff Notes!

However this danger is only a threat to our claim that only we, university professors, can properly assess student knowledge for any purpose of the State, leading to degrees with “rights and privileges thereunto attached” (as the diplomas, will, candidly, state). Much on this will be written (along with bemoaning “grade inflation) so I will focus again on my 2011 thought experiment:

What if ChatGPT gets used as a tool for education away from State control?

References

Varenne, Hervé, G. Andrews, A. Hung, and S. Wessler   2013     “Polities and politics of ongoing assessments: Evidence from video-gaming and blogging.” in Discourse 2.0. Edited by D. Tannen and A. Trester, Georgetown University Round Table on Language and Linguistics series. Washington, D.C: Georgetown University Press. pp. 27-46.

on rewriting the history of one’s blogging

I have a question of etiquette (?) that is actually related to Gus Andrews’ research.  She noticed a formatting error in my post on ‘LOL’.  I had been lazy about fixing it up.  Now that she mentioned it in the comment, if I were to fix it, then aspect of the comment would not make sense.  What should I do?

In other words: can I rewrite my own history to erase an error that was noticed, thereby making someone else’s post puzzling or even “incorrect”?

[I just did make the correction]

 

[print_link]