Category Archives: on assessment

On the political philoshophy of assessment

on assessment

On grades as statements: to whom?

1 Comment

Ray McDermott and Jean Lave once told me that they asked Rancière whether his writing influenced his teaching.  As they tell it, he looked surprised and answered something like “not at all!”  A reader of Successful Failure once asked me whether I still gave grades.  Besides stuttering, I said something like: “I am required to (by my university and New York State)” and/or “students would not accept my not giving them grades.”

Over my 40years+ at Teachers College ,I have also been part of several faculty-wide conversations about “grade inflation.”  These never went anywhere and, by now, I gather from various sources, only about three grades are given: A, A-, & B+.  Personally that is, mostly, what I do and it is not altogether different from distributing grades among A, B, & C, except that it limits, mostly again, student complaints.  And while I do not grade “on the curve,” I do get nervous when I find myself only giving A’s.

Now, of course, what is the point of giving differentiated grades?  More specifically, what difference does it make? to whom? and with what consequences?  Taking the “gift” of grade as a statement, who is the audience?

A grade is structurally in the position of the “assessment” moment in Mehan (and many others)’s model of the “lesson.”  The teacher sets a curriculum, asks students to do something related to “the class,” and then differentially assesses how well each individual students performs the task (“has learned” in the current authoritative language among accreditation agencies).  The grade then becomes a datum (actually just another word, in latin, for “gift”) to the student.  But a grade is also a gift to others besides the student—though not to everyone given various legal strictures about who may see a student’s grade (tracing who may see a grade when and for what purpose would actually be a way of revealing the structure of social reproduction).  These “others” may then legitimately mete various consequences that have nothing to do with the original class, e.g. they may give the student various privileges, including, at the high school, college or Masters level, admission to a further degree program.  Thus the grade that looks like a private communication between teacher and student, is also a coded statement to powers-that-be (admissions officers, funding agencies, accreditation bodies, etc.).  Which is why, of course, grades are a political issue and “grade inflation” a political problem (see also my post on Lake Wobegon).

What does all this have to do with “education”?  Little, I say, with many others.  In recent years, I have gotten to say that I translate my current designation as a “professor and advisor of graduate students” into a “masters of apprentices.”  In that perspective, I maintain that I give grades because I am required to do so but that they should only be taken as a statement about a progression and my potential willingness to work with the student as apprentice.  The grades I give are not about individual learning per se.  This “faction” (fact making that may constrain in some future) is easier to maintain at the doctoral level where it is actually the case that one receives a doctorate not by accumulating grades but by demonstrating that one can be recommended for entry into a discipline or profession.  So, I’d say:

Code equivalent to a statement like:
A+ = “Wow!”
A = “You are at mastery at this stage.”
A- = “You are well on your way.”
B+ = “OK, but discipline yourself”
B = “You may be in the wrong career given your talents”

In the long run, my “real” assessment of a person work is the enthusiasm of my letters of recommendation whether for funding or professional positions.  And these letter never never mention grades since “Pass” is the only possible one at the final levels.

Print This Post Print This Post
comprehensive, on assessment, on schooling policy

On education on Lake Wobegon

1 Comment

Everytime I introduce my work with Ray McDermott, I echo something he probably says more eloquently than I: “What schools all about? They are about determining which 50% of children are below average!”  Given that much of this is done through testing, and that the good test “discriminates,” then I sometimes say, to provoke, that schools are all about discrimination. (See for example a short introduction to “Interpreting the Index of Discrimination” )

Such statements grab the attention of students, but I am not always quite convinced that the answer is more than a provocative quip.

And then I read paragraphs like one that introduced a recent story in the New York Times:

Ohio seems to have taken a page from Lake Wobegon, where all the children are above average. Last month, state officials releasing an early batch of test scores declared that two-thirds of students at most grade levels were proficient on reading and math tests given last spring under the new Common Core requirements. (October 6, 2015)

The story is of course not about how successful the schools of Lake Wobegon or Ohio are.  The story is about “the problem that a lot of policy makers and educators were trying to solve,” as “Karen Nussle, the executive director of the Collaborative for Student Success, a Common Core advocacy group,” is quoted as saying.  As summarized by the New York Times, “The Common Core was devised by experts convened by state education commissioners and governors to set uniform benchmarks for learning. … But as the results from the first Common Core tests have rolled out, education officials again seem to be subtly broadening definitions of success.”

In other words, as McDermott and I argued, success if indeed defined by failure (1998).  It is necessary to fail students in order to demonstrate that other students are successful.  It cannot be that all children (or even most, or even more than some measure of the average) should be “proficient.”  The label must apply only to a certain percentage.

The “debate” (though the New York Times is not really debating as the article clearly sides with Common Core policy makers) is thus about labels, statistical uniformity, comparability across the United States—and forms of unacceptable tinkering if not cheating.

The debate is not about learning, and even less about education.

“Only in America” am I tempted to say, except that, actually, there is something interesting going on here that a call to political theories of cultural arbitrary (as all theories of culture, from Boas onwards have been, when taken strictly) should highlight.  The story is also about a political struggle among the elites about precisely how America should work, in general, and in the detail of the lives of politicians, schools administrators, principals, teachers, parents and other adult who might express opinions or vote about all this—not to mention university professors designing tests, billionaires funding “school reform,” union leaders and many others.

I make this list to bring attention to the evidence that all these people, in the worlds that they inhabit will talk and act in ways that will often make problems for each other, and that they will do that purposefully (systematically and deliberately to cross-reference Larry Cremin and my take on “education”).  In relatively neutral language they are conversing (which is not quite the same thing as “negotiating”) often with the hope of producing something different than the probable or expectable.  They are not simply acting in terms of their dispositions (habitus, etc.).

I make the list also to move further than where Ray McDermott and I were when we completed Successful Failure.  As Jill Koyama (2008) noted, we mentioned “America” but did not quite show how it actually produced what we observed, in temporality.  We had essentially worked by drawing a structural model of a historical moment (“culture”) that emphasized the relationship between democracy, meritocracy, the institutions that they produced, and the consequences for individuals (to simplify of course).  We were directly inspired by Louis Dumont (1980 [1961]) on the relationship between individualism and racism.

This kind of (Lévi-Straussian) structuralism can be helpful, but it never was able to specify how what was modeled actually came into reality in the day to day life of those caught by the culture.  So, more or less explicitly, social theorists implied or stated that what was modeled was real and powerful enough to generate what could be observed.  McDermott and I wrote extensively against this move to “structuring structures” (to quote Bourdieu’s jargon).  But we did not quite find a way to state how the democratic fight against birth privilege ends up producing discriminatory tests, the failing of teachers who do not “add value” to children and all other policies justified by calls to the discovery and reward of individual merit.

Thus my interest in following what the New York Times reports, and how it writes its reports.  I take these as statements within a conversation, in the same spirit as McDermott wrote about Rosa’s “I could read it”: the statement makes sense given the conditions but it is not produced by the conditions.  The conditions are set by earlier statements, most of them made by other people, far away and long dead, as McDermott and I like to say.  But the actual statement (act) is produced by a particular person, caught together with specific persons (consociates), at a given time.  In that perspective, it makes sense for bureaucrats in Ohio to move the boundary between proficient or not.  And it also makes sense for others powers-that-be to try and move it back.

What of course no theory of culture can answer is “why should it make sense?”  except perhaps if “a” culture (epoch, episteme, …) is understood, again, as a statement making sense in terms of earlier statements (culture…).  Thus, the shift to democracy, meritocracy, schools, testing, might be seen as a response to earlier discourses and institutions for elite production.  How to move the conversation to its next statement is our problem, as political actors and, I would say, as educators attempting to convince various audiences that they are on a track that may only make matters worse.

References

Dumont, Louis 1980 [1961] “Caste, Racism and ‘Stratification’.” Chicago: The University of Chicago Press.Tr. by M. Sainsbury. Rev. ed.. In his Homo hierarchicus

Koyama, Jill 2010 Making failure pay: For-profit tutoring, high-stake testing, and public schools. Chicago: The University of Chicago Press.

 

Print This Post Print This Post
comprehensive, on assessment, on schooling policy

Where do (psycho/socio)- metricians fit?

Leave a comment

Recently, March 28 2012, I spent the afternoon at the plenary session of an “International Conference” on “Educational Assessment, Accountability, and Equity: Conversations on Validity around the World.”  The plenary speaker was Michael T. Kane, “The Samuel J. Messick Chair in Test Validity” at the Educational Testing Service.  He talked about validity as measurement scientists deliberate about it, and about some of their soul-searching when they consider the impact of their measurements.  Or, as I would put it, wearing my “anthropologist of Nacirema” hat, he talked about the misgivings of an obscure priesthood specializing in an abstruse numerology few understand outside their rarified convents.  Kane, as a master in this polity of conjurers of numbers, gave us, the uninitiated or very peripheral, a glimpse of his doubts and those of other masters as they discover that they are now at the very center of political storms where their more abstruse spells are thrown at opponents for all sorts of reasons having little to do with numerology.
NYC value added model for teachers

To the extent that I understand it (and I am very far at the periphery of numerology, or rather, I am at the periphery of the gravity well that might have made me, at some point in my career, a legitimate peripheral participant), it all has to do with the “interpretation” of the test that leads to its being used in a particular case.  But Kane and his peers are not quite where Geertz and his peers have been.  For one, Kane is deeply concerned with specifying and justifying the interpretive steps.  For another, he and is peers have, precisely been thrust into the center, while symbolic anthropologists are pushed even further away from it.

This occasion was the second in recent weeks when I heard thoughtful (psycho-)metricians wonder about the public face of their craft.  I had not suspected how much debate do happen among the scientists of the measurement of individual behavior about what happens with the measurements when these measurements are used outside the world of measurement.  Kane taught me something about the relationship between the “datum” (an answer on a test question) and the inferred “claim” (that Johnny failed the test) and the “warrant” that allows on to make the claim based on the datum.  The warrants themselves are “backed” by empirical studies.  Thus, everything depends on the quality of the studies which back the warrant that allows for the inference.  Things are even more difficult since the various inferences that can be made about the individual as this_test taker can be transformed into inferences about this_field (that is, that Johnny who failed this reading test do not know how to read), and then transformed into even more general properties of the individual as performed in any_field (that Johnny is “with” this or that syndrome), and then transformed into properties of a population (White vs. Black, poor vs. prosperous, American vs. Chinese).

As I listened, I was particularly struck by his discussion of “warrants” in the making of inferences and the place of various logical and mathematical ways of explaining how one gets to the inference.  Listening to this, I understood better why ethnography is looked askance by measurement scientists: we, anthropologists, could be said to be “warrant-challenged” when we watch a cock-fight and then make inferences about humanity…

And then things became truly interesting.  Kane started to talk about a particular type of inference that shift from identification (Johnny is with X or Y) to the meting of high stake consequences (that Johnny should be shifted to a special education classroom, that he should not receive a degree, that he should not be hired or promoted).  He illustrated the difficulties by reminding us of the Supreme Court ruling in the case of Griggs v. Duke Power where the issue was the use of a test (or more precisely inferences about the people who had taken the test) for employment, that is as a step in the making of a high stake decision that could have heavy negative consequences.  In effect, the Court extended the notion of validity to include the impact of the test on the life of the taker.

I am about sure that no inference from anthropology has ever been debated in the Supreme Court of the United States.

Thus, the Court also, and by implication of course, definitely placed (in the Garfinkel sense) testing as the proper instrument of high stake decision making and the testing scientists as perhaps the most powerful engineers of social structural production (along with the professional in charge of diagnosing decease and its legitimate political implications).  That is, by requiring that tests be “reasonably related” to the job for which the test is required, the Court fully legitimated a process of assembling people and practices that had fully flowered with Thorndike and other measurement specialists when they convinced school people that psychological testing might produce what Dewey and others had appeared to call for: a democratic educational system where the real properties of the child were the sole criteria for the advancement of the child through the rewards of that part of social life (for examples being hired for a job) that the state, through its courts, can regulate.

Thus, the Supreme Court, and by implication of course, placed ETS at the core of the political process and thus made a particular class of scientists the arbiter of this process—all the more so that only they fully understand the means they use (regression formulas and the like) to produce something that later allow human resources personnel, or college admissions officers, make decisions without appearing to have made them.  When I talked about terminating Skynets in my last entry, I did not yet know that I was echoing was some measurement scientists have actually said:

Quantification is a way of making decisions without seeming to decide (Porter 1995: 8).

 

Porter, T.
1995 Trust in numbers: the pursuit of objectivity in science and public life. Princeton, NJ, Princeton University press.

on assessment

What’s a teacher to do?

Leave a comment

New York City found out on February 28 that an elementary school teacher I know well rates a “34 (7-73) 32 (5-84)” in Reading and a “63 (41-82) 77 (42-91)” in Math.

A few months ago, teachers had received from their school summary documents that looked like this:
Math result for a NYC teacher
What is an individual teacher to do about any of this?  What, on a day to day basis should a teacher do to “improve” on a 34 and maintain a 77?

These numbers are somewhat related to the wonderful awful formula:

NYC value added model for teachers

What are the values of these variables for any particular teacher? Which of these variables are under an individual’s control?  On what day of the year?

I venture that neither common sense, habituation into any cultural world, guesswork, or any other process proceeding from the individual teacher as teacher or person, is likely to help in answering these questions.  I suspect that a whole new class of professional consultants is now being inducted into fuller and fuller participation in new polities in all sorts of institutions.  They will be sold as the interpreters of the ratings.  They will also be people with children and mortgages who will have ever more interest in keeping the formula opaque.  They will be joined by the psycho/socio-metricians tinkering with the formula to “improve” it so that they can report to the New York Post that “the complaints of the teachers have been addressed,” various software engineers, etc.  And the web keeping everybody in place will get tighter and more difficult to escape.

The question we need to raise is, of course, whether teachers should have to ask questions about manipulating variables on a formula.  The formula may be wonderful as a research tool, but it is awful as a method for hiding political decisions and making it appear that these decisions are removed from precisely political activity at all levels of schooling.  As a political tool it may be intended to take the place of a terminating Skynet where evaluation, like the response to some foreign threat.  Evaluation, it appears, is taken out of the political realm of principals meeting teachers in a school, and into the realm of automatized mechanisms noone quite understand but are un-impeachable, as well as altogether unaccountable.  That people will be hurt people is their problem will the newly powerful say: “good” teachers (the top 50%? 75%? 25%?) “have nothing to fear” and “bad” teachers should fear dismissal (unless the whole exercise is pointless).

Whether any of this will do anything to improve education in any of its senses in the question may be a question one asks at one’s peril.

comprehensive, on assessment

On Political Deep Play – a coda on experimentation

Leave a comment

My entry from March 2 played, very seriously, with the kind of deep play policy makers in the world of schooling engaged in when they released invalid scores purported to tell how well individual teachers taught.  On March 9th, James B. Stewart of the New York Times, asked “Would Americans be better off if General Motors and Chrysler had simply gone bankrupt, without benefit of taxpayer assistance?” and he raised the question of the kind of evidence one could use to answer such a question.  What picked my curiosity is the following comments:

Unlike a science experiment, in which variables can be changed and the experiment repeated, we can’t turn back the clock, let the auto companies go bankrupt and compare the results with what we have today, which is an American auto industry that is, by nearly all measures, healthier than it’s been in many years. G.M. and Chrysler, not to mention Ford, which didn’t get taxpayer money but benefited indirectly, are profitable, hiring more workers, competing more effectively, gaining market share and building better cars and trucks.

He then proceeded to make comparisons with other companies that were, or not, helped by the government when they face bankruptcy.  Essentially, he was using history rather than “evidence-based” empirical research to argue in favor of a political decision.

Now, of course, history, like anthropology, is precisely not an experimental science and yet it may more useful to “politic” makers, that is politicians, as actors, rather than “policy” makers as advisors to the actor.  The very small group (Obama, Geithner, ??) who decided to bail out General Motors could not rely on “evidence.”  They had to rely, in the best sense of all these words, their ideology, their common sense, and the conversations they must have had.

In other words, they placed a major bet.  It looks like they won.  But this was about the deepest of deep plays.  The only deeper I can imagine is Roosevelt or Wilson getting America into World Wars.

Back in New York, it is probably the case that Bloomberg made a similar bet when he had the test scores released.  He could not wait for the “evidence” that this release would lead to better teaching.  By the time this evidence was in, then the political problem would probably have been moot.  We can disagree with his decisions.  We can note the irony that people who have prided themselves on being “data-driven” made a major decision in the absence of data.  But we see the decision for what it is, a political decision, not a policy decision.  And as one approaches political decisions, history, and anthropology, may be more useful than “experimental” social sciences.

How can we convince policy/politic makers that evidence-based research is not the way to a better democracy?

comprehensive, on assessment

MOOC: Education, degrees, careers?

Leave a comment

I ended a recent paper for the Gordon Commission on the Future of Assessment with comments about the possible disassociation of schooling from assessment.  My argument was that the association between the two has been 1) good for schooling in that it has massively increased its reach and claim on resources, and also 2) terrible as it makes it less and less relevant to education.  The recent, and ongoing, controversy about publicizing teachers’ scores on invalid measures is a case in point: teacher-ing, with ever more symbolic violence (a.k.a “accountability”), is made into a skill for putting measurable bits into students’ head.  That teacher-ing might be about participating in shaping a mind is left aside as not a concern for the State.  But why, may we ask, should the State be concerned with education?  Well, of course, because

the tests
…..that lead to the degrees
……….that accredited  schools grant
……………that employers use to open or close doors to careers

are essential for the representation that a political system is indeed “democratic” and that state rewards are indeed distributed on merit rather than birth privilege (in its racial, ethnic, class, gender, etc. forms).

In many ways, as the people of the School have been saying, focusing on tests leading to degrees is a radical narrowing of what was the mission of schooling.  Arguable, the battle has been lost as much (most?) of what was included in this mission has now been distributed out to the family, the media, religious institutions, etc.  But schooling, as an institution, appeared to remain central because it has kept its monopoly on the granting of degrees.

What if this changed?  What if a successful challenge was mounted to legitimize other doors to adult careers than those controlled by the current schools, colleges and universities (and their teachers or faculty)?

In the past week, the New York Times published evidence that this challenge has started.  One is an opinion piece by Charles Murray asking an “energetic public interest law firm” to challenge “the constitutionality of the [bachelor’s degree] as a job requirement” (March 8, 2012).  The Supreme Court, I did not know, has made it unconstitutional to make test scores the key to employment unless there is a tight link between the test and the job.  Demonstrating the link between almost any college degree and almost any job might be difficult.

The other report may be a more immediate and less ideological challenge, and possibly much more difficult for school people to block (particularly since some of them are profiting from this challenge).  I am talking here about “Massive Open Online Courses” (MOOCs).  Stanford offered a classes for 160,000 students in 190 countries!  Another class, on Machine Learning, was given for 104,000 students with 13,000 completing the course.

The figures are astounding.  They are about something that is happening now and will have social and political consequences.  Of the 13,000 who complete the course on Machine Learning, most of them must have learned something but none got a State recognized certificate or a degree.  So, at this point, they completed the course “for nothing”–that is just for the education of it.  Some of them may also perform their professional tasks better.  I suspect those will not accept for very long to be passed for promotion by people who have a degree.

If the State finds ways to accredit (“give credit for”) the taking of MOOC (perhaps by asking a company like ETS to give an independent and controlled test), the implications for universities and their faculty are staggering.  The New York Times quote one of the Stanford professors who taught one of the MOOC as saying that he does not want to go back to teaching just twenty students in a small classroom.  When Stanford has found a way to charge people for the course, and reward the faculty member in commensurate fashion, the whole economic basis of colleges is transformed.

We may be seeing the end of schooling as we have known it (and for people of my generation profited from it).  It is going to be quite a ride.

comprehensive, on assessment

On ecologically valid assessments

1 Comment

At some point during the mini-conference on the future of assessment (held on April 11, 2011), Ray McDermott raised questions about the validity of the kind of tests the Educational Testing Service and such design.  He told of the work he conducted in the late 1970s as part of Michael Cole’s Laboratory of Comparative Human Cognition.  Then McDermott, Cole and others wondered about the relationship between tests and the settings about which the tests were supposed to say something.  As they showed (1979, 1998: Chapter 1), the relationship between, for example, a reading test and baking banana bread by reading a recipe is tenuous, at best.  In the setting of a cooking club, so much else happens (from confused writing to interpersonal tensions) that ability to read is the least of the problem the children have to deal with.  The generalizability of these observations across settings and populations is now well established through repeated observations.

What has been left open in this work is the question of finding out what ecologically valid assessments would actually look like.

Soon after the conference, another participant, Katie Anderson-Levitt (U. of Michigan-Dearborn), suggested we look at Paradise and Rogoff’s recent paper about ongoing learning in families (2009).  In that paper, Paradise and Rogoff mention all the work done in the Cole tradition over the past 30 years with a new twist that fits well with my own sense of what I call ‘education.’  In everyday life, at home, “learning” is not a simple automatic matter proceeding below deliberation or symbolic expression.  In everyday life “teaching” (and assessing) is—probably—ubiquitous.

As I reflected on all this, I saw a route I have not yet quite explored and that could lead to further research expanding on the Cole, Lave, etc., traditions.  Starting with an expansion of the point Paradise and Rogoff made, I suspect that  the movement through publicized ignorance is accompanied by all sorts of speech acts, many of which fit in the paradigm of knowledge assessment.  Developing all this is also an expansion on Garfinkel, as I take him.

Garfinkel has kept arguing that maintaining any order requires ongoing work, including the work of figuring out what is going on.  Conversational analysts has given abundant evidence that this is indeed correct.  More recently, Garfinkel wrote about ‘instructions’ as a necessary aspect of this work.  The paper ends with one of my favorite quotes about screwing around and getting instructed (2002: 257).  What I do not think Garfinkel noted, and what I know I never noted myself, is that the instruction moments proceed either from an earlier assessment, or themselves constitute an assessment.  This is also an implication of Gus Andrews recent dissertation (2010) on blog comments when these are assessed as being “wrong” in some way that is specified by a later comment (“this comment does not belong here,” “you should not write your social security number here,” etc.).  In an interactional sequence (conversation?) utterance of the type “Do X differently!” are probably essential mechanisms for maintaining order, constituting emerging orders, moving participants into new positions, etc.

I am quite sure that such ongoing assessment is ubiquitous and should probably added as a function in Jakobson’s model of communication (1960 — though he might have classified it as an aspect of the metalingual function).  Much of the recent work on metapragmatics may also fit here.

In brief, and for our purposes, we could say that Ethno-methodology is at the service of ethno-science (what is the world made of?), and ethno-politics (how do we maintain the order within which we are now caught?), it also at the service of ethno-assessments. [or should we say that (ethno) Methodology is at the service of (ethno) Science, (ethno) Politics, and (ethno) Assessment?]

If this proves a useful direction for inquiry, it suggests that assessment is not an extra-ordinary task.  It also suggests how school assessment has drifted away from the ordinary [I am not sure that ‘drifted’ is the right work, but it will do for today].  The well known school-based QAE (Mehan 1979) model is formally equivalent to what might get known as the SARS model (Statement, Assessment, Re-statement) except that the former starts with the assessor’s question while the later starts with a seeker’s request that may then lead to an assessment (though this proposal may not have been presented as such).  In other words, the sequence starts with ignorance grounded in the here and now (“ecologically valid ignorance”?) and proceeds with statements of local knowledge that are themselves proposals for what it is that the seeker may plausibly not know (I am using the word ‘seeker’ rather than ‘learner’ since it will remain a question wether the subject whose ignorance is marked will learn anything out of the encounter).  This sequence is what I would now say my earlier statements about “productive ignorance” were about.

The question to designers of future tests is something like: how might you produce assessments that are triggered by acknowledgments of ignorance, whether generated by the subject (“I would like to know about X”) or by a co-participant in the polity (“you really should learn more about X”).  The challenge is to find the moment in the sequence of a life when the co-participant teachers will enter.  In everyday life it is a non-problem to the extent that co-participants or “consociates” have the built in or self-generated (legitimate) authority to assess (as siblings may have).  When social distance increases, that is when the network links between those who set what is to be assessed, what is to count as ignorance, and what should be done about include many persons in many institutions, then the problem gets acute.  It may even be unsolvable unless we find ways to reposition the official assessors within the network so they are closer to the performance in such a way that they can get a better sense, in real time, of the feedbacks that the seeker (learner) provides.

(More on what I am trying to formulate about network linkages later)

comprehensive, on assessment

Practical assessments, perhaps

1 Comment

This is my third entry developing some of the points we discussed during the mini-conference on the future of assessment (held on April 11, 2011).  The first two entries (on audiences, and on utopias), and in my initial one before the conference (on political philosophy) were essentially analytic with a definite pessimistic lean.  (School) assessments are bad for the health, and yet they are here to stay given all the powers they serve—including idealistic ones.

Robbie McClintock re-started me on a different track I briefly explored in my last contributions (2010) to the series on Comprehensive Education Ed Gordon and I edited (2008, 2009, 2010).  There I mused about possibilities for forms of institutionalized education that did not proceed from schools.  I am convinced that adults gain their most significant knowledge (about, say, health, emerging scientific and engineering developments, trends in high and popular culture, etc.) from institutions (for example, journalists and television reporters or producers) that are not controlled by the State.  Most of these institutions do not present themselves as primarily in the education business.  But perhaps they should, and draw the consequences.

McClintock emphasized the Web as another source of an education that is not necessarily packaged as, precisely, “education” but perhaps only as “information,” “entertainment,” etc.  Some writers and producers for the Web may present themselves as educating but none, at this moment assess what one may learn by reading their offerings.  Yet, as McClintock pointed out, the new technologies, as they are evolving, afford for possibilities for ongoing, real time assessment that older technologies (for example the printing press) do not afford.

This is intriguing.  Time for a little science fiction.

Imagine a new Google service.  At this moment, Google answers questions of the type “where can I find about X?”  Whether the seeker is satisfied or not with the answer, whatever the seeker does with answer, Google remains silent after providing a list of possible answers ranked by Google’s best guess as to the seeker’s intent.  If the seeker is dissatisfied, he may ask again and Google will answer, but Google has no memory of what this seeker asked and Google’s answers will not evolve as answers do evolve when, say, a child ask a parent about X.  So, last week, while writing my preceding blog entry, I looked for the creationism museum I had read about.  I entered “creationism” in the Google box (search page saved on of 4/20/2011), clicked on the first of 3,150,000 results (a suspiciously ‘round’ number), and found myself in the Wikipedia entry that started, on that day, with “creationism is the religious belief …”  There are no links to the museum in that entry, and so I asked Google again, found that it was second in its list of possibilities (and I also found out, serendipitously, a “Conservapedia” with an entry on creationism that is close but interestingly different from Wikipedia’s.  I am, of course, on my way to educating myself about creationism—not so much as a belief but as an institution with, among others, curricula experts quite deliberately teaching that which School people are quite sure should not be taught.  But Google never intervened in my education.

Imagine that Google did intervene.  Imagine that Google, as run by some revolutionary government, traced my queries, adapted its answers to my renewed queries and, mor or less insistently started asking me about my beliefs and worked at correcting them.  Parents do this with their children.  And school teachers do this with their pupils.  Why shouldn’t Google?

I suspect that the technological infrastructure of such a service is already in place and that it would not take much tweaking of the various flavors of social software already available to make them serve the new function.

The challenge is multiply institutional.  Who is to start it? Fund it? Control it?  Who is to credential (authorize) the persons or software assessing self-sought knowledge in real time?  Who is to establish the curriculum and it goals to which seekers are to be brought back even as they explored far and wide?  Individuals, clubs, associations, etc., may already provide feedback in the sense that their web sites are less informational than argumentative.  For example, the fifth set of links to ‘creationism’ on Google mostly lead to “atheist” web sites specifically addressing creationist links and debunking the claims.

I have never heard of States getting into this as deliberately as States get in the business of setting school curriculum.

The libertarian and anarchist in me whispers: why would you want the State to get involved?  The school critic from the left and the neo-liberal from the right might ask the same question.  States have a wonderfully awful (or is it awfully wonderful?) track record of setting the curriculum for the mass populations they govern.  Why would they do better with real time ongoing assessment than they have with the usual forms of test or examination based assessments for which schools are (in-)famous?

The statist in me begs to differ.  Free, state regulated, public schools have done much that is good in transforming what is to count as the knowledge on which public and private lives should be based, as well as the means for the production of such knowledge.  Rationally based, modernist (?) expertise should probably remain at the core of what States support, propagate, and assess as, precisely, rationally based.  If Latour (1993 [1991]) is correct, “We may never have been modern.”  Modernism itself is a cultural construction that must be reconstituted on an ongoing basis to remain the order of the day.  True enough, but what else might we wish to build?

What ongoing assessment software should be now build?

comprehensive, on assessment

Utopias and dystopias: Futures for education, technology, and the assessment of authority over knowledge.

2 Comments

When we met last Monday (April 11, 2011) for the mini-conference on the future of assessment, we, of course, talked a lot about the impact of the new technologies.  Without much prodding from Robbie McClintock we mentioned the oft celebrated opening of access to knowledge as well as the distribution of the production of widely accessible knowledge.  The new technologies open routes to knowledge that do not pass through the gate-keepers we are familiar with: universities, research libraries, newspapers, state-controlled curricula, medical institutions (not to mention the theologians and priests who, until recently appeared to be fading as gate-keepers).

Someone then mentioned that all this has a very dark side.  With the distribution of access and production comes a distribution of the authority to interpret and constitute knowledge so that it can guide practical action in the world.  This can be scary.

In brief, everyone, about, can read Wikipedia (and other sources available on the Web).  But Wikipedia writers cannot assess what people are doing with the reading, whether in real time or, in a posteriori tests.  Readers may learn, as individuals.  More significantly for us, they may then act publicly and practically.  Disagreeing with Darwin for religious reasons is one thing.  Building a “Creation Museum” is something else together (not to mention trying to enter creationism into the public school curriculum).  Doubting the efficacy of vaccines is one thing, not getting a child vaccinated is another one.

Can “democracy”—as Dewey might have meant it when he titled a book “Democracy and Education” (1916)—survive the full democratization of knowledge?  This, of course, is an old problem.

I will stay with the technical issues and keep the implications for political philosophy implicit.  The issue we must raise concerns the interpretation of two sets of now common observations about the entry of the new technologies into public common practices.

On the one hand there is the evidence for the power of crowd sourcing to establish, through communal self-correction, what is, for an intent, purpose, and polity, expert knowledge.  Much of Wikipedia fits here and there are reports that many fields are using crowd-sourcing for scientific purposes in the modernist sense.  McClintock’s own experiment with StudyPlace is a valiant attempt to use wiki software for more than the writing of encyclopedias.

On the other hand there is also evidence for the power of any number of other self-constituting polities to constitute, propose, defend, etc., forms of knowledge that other polities—says academics—will characterize as erroneous.  Post-modernists, critical theorists, performance artists may celebrate this anarchic blooming but the flowers of such blooming can be toxic.

In both cases one has “education” in my sense of “difficult collective deliberations” (2007) particularly as the various polities discuss and address each other.  In both cases one has evidence for collective ongoing “assessments” of earlier statements (Andrews 2010) by people who specifically claim some expertise, or who just proceed as if the assessment had been made by an expert.  And both cases leave state-regulated or controlled “public” schools in a difficult position.  For a century or two school people have been charged with developing knowledge (in research universities for example), deciding which subset of this knowledge is appropriate for what kinds of publics (through the setting of curricula), experimenting with methods (pedagogies) to impart this knowledge.  They have also been charged with vouching that particularly populations have incorporated this knowledge—through achievement tests for example.  In this arc of knowledge (re-)production, alternate forms of knowledge were either specifically discounted (“grand-mothers’ knowledge”) or just ignored.

For at least half-a-century, critics have protested this arc but they did it mostly from the safety of various ivory-towers (think Bourdieu blasting La Sorbonne from his post at the Ecoles des Hautes Etudes en Sciences Sociales).  They rarely had to deal with the practical consequences of direct attack on academic expertise from putative experts who now have the tools to be heard.  It is only recently that the population at large has had the technological means to make fully public these alternate forms of knowledge.  It may be that finally, post-modernism is being institutionalized.

And so, those of us, particularly in anthropology, who see merit in the argument that constituted knowledge is founded on arbitrary institutional (political) means must confront the consequences of the mass distribution of constitutive tools, and thus the appearance of new forms of arbitrary political power (in all the sense of “arbitrary”—the political as well as the semiotic).

In a future that may already be here, schools may become irrelevant as producers, transmitters, or assessors of knowledge—though they could remain as essential baby sitting institutions, and as gate-keepers into the service industries.  This future may be wonderful, or awful.  Stressing the reality that the web makes public all knowledge and frees inquiry from the mediation of the institutions that limited access, can turn into technological utopia.  Stressing the reality that unmediated information can lead to all forms of altogether dangerous knowledge, can turn into technological dystopia.  It is probable that the 21st century will be a time when the classic debates about letting the Christian faithful read the Bible will be reprised (and will not, one hopes, lead to the kind of violence Europe experienced in the 16th century).

on assessment

Assessing audiences: identifying reachable designing assessors

1 Comment

In an earlier post, I mentioned my confusion when I was asked by Ed Gordon to consult for a “Commission on the Future of Assessment” he is convening.  The first meeting of this commission is to happen in June and I am now expected to write a few pages based on a mini-conference held last Monday (April 11, 2011) at Teachers College.  A longer paper is to be written later.

The mini-conference brought together Robbie McClintock, Ray McDermott, Kathryn Anderson-Levitt and half-a-dozen recent doctorates (Gus Andrews, Alex Posecznick) and students.  A grand intellectual time was had by all, and I will be writing about various highlights of the conversation in future blogs entries.

For today, I want to muse about one theme to which we kept coming back: who is our audience?  Are we addressing each other, sympathetic anthropologists, students (“the choir”)? Are we addressing Ed Gordon who convened us? The Educational Testing Service that is funding the commission and is, of course, a most powerful designer of tests and other means to assess large populations?  The universities and other institutions that buy the tests or their results, and that may also shape mass assessment, whether directly (through, for example, their admissions practices), or indirectly (through research and other activities that justify or interpret assessment)?  The “policy makers,” an altogether opaque “group” (?) who propose programs to deal with what the test results appear to reveal? The politicians who may or may not accept what policy makers propose? The tested people, or their parents or guardians, who organize their life around future test taking, of past test results?

Those who know my recent work as it has been influenced by students like Jill Koyama or Gus Andrews will recognize in this paragraph a very sketchy sketch of a “network” in what may be Latour’s sense.  In this network each node (which is itself a complicated network) is constrained by what happens elsewhere and constraints what can happen there.  Together, they may constitute what Ray McDermott and I talk about when we write about “the School America builds” (1998).

Ah! The pleasures of analysis!

This leaves us with the practical problem: Given our position in the network (some backwater of the research university), who will even notice that we have spoken?  Where should we stand so that we will have a better chance not only to be heard, but also listened to?

This, of course is a general and classical problem for anthropologists.  Different anthropologists, from Boas to Margaret Mead, to Geertz (to mention only some who are now part of the history of the field), have given very different answers and entered very different publics–or withdrawn from them.  As faculty in schools of education, all of us have had to accept, mostly willingly I would say, the position of spokesperson about anthropology or philosophy to an audience of students, and some faculty colleagues, whose main concerns are with action in the world beyond the analytic or interpretive action most typical of academic researchers.  It is said that some of our colleagues in academic anthropology argue that anthropology should not be “applied” and that, I guess by implication, anthropologists should not address the public.

Many anthropologists disagree with this.  The American Anthropological Association regularly passes resolutions on the hot political topic of the day.  But we are talking here about doing much more than that.  We are talking about entering ongoing conversations about assessment with professionals and political actors of all types and, potentially, or inevitably, run the risk of being co-opted back into what we criticize.  We are talking about changing discourses and, we hope, practices not mainly in classrooms and local schools, but rather at the level of the governmental institutions that control what is to be known legitimately about individual schools, teachers and, of course, students.  We are talking about entering the political fray and finding a way to be listened to.

We certainly did not find out how to do this.  But we need to keep addressing the issue.