comprehensive, on assessment

On Political Deep Play – a coda on experimentation

Leave a comment

My entry from March 2 played, very seriously, with the kind of deep play policy makers in the world of schooling engaged in when they released invalid scores purported to tell how well individual teachers taught.  On March 9th, James B. Stewart of the New York Times, asked “Would Americans be better off if General Motors and Chrysler had simply gone bankrupt, without benefit of taxpayer assistance?” and he raised the question of the kind of evidence one could use to answer such a question.  What picked my curiosity is the following comments:

Unlike a science experiment, in which variables can be changed and the experiment repeated, we can’t turn back the clock, let the auto companies go bankrupt and compare the results with what we have today, which is an American auto industry that is, by nearly all measures, healthier than it’s been in many years. G.M. and Chrysler, not to mention Ford, which didn’t get taxpayer money but benefited indirectly, are profitable, hiring more workers, competing more effectively, gaining market share and building better cars and trucks.

He then proceeded to make comparisons with other companies that were, or not, helped by the government when they face bankruptcy.  Essentially, he was using history rather than “evidence-based” empirical research to argue in favor of a political decision.

Now, of course, history, like anthropology, is precisely not an experimental science and yet it may more useful to “politic” makers, that is politicians, as actors, rather than “policy” makers as advisors to the actor.  The very small group (Obama, Geithner, ??) who decided to bail out General Motors could not rely on “evidence.”  They had to rely, in the best sense of all these words, their ideology, their common sense, and the conversations they must have had.

In other words, they placed a major bet.  It looks like they won.  But this was about the deepest of deep plays.  The only deeper I can imagine is Roosevelt or Wilson getting America into World Wars.

Back in New York, it is probably the case that Bloomberg made a similar bet when he had the test scores released.  He could not wait for the “evidence” that this release would lead to better teaching.  By the time this evidence was in, then the political problem would probably have been moot.  We can disagree with his decisions.  We can note the irony that people who have prided themselves on being “data-driven” made a major decision in the absence of data.  But we see the decision for what it is, a political decision, not a policy decision.  And as one approaches political decisions, history, and anthropology, may be more useful than “experimental” social sciences.

How can we convince policy/politic makers that evidence-based research is not the way to a better democracy?

comprehensive, on assessment

MOOC: Education, degrees, careers?

Leave a comment

I ended a recent paper for the Gordon Commission on the Future of Assessment with comments about the possible disassociation of schooling from assessment.  My argument was that the association between the two has been 1) good for schooling in that it has massively increased its reach and claim on resources, and also 2) terrible as it makes it less and less relevant to education.  The recent, and ongoing, controversy about publicizing teachers’ scores on invalid measures is a case in point: teacher-ing, with ever more symbolic violence (a.k.a “accountability”), is made into a skill for putting measurable bits into students’ head.  That teacher-ing might be about participating in shaping a mind is left aside as not a concern for the State.  But why, may we ask, should the State be concerned with education?  Well, of course, because

the tests
…..that lead to the degrees
……….that accredited  schools grant
……………that employers use to open or close doors to careers

are essential for the representation that a political system is indeed “democratic” and that state rewards are indeed distributed on merit rather than birth privilege (in its racial, ethnic, class, gender, etc. forms).

In many ways, as the people of the School have been saying, focusing on tests leading to degrees is a radical narrowing of what was the mission of schooling.  Arguable, the battle has been lost as much (most?) of what was included in this mission has now been distributed out to the family, the media, religious institutions, etc.  But schooling, as an institution, appeared to remain central because it has kept its monopoly on the granting of degrees.

What if this changed?  What if a successful challenge was mounted to legitimize other doors to adult careers than those controlled by the current schools, colleges and universities (and their teachers or faculty)?

In the past week, the New York Times published evidence that this challenge has started.  One is an opinion piece by Charles Murray asking an “energetic public interest law firm” to challenge “the constitutionality of the [bachelor’s degree] as a job requirement” (March 8, 2012).  The Supreme Court, I did not know, has made it unconstitutional to make test scores the key to employment unless there is a tight link between the test and the job.  Demonstrating the link between almost any college degree and almost any job might be difficult.

The other report may be a more immediate and less ideological challenge, and possibly much more difficult for school people to block (particularly since some of them are profiting from this challenge).  I am talking here about “Massive Open Online Courses” (MOOCs).  Stanford offered a classes for 160,000 students in 190 countries!  Another class, on Machine Learning, was given for 104,000 students with 13,000 completing the course.

The figures are astounding.  They are about something that is happening now and will have social and political consequences.  Of the 13,000 who complete the course on Machine Learning, most of them must have learned something but none got a State recognized certificate or a degree.  So, at this point, they completed the course “for nothing”–that is just for the education of it.  Some of them may also perform their professional tasks better.  I suspect those will not accept for very long to be passed for promotion by people who have a degree.

If the State finds ways to accredit (“give credit for”) the taking of MOOC (perhaps by asking a company like ETS to give an independent and controlled test), the implications for universities and their faculty are staggering.  The New York Times quote one of the Stanford professors who taught one of the MOOC as saying that he does not want to go back to teaching just twenty students in a small classroom.  When Stanford has found a way to charge people for the course, and reward the faculty member in commensurate fashion, the whole economic basis of colleges is transformed.

We may be seeing the end of schooling as we have known it (and for people of my generation profited from it).  It is going to be quite a ride.

on anthropological theorizing

Value Added Deep Play

Leave a comment

Geertz’s “Deep play: Notes on the Balinese cockfight” (1972) is justly famous and yet its central argument, that the Balinese, sometimes, play “deeply’ in such a way as possibly to endanger their status, has had surprisingly little impact.  Maybe it is because it is set in a mysterious island far, far away.  Maybe it is because “we” do not bet on cockfights.  Maybe it is because, though quite a few of “us” play more or less deeply in Las Vegas and other such venues, “our” institutions, dominant ideologies, etc., do not play much and frown on any play dangerous enough to risk our health or well-being.  Quite the reverse indeed as “we” pride ourselves on ever more detailed protective regulation of what must be done while “at play” (in play grounds, riding cycles, etc.).  But, of course, we, personally and as implicated by more or less enlightened leaders do “deep play”–though not necessarily in ways so labeled.

Sherry Ortner is one of the few who worked at expanding what Geertz must have meant.  She presented a study (1999) of Sherpas helping people climb the Everest and other Himalayan mountains as a study of, precisely, deep play given the well-known dangers involved in such endeavors.  But Sherpas, and the rich people who hire them to make it possible for them to reach the summit, may still be dismissed as exotic people living at the edge of our safety nets and of only passing interest.  And yet we, social scientists, should pay attention.

A few years ago, I got interested in deep play as something potentially central to culture theory in anthropology.  I was working with Mary Cotter analyzing a moment in a woman’s hospital labor when she tells as a joke about an earlier labor when she, a physician herself, had lied to an anesthesiologist about pain she was not feeling to get more pain killers, thereby endangering her and her child (Cotter 1996; Varenne and Cotter 2007).  Everyone present, anesthesiologist, husband, nurse, researcher, laughed.  The joke was certainly “play” in the usual sense.  Was her earlier lie also a gamble, deep play?

In a forthcoming paper in the Educational Researcher, Jill Koyama and I propose we answer this question positively and expand the relevance of “(deep) play” to include all acts, by an individual or a polity through its leaders, that threaten their status or the status of some of the people the act involves.  In our perspective, it does not matter what the motivation of the actor (gambler) may be, or the type of rewards the act might produce, or its initial cost.  What would matter is the level of uncertainty about the act producing what it is supposed to produce and the severity of the effects, should the act not produce what was hoped for.  It has made sense to think about the activity of traders of Wall Street as, also, gambling.  What about the activity of politicians and other regulators when act to, in what would be their term most probably, “generate reform of our educational system in fundamental ways” (from the introductory paragraph of the report A Nation at Risk, National Commission on Excellence in Education 1983)?

In this perspective, the decision by the New York City Department of Education to publish the names of individual teachers and their score, is a prime example of very deep play by policy makers and the politicians who accepted their recommendations.  Who made the decision is actually unclear, but it was forcefully defended by Mayor Bloomberg.  There is no evidence, “data based” or otherwise, that such a publication will achieve what reformers wish to achieve.  There is much debate about the validity of the figures.  But, somewhere, a decision in uncertainty was taken.  It involved ten of thousands who, willy nilly, are caught in its consequences and for whom the act might transform status in what Garfinkel once referred to, chillingly, as “successful degradation ceremonies” (1956).

And now, unsurprisingly, a slow motion drama is unfolding.  This drama, like the Balinese cockfight, is taking place in the public square, and so will many of the status changes that will result as the main bet, as well as many of the side bets that are also being made, is followed by its consequences.  Political careers will be made and lost.  Administrators will be fired while others will move up.  Unions might be reinvigorated, or not.  One can hope that the shame as much as half of the teachers of New York City are experiencing this month as they are publicly labeled “below average” will be short lived (though one can imagine the psychological scars some will bear as journalists and parents berate them).

It may seem insensitive for social scientists to watch the unfolding of the drama as evidence for refining our understanding of social processes.  And yet it is our duty to do such.  Publishing individual scores, whatever good might come of it, is, also a full scale experiment which would probably never have been allowed to proceed if it had been proposed as “research.”  University review boards are very vigilant about the extent to which individuals might be harmed by research, even unwittingly or in indirect ways.  But, of course, policy acts are not systematically reviewed for the harm they might cause.  Perhaps they could not be or political decisions could not be taken in a timely fashion.  But the issue remains and as social scientists who advise policy makers, we must pay attention and play our analytic role.  The publishing of the scores is a research boon as it allows us to test various analytic methods (as Koyama and I are doing) that will allow for understanding more systematically the networks of authority and power in which we are all caught, and particularly the relationship of motivation to act to social consequences.  In this effort we might do worse than facing the extent to which social life is, also, (deep) play, and then making suggestions to policy makers about what is most probably a fundamental feature of humanity.

comprehensive

On the archaeology of action networks

1 Comment

This morning, Gus Andrews sent me the following link: http://wrttn.in/04af1a.

There, in anonymous fashion, someone who signs “An engineer” gives a brief history of a case when “institutional archaeology” was needed because of the loss of “institutional memory” in a large corporation that required “reverse smuggling” of forgotten secrets by a person who knew the secrets but could not have the secrets told him.  Specifically, the corporation and its engineers had become ignorant of how one of their factories worked—even though it did work, and actually worked so well that they wanted to expand it!  But expansion required a knowledge that had been lost through several generations of reorganizations, partial digitization, retirements, etc.  So the corporation brought out of retirement one of the engineers who had designed parts of the machines.  The author of this brief case study has a good sense of all the ironies involved as the business, legal, and engineering parts of the corporations worked together, though in ways they could not mutually acknowledge, to keep the whole working (which it never ceased doing).  As Sacks would say “everyone has to lie” (1975).

This case is another nice demonstration that:
1) “things have agency” (Latour 2005);
2) writing instruction manuals is impossible because, as Garfinkel pointed out one cannot imagine all the settings within which the manual will be read (2002).  One has to add a temporal dimension to this: one cannot imagine which is the part of the manual that will become inaccessible as the manual, as object, disintegrates;
3) Terminator-like, the machines keep working, industry hums along, the humans scratch their head (that this is mostly the case is what Durkheim tried to capture that Latour and other critics cannot quite explain unless they join him in accepting that complex systems have a life of their own, that is they have “agency as things” and there must be mechanisms through which the machines “tell” or “instruct” the humans about what the machines need in order to keep functioning);
4) as the live humans scratch their heads to develop a meta-knowledge about the machines (above the everyday knowledge they may gain by interacting with the machine), they find ways (à la Rancière) to develop a new kind of situated knowledge (since the new version of how the plant currently works is not the same as the earlier version of how the plant would work).  For example, they hire retired engineers as consultants who themselves figure out where they put in their garage partial blue-prints they were not supposed to have taken out of the factory, and how to smuggle them back into the factory …);
5) those who read Lévi-Strauss (1966 [1962] ) must be told again that the distinction bricolage/engineering is a critique of rationalism: real engineers are always bricoleurs.

Latour, Bruno Reassembling the social: An introduction to actor-network theory. New York: Oxford University Press. 2005

Sacks, Harvey “Everyone has to lie.” in Sociocultural dimensions of language use. Edited by M. Sanches and B. Blount, 57-79 . New York: Academic Press 1975

on anthropological theorizing

On the (pre-)judicial brain

Leave a comment

The brain, we are told, acts first, thinks later–literally, by a factor of milliseconds.  That is, according to the kind of research David Eagleman summarizes in his Incognito: The secret lives of the brain (Pantheon, 2011), initial neural responses are faster than “conscious” ones.  This is the kind of research which, as I mused about earlier, confirms methodological individualists’ understandings.

That the brain should be so swift to act is actually essential to our surviving such tasks as, to pick up a recently evolved challenge to human survival, driving a car down a highway.  It is also problematic given that initial responses are easily correlated with various “biases.”  Brain researchers are getting quite good at devising subtle experiments to show how these biases might operate, at the millisecond level, even when the finalized act is non-biased (or biased in the correct direction).  I particularly “liked” (because of its extra-vagance) the experiment that depended on the trajectories of mouse movements towards “White people liking ‘Black People’” (Wojnowicz, Ferguson, Dale, and Spivey 2009).

All this could be used to confirm perennial working hypotheses about the relationship between strongly “learned” responses (probably because they happened repeatedly in early life) and later pre-judicial acts.  Such hypotheses have made sense to many Boasian “culture and personality” anthropologists, to sociologists and others following Bourdieu, and to a host of others over the past century.  In this perspective, the “judicial” brain literally does not, and indeed cannot, know what the larger brain gets the body to do.

For the critics among us, there is work to do here to challenge the neuroscientists on the implication of their altogether wild experiments.  At the simplest level, the point of having a “judicial” brain is precisely to control and repair what the pre-judicial brain may attempt to do, or has done.  The judicial brain might say “drive home” and then leave the pre-judicial one to do the driving.  At night, in Riverside Park, a white man’s pre-judicial brain may tighten muscles at the sight of young black man walking towards him, and it is probable that the black man will notice this tightening and this may produce a pre-judicial triggering of “racism.”  As they move away, each man may feel stressed and unhappy and actually review the encounter with their judicial brain—and then even perhaps blog about it.

But this last classic example raises an issue that neuro-scientific work appears very specifically to ignore: what are the implications of pre-judicial brain activity for routine social interaction?  Take driving down a highway.  Neuro-science tells us that we cannot describe in detail the sequence of muscular acts necessary to change lanes.  What it does not consider is that the first act when changing lane is figuring out whether there is another car in that lane and, if there is, whether one should accelerate or break before starting the change, and whether one should change an initial decision given what the other car is now doing as it noticed what we are doing.  Garfinkel (2002: 92-93) has made much of such joint activities across many bodies assembled on the highway that are required for the necessarily always emergent and yet “immortal” ordering of “driving down the highway.”

All sociologists working at the level of the “adjacency pairs” (and I include here interactional and conversational analysts, ethnomethodologists, etc.) should be the first to confront the neuroscientists since they work at very similar time scales but with radically different understandings of the units needed to analyze the same overall act at the next time scale when the act is concluded.  Other sociologists and anthropologists will have an easier time since we are almost always working with “consciousness.”   After all, our task is not to explain the tightening of muscles or the flashes of stress and anger black and white men passing each other may experience, but rather the evolution of the machineries (vocabularies, discourses, practices, laws, etc.) which make identify human beings as different from each other (from 19th century slavery, to early 20th segregation , to late 20th century civil rights and various kinds of resistances to it to any of these classificatory assemblages.

How the these assemblages influence the struggles within brains as persons ad-judicate each others pre-judicial movements, and change this adjudication as they find out what others have done, or what consequences they are drawing, should remain an open question that will not be answered by experiments that segregate human beings from human beings, and their pre-judicial movements from the delayed judicial ones.

References:

Garfinkel, H. (2002).  Ethnomethodology’s program: Working out Durkheim’s aphorism .  Lanham, MD: Rowman & Littlefield.

Wojnowicz, M. T., Ferguson, M. J., Dale, R., & Spivey, M. J. (2009). The self-organization of explicit attitudes. Psychological Science, 20, 1428-1435. doi:10.1111/j.1467-9280.2009.02448.x

comprehensive

Putting 2 and 2 together, and following up

2 Comments

Here is an addition to my gallery of educational events.  Here is the story of someone(s) who figured out that plastic bottles can provide lighting…

bottle in roof to be use as lighting

The link came to me through my son who got it from …. (probably a long list of referrals with mention of “amazing video”–and this is also an educational event)  Here is the link:

http://www.wimp.com/innovationfinest/

The story is told as one person coming up with an idea, refining it, and then convincing people in his “squatter area” (in English in the local language) to use it.  There was here a hint of network that transformed an individual act into a communal educational event.  And this, of course, is what is wonderful about the event. The editors of wimp.com provide a link to Isang Litrong Liwanak, a web site for the project.

This link reveals that, not so surprisingly anymore, the network has wildly expanded.  And that of course, suggest more educational events by more and more people.

This another case that would make a delightful research project.  Who did what when, for the first time, using what machines, and, most importantly, with whom?

on anthropological theorizing

Tequila and Mel Gibson’s brain

Leave a comment

As cohorts of doctoral students in anthropology at Teachers College know well, the second Thursday of the first year colloquium is dedicated to pondering “social facts” and rules for studying them.  What those students do not know at this point, and will struggle against throughout the session, is that they were set up to reveal a social fact: that the major points to be made are about the same points that were made the year before, and the year before that, going back to my first participation in the colloquium in a time that now appears to me “immemorial” (actually probably in the Fall 1991).  Specifically, the students will resist separating the social from the individual in the name of the individual (with due apologies to individual differences in the manner of this resistance).

They are of course in very good company.

Ah! The individual! The mind! The brain!

The same week, I read a somewhat popularized book by David Eagleman, a “neuroscientist.”  In his Incognito: The secret lives of the brain (Pantheon, 2011), he summarizes what excites people in his discipline.  I picked up the book after reading a summary of its argument: much that happens in the brain far below “consciousness.”  As Eagleman notes in his appreciative comments on Freud, this is not an original statement as such.  His point is that there is now a large body of experimental evidence that this is so.  Eagleman reviews evidence mostly from the fields of motion, perception, moral judgement, etc.  He does not review evidence from his colleagues concerned with language processing but this work would point in the same direction.  Driving a car, recognizing a face, making a value judgment, in the real time of everyday life (as it can be modeled in experiments) depends on processes that can be measured to happen on a scale much shorter than is required when any one of these matters are brought to consciousness.  This is also true of both the syntagmatic and paradigmatic processes we use when speaking.

And then he starts a major chapter by discussing Mel Gibson.

In an infamous incident in 2006, the actor is arrested for drunken driving.  As he is, he makes a violently antisemitic rant against the arresting officer.  In the following days he issues two apologies stating (through his publicist) “Please know from my hear that I am not an anti-Semite.  I am not a bigot” (Aug 12, 2006 – CBS News video).  Which is the “real” Gibson?  Elsewhere in the book, Eagleman reports on a study (Wojnowicz, Ferguson, Dale & Spivey 2009) involving moving a computer cursor:

Imagine that you start with your cursor positioned at the bottom of the screen, and in the upper corners of the screen you have buttons labeled “like” and “dislike”. Then a word appears in the middle (say, the name of a religion), and you are instructed to move the mouse as quickly as you can to your answer about whether you like or dislike people of that creed. What you don’t realize is that the exact trajectory of your mouse movement is being recorded-every position at every moment. By analyzing the path your mouse I raveled, researchers can detect whether your motor system started moving toward one button before other cognitive systems kicked into gear and drove it toward the other response. So, for example, even if you answered “like” for a particular religion, it may be that your trajectory drifted slightly toward the “dislike” button before it got back on track for the more socially appropriate response. (Eagleman 2011: 60-61)

What Eagleman never considers (and neither did the authors of the research) is the question of what makes a response more appropriate than another.  This also characterizes all his discussions of morality.  Actually it also characterizes his discussions of apparently purely cognitive tasks.  Early in the book he discusses plane spotters during the Battle of England of the Second World War.  But he does not ask: What led to this war? Why should spotting planes be important (and why might it not have been important during the First World War?

Now, of course, asking such questions led Durkheim to consider the possibility of “social facts.”  Asking similar questions leads recent sociologists and anthropologists claiming Durkheim as an ancestor to ponder the mechanisms for establishing that this is disapproved, or for enforcing the disapproval (policemen, anti-defamation leagues, etc.).  In Gibson’s case one might also ponder where the alcohol comes from (when, how, through whom did Tequila enter Hollywood?).  And, of course, how did antisemitism become something to sanction when the reverse had been true for so many centuries?

But, a keen doctoral student might ask, is it not the case that neuroscience confirms Bourdieu (and Parsons, etc.) on the foundations of sociability and the reproduction of social patterns?

More on that another day…

Wojnowicz, M. T., Ferguson, M. J., Dale, R., & Spivey, M. J. (2009). The self-organization of explicit attitudes. Psychological Science, 20, 1428-1435. doi:10.1111/j.1467-9280.2009.02448.x

comprehensive

detouring the internet and making it legal

Leave a comment

Two news items caught my eyes recently as more examples of the kind of activities I deem “educational” and that I encourage doctoral students to investigate.  Here is another set of “free dissertation topics” to expand our collection of case studies of what so many of our colleagues ignore.

The first item is from Afghanistan, the other from Guinea by way of New York City.  Both deal with people in serious trouble figuring out what are their conditions and trace a path through them (“enunciate their landscapes” to cross-reference Merleau-Ponty/de Certeau’s jargon).  In both cases the people are caught in mid-stream, successful-so-far in surviving, but with no guarantee that the future will be less difficult than their recent past has been.  Watching them struggling can teach us much about the world we, ourselves, also face–for it is the same one they do face.

Afghans working on laptop
Volunteers have built a wireless Internet around Jalabad, Afghanistan, from off-the-shelf electronics and ordinary materials (NY Times, 6/12/2011)

On Sunday June 12, the New York Times reported, on page 1 no less, that “U.S. underwrites internet detour around censors.”  The subject of this headline, not surprisingly is “the U.S.”  The subject is further specified in the first paragraphs as “the Obama administration,” “the State Department,” “a fifth-floor shop on L Street in Washington.”  It is, by the Time standard an “American effort.”  The picture that accompanies the piece, placed smack center at the top of the page, is of two Afghan men and an Apple laptop.  The laptop is set precariously on a chair on a roof.  It is tied to some machines.  One man sitting on the edge of the roof is peering down at the screen, one hand to his mouth, in the classical pose of the person who waits for a computer to complete an important task, or who is trying to interpret what the computer has provided.  The other man is peering into the distance through binoculars.  The legend below the picture says: “Volunteers have built a wireless Internet around Jalabad, Afghanistan, from off-the-shelf electronics and ordinary materials.”

Who are these “volunteers”? How were they trained? By whom? What exactly does it mean to “build a wireless internet”? Who else might also be building wireless internet that bypass state controls?  What are the other controls that may not be bypassed? How does one get an Apple laptop to Afghanistan (and the power to recharge it, etc.)?  (According to Google, there are Apple stores in New Delhi and Lahore, but apparently not in Kabul.)  There are many other subjects there than the “U.S.” and probably a lot of “fifth-floor shops” where the actual possibilities made available to local “volunteers” are explored, exploited, transformed.

The other story implies the existence of other kind of “fifth-floor shops,” this time in the Bronx “where many in New York City’s small Guinean population have blended in among other West African immigrant groups in neighborhoods like High Bridge, north of Yankee Stadium, Claremont and Morrisania” (NYT June 15, 2011).  The story is brief biography of the woman who is accusing Dominique Strauss-Kahn of sexually assaulting him.  The details that struck me is the following:

It is not clear how the woman gained entrance to the United States. In the 12 months ending in September 2002, the United States issued 4,410 visas to Guineans, a vast majority for business trips or tourism, officials said.  But by the time she began her job as a housekeeper at the Sofitel in 2008, she had legal status and working papers, her lawyers said.

village in Guinea
The village of Thiakoulle, Guinea, where the hotel housekeeper who accused Dominique Strauss-Kahn of sexual assault grew up (NY Times 6/15/2011)

How does one get legal status?  How does one do that, in the details of the peoples, bureaucracies, financial resources that must be involved?  How does one figure it out?  One can imagine the moment when it becomes clear that there are routes that can be followed to go from the village of Thikoulle, Guinea, to New York City, the efforts to find these routes among the maze of other routes that would take one elsewhere, and then the struggles to do whatever it takes to actually follow the route, and then, as is still going on for this woman—for there is no end to all this—, to recast one’s plans as one faces new obstacles and new possibilities.  At every moment and point the woman “learned” something and this something became almost immediately moot since the task that had been accomplished would not repeat itself, while new tasks appeared.

In this case, the one matter that would be worth investigating closely is the network of people who are involved in getting “legal status and working papers,” and to sort out what exactly one has to do.  I am sure that the State Department (or is it the Department of Homeland Security…) has web sites with instructions to follow.  But we also know (as per Garfinkel 2002) that these instructions cannot possibly be enough.  The bureaucratic rules have to be translated though not exactly into a different ‘language’.  Rather, they have to be re-written to become useful at the particular moment when a particular person has to perform whatever it is the rule says this person should do next.  Even more important to investigate are the processes when one finds out that the rules exist, and which of the millions of rules is the one a person should pay attention to now whether to imagine a possible route, or to get through the next gate along the way.  Doing something may just be secondary to finding out that it has to get done.

comprehensive, on assessment

On ecologically valid assessments

1 Comment

At some point during the mini-conference on the future of assessment (held on April 11, 2011), Ray McDermott raised questions about the validity of the kind of tests the Educational Testing Service and such design.  He told of the work he conducted in the late 1970s as part of Michael Cole’s Laboratory of Comparative Human Cognition.  Then McDermott, Cole and others wondered about the relationship between tests and the settings about which the tests were supposed to say something.  As they showed (1979, 1998: Chapter 1), the relationship between, for example, a reading test and baking banana bread by reading a recipe is tenuous, at best.  In the setting of a cooking club, so much else happens (from confused writing to interpersonal tensions) that ability to read is the least of the problem the children have to deal with.  The generalizability of these observations across settings and populations is now well established through repeated observations.

What has been left open in this work is the question of finding out what ecologically valid assessments would actually look like.

Soon after the conference, another participant, Katie Anderson-Levitt (U. of Michigan-Dearborn), suggested we look at Paradise and Rogoff’s recent paper about ongoing learning in families (2009).  In that paper, Paradise and Rogoff mention all the work done in the Cole tradition over the past 30 years with a new twist that fits well with my own sense of what I call ‘education.’  In everyday life, at home, “learning” is not a simple automatic matter proceeding below deliberation or symbolic expression.  In everyday life “teaching” (and assessing) is—probably—ubiquitous.

As I reflected on all this, I saw a route I have not yet quite explored and that could lead to further research expanding on the Cole, Lave, etc., traditions.  Starting with an expansion of the point Paradise and Rogoff made, I suspect that  the movement through publicized ignorance is accompanied by all sorts of speech acts, many of which fit in the paradigm of knowledge assessment.  Developing all this is also an expansion on Garfinkel, as I take him.

Garfinkel has kept arguing that maintaining any order requires ongoing work, including the work of figuring out what is going on.  Conversational analysts has given abundant evidence that this is indeed correct.  More recently, Garfinkel wrote about ‘instructions’ as a necessary aspect of this work.  The paper ends with one of my favorite quotes about screwing around and getting instructed (2002: 257).  What I do not think Garfinkel noted, and what I know I never noted myself, is that the instruction moments proceed either from an earlier assessment, or themselves constitute an assessment.  This is also an implication of Gus Andrews recent dissertation (2010) on blog comments when these are assessed as being “wrong” in some way that is specified by a later comment (“this comment does not belong here,” “you should not write your social security number here,” etc.).  In an interactional sequence (conversation?) utterance of the type “Do X differently!” are probably essential mechanisms for maintaining order, constituting emerging orders, moving participants into new positions, etc.

I am quite sure that such ongoing assessment is ubiquitous and should probably added as a function in Jakobson’s model of communication (1960 — though he might have classified it as an aspect of the metalingual function).  Much of the recent work on metapragmatics may also fit here.

In brief, and for our purposes, we could say that Ethno-methodology is at the service of ethno-science (what is the world made of?), and ethno-politics (how do we maintain the order within which we are now caught?), it also at the service of ethno-assessments. [or should we say that (ethno) Methodology is at the service of (ethno) Science, (ethno) Politics, and (ethno) Assessment?]

If this proves a useful direction for inquiry, it suggests that assessment is not an extra-ordinary task.  It also suggests how school assessment has drifted away from the ordinary [I am not sure that ‘drifted’ is the right work, but it will do for today].  The well known school-based QAE (Mehan 1979) model is formally equivalent to what might get known as the SARS model (Statement, Assessment, Re-statement) except that the former starts with the assessor’s question while the later starts with a seeker’s request that may then lead to an assessment (though this proposal may not have been presented as such).  In other words, the sequence starts with ignorance grounded in the here and now (“ecologically valid ignorance”?) and proceeds with statements of local knowledge that are themselves proposals for what it is that the seeker may plausibly not know (I am using the word ‘seeker’ rather than ‘learner’ since it will remain a question wether the subject whose ignorance is marked will learn anything out of the encounter).  This sequence is what I would now say my earlier statements about “productive ignorance” were about.

The question to designers of future tests is something like: how might you produce assessments that are triggered by acknowledgments of ignorance, whether generated by the subject (“I would like to know about X”) or by a co-participant in the polity (“you really should learn more about X”).  The challenge is to find the moment in the sequence of a life when the co-participant teachers will enter.  In everyday life it is a non-problem to the extent that co-participants or “consociates” have the built in or self-generated (legitimate) authority to assess (as siblings may have).  When social distance increases, that is when the network links between those who set what is to be assessed, what is to count as ignorance, and what should be done about include many persons in many institutions, then the problem gets acute.  It may even be unsolvable unless we find ways to reposition the official assessors within the network so they are closer to the performance in such a way that they can get a better sense, in real time, of the feedbacks that the seeker (learner) provides.

(More on what I am trying to formulate about network linkages later)

comprehensive, on assessment

Practical assessments, perhaps

1 Comment

This is my third entry developing some of the points we discussed during the mini-conference on the future of assessment (held on April 11, 2011).  The first two entries (on audiences, and on utopias), and in my initial one before the conference (on political philosophy) were essentially analytic with a definite pessimistic lean.  (School) assessments are bad for the health, and yet they are here to stay given all the powers they serve—including idealistic ones.

Robbie McClintock re-started me on a different track I briefly explored in my last contributions (2010) to the series on Comprehensive Education Ed Gordon and I edited (2008, 2009, 2010).  There I mused about possibilities for forms of institutionalized education that did not proceed from schools.  I am convinced that adults gain their most significant knowledge (about, say, health, emerging scientific and engineering developments, trends in high and popular culture, etc.) from institutions (for example, journalists and television reporters or producers) that are not controlled by the State.  Most of these institutions do not present themselves as primarily in the education business.  But perhaps they should, and draw the consequences.

McClintock emphasized the Web as another source of an education that is not necessarily packaged as, precisely, “education” but perhaps only as “information,” “entertainment,” etc.  Some writers and producers for the Web may present themselves as educating but none, at this moment assess what one may learn by reading their offerings.  Yet, as McClintock pointed out, the new technologies, as they are evolving, afford for possibilities for ongoing, real time assessment that older technologies (for example the printing press) do not afford.

This is intriguing.  Time for a little science fiction.

Imagine a new Google service.  At this moment, Google answers questions of the type “where can I find about X?”  Whether the seeker is satisfied or not with the answer, whatever the seeker does with answer, Google remains silent after providing a list of possible answers ranked by Google’s best guess as to the seeker’s intent.  If the seeker is dissatisfied, he may ask again and Google will answer, but Google has no memory of what this seeker asked and Google’s answers will not evolve as answers do evolve when, say, a child ask a parent about X.  So, last week, while writing my preceding blog entry, I looked for the creationism museum I had read about.  I entered “creationism” in the Google box (search page saved on of 4/20/2011), clicked on the first of 3,150,000 results (a suspiciously ‘round’ number), and found myself in the Wikipedia entry that started, on that day, with “creationism is the religious belief …”  There are no links to the museum in that entry, and so I asked Google again, found that it was second in its list of possibilities (and I also found out, serendipitously, a “Conservapedia” with an entry on creationism that is close but interestingly different from Wikipedia’s.  I am, of course, on my way to educating myself about creationism—not so much as a belief but as an institution with, among others, curricula experts quite deliberately teaching that which School people are quite sure should not be taught.  But Google never intervened in my education.

Imagine that Google did intervene.  Imagine that Google, as run by some revolutionary government, traced my queries, adapted its answers to my renewed queries and, mor or less insistently started asking me about my beliefs and worked at correcting them.  Parents do this with their children.  And school teachers do this with their pupils.  Why shouldn’t Google?

I suspect that the technological infrastructure of such a service is already in place and that it would not take much tweaking of the various flavors of social software already available to make them serve the new function.

The challenge is multiply institutional.  Who is to start it? Fund it? Control it?  Who is to credential (authorize) the persons or software assessing self-sought knowledge in real time?  Who is to establish the curriculum and it goals to which seekers are to be brought back even as they explored far and wide?  Individuals, clubs, associations, etc., may already provide feedback in the sense that their web sites are less informational than argumentative.  For example, the fifth set of links to ‘creationism’ on Google mostly lead to “atheist” web sites specifically addressing creationist links and debunking the claims.

I have never heard of States getting into this as deliberately as States get in the business of setting school curriculum.

The libertarian and anarchist in me whispers: why would you want the State to get involved?  The school critic from the left and the neo-liberal from the right might ask the same question.  States have a wonderfully awful (or is it awfully wonderful?) track record of setting the curriculum for the mass populations they govern.  Why would they do better with real time ongoing assessment than they have with the usual forms of test or examination based assessments for which schools are (in-)famous?

The statist in me begs to differ.  Free, state regulated, public schools have done much that is good in transforming what is to count as the knowledge on which public and private lives should be based, as well as the means for the production of such knowledge.  Rationally based, modernist (?) expertise should probably remain at the core of what States support, propagate, and assess as, precisely, rationally based.  If Latour (1993 [1991]) is correct, “We may never have been modern.”  Modernism itself is a cultural construction that must be reconstituted on an ongoing basis to remain the order of the day.  True enough, but what else might we wish to build?

What ongoing assessment software should be now build?

Musings about possibilities in the scholarly life of a professor of education and anthropologist