Is Ethnography a psuedoscience?

[jimmy olsen]

If this article is true, sounds like it.

Ethnography is crap

The Ethics of Ethnography

Alice Goffman's heralded book about inner-city life has come under fire for inconsistencies and inaccuracies. Is the author to blame—or does the fault lie with her field?

By Leon Neyfakh

Late last month, a Northwestern University law professor published an article calling into question the veracity of a widely lauded book by Alice Goffman, one of sociology's brightest young stars. The book, On the Run: Fugitive Life in an American City, is an ethnographic study of a black neighborhood in Philadelphia where, according to Goffman's research, residents live in a mini–police state, constantly in fear of being arrested and sent to jail or prison, often for minor offenses. Goffman conducted her fieldwork, first as an undergraduate at the University of Pennsylvania and later as a graduate student at Princeton University, by embedding herself with a group of men from the neighborhood—they are all given pseudonyms in the book—and carefully tracking their lives over the course of about six years. The result is an extraordinarily detailed portrait of a community—nicknamed "6th Street" by Goffman and never identified by its actual location—in which the criminal justice system dominates people's lives and systematically cuts them off from opportunity.


On the Run received nearly universal acclaim upon its publication. But according to Steven Lubet, the Northwestern law professor, the book is seriously flawed. Lubet points to two anecdotes that he believes could not have happened as described and a third that seems to implicate Goffman in a felony. The article in which Lubet laid out his concerns touched off a debate both within the academic community and outside of it, one that spilled from sociology message boards onto the pages of the New York Times and caused some observers to wonder whether they were witnessing the opening scenes of an all-too-familiar story of intellectual deception, exposure, and professional disgrace.


Lubet's article, originally published in the New Rambler Review and adapted by the New Republic, may have touched off the public backlash against Goffman, but he was not the first critic to attack her credibility. A few weeks earlier, many in the world of sociology had become transfixed by a strange unsigned document that had been sent to scores of influential scholars in the field, along with Goffman's department chairwoman at the University of Wisconsin–Madison. In the document, which quickly made its way online, an unnamed critic made the case against On the Run over the course of more than 30,000 words and 45 numbered "problems," highlighting what appeared to be inconsistencies in the book's chronology and casting doubt on the integrity of the research behind it. The document appeared to be the work of an obsessive with an agenda—its tone was at times angry and bitter, and its reasoning could be muddled and hard to follow—but some of the author's observations seemed damning, or at least in need of explanation. Goffman felt compelled to write a line-by-line rebuttal and submit it to her department.

I spoke to Goffman on the phone on June 5, not long after the University of Wisconsin–Madison released a brief statement saying that it had found the accusations of academic misconduct that had been leveled against Goffman in the anonymous letter to be "without merit." With that statement in mind, I asked Goffman about what I considered the most problematic points in the letter and asked her to respond to them. From there we talked more generally about her book and how it compares to other works of nonfiction aimed at exposing the strife and degradation suffered by underprivileged populations.


I came away from the conversation with a sense that there are indeed factual inaccuracies throughout On the Run. However, they are not the product of the kind of fraud we're used to seeing in publishing scandals, and it would be unfair to say they place Goffman in the company of fabulists like Stephen Glass or data-cookers like Michael LaCour. That's because the majority of what I'm calling "inaccuracies" were introduced into On the Run because the conventions of sociological ethnography required them. In keeping with the methodological protocols of her chosen discipline, which typically demands that researchers grant their subjects total anonymity, Goffman changed details and scrambled facts in order to prevent readers from deducing the identities of the people she was writing about. In the process, she made her book all but impossible to fact-check.


The imperative to anonymize subjects is, in most cases, specifically mandated by the institutional review boards that approve social science research in American academia. And while it does seem possible to me that Goffman embellished some aspects of her narrative in order to tell a more compelling story, it was the steps she took to protect her sources—many of whom commit serious crimes in the book—that have made On the Run all but defenseless against skeptics like Lubet and the author of the anonymous letter. In other words, there is a good bit more than just the credibility of On the Run at stake here. At the heart of this controversy are the fundamental limitations of ethnography as a mode of inquiry. As practiced by many scholars, what is supposed to be a scientific undertaking aimed at systematically revealing truths about the world looks more like an uncomfortable hybrid of impressionistic data gathering, soft-focus journalism, and even a dash of creative writing.


Though it may be Goffman in the hot seat right now, any number of her colleagues could find themselves in a similar position, for the simple reason that producing research that is detached from reality to the point of being unverifiable is a central tenet of their discipline. As a result, some of the most vital and nationally relevant findings that come out of their field, including research on topics of urgent importance, like the conditions of inner city life, are vulnerable to questions about how much truth—and what kind of truth—they actually contain.


* * *


Before turning to the broader problems On the Run brings up, it's worth looking first at the book's alleged flaws. I'll start with the complaints brought up by Lubet, who suggested Goffman was guilty of several distinct varieties of malpractice.


The first was that she reported as fact stories she had heard from her subjects, without checking them independently. In particular, Lubet flagged an anecdote about an 11-year-old child being sentenced to three years of probation after being charged as an "accessory" to "receiving stolen property" for riding in the passenger seat of a stolen car. Lubet wrote that there was no way the anecdote could be true because there is no law in Pennsylvania against riding in a stolen car. After speaking with two public defenders and a prosecutor in Philadelphia, he speculated that Goffman was told the story by someone whose word she shouldn't have taken at face value. "I do not know what actually happened," Lubet wrote, "but neither does Goffman."


Lubet was also troubled by a scene that takes place in a hospital and involves a resident of "6th Street" called "Alex," who is arrested in the maternity ward where his girlfriend has just given birth to their child. As narrated by Goffman, police officers notice Alex's name on a visitor's sign-in sheet and, realizing there's an outstanding warrant for his arrest, take him into custody. It's a heartrending scene, with Alex's girlfriend begging the officers not to take away the father of her newborn and promising to turn Alex in the following day if only they let him spend the night at home. The police do not relent, and Alex goes to jail.


Goffman says she witnessed this interaction with her own eyes and was told in interviews with unnamed Philadelphia police officers that looking on hospital sign-in sheets for wanted criminals was a standard departmental procedure. Lubet doesn't buy it, having spoken to a source in the Philadelphia Police Department who called Goffman's account "outlandish" and denied that any local hospital would have ever shared its sign-in sheet with the police.


Goffman's response is simple: I was there. Lubet's is simpler still: Why should we believe you?


To that question, Goffman doesn't have a good answer. As Lubet points out, she so disguised the people and locations that appear in On the Run as to make her accounts effectively unverifiable: She changed the name of every person in her book and altered what she refers to in the preface as people's "identifying characteristics." According to Goffman, that meant changing the names of the places her characters inhabit and visit, changing people's ages and jobs, adjusting the number of people who were present at certain events, moving some of those events around in time. What's more, Goffman revealed in an interview with the Philadelphia Inquirer that she shredded all of her notebooks and disposed of the hard drive that contained all of her files out of fear that she could be subpoenaed and thereby forced to incriminate her subjects.


On their own, Lubet's suspicions about the hospital scene don't prove anything—maybe his sources are right about the way Philadelphia cops behave in hospitals, maybe they're wrong. (In a follow-up to his original article, Lubet wrote that he was willing to believe that some officers have occasionally "peeked" at hospital sign-in sheets; what he takes issue with is Goffman's assertion that using them as hunting tools is a widespread practice.) The problem is that, because of Goffman's efforts to protect her subjects' identities, she can do little to rebut Lubet's claims, other than to say, in essence, "Trust me." She has encouraged those who don't to talk to her old professors at Penn and Princeton, some of whom met her research subjects during the course of her fieldwork and saw her field notes before she destroyed them. I called the sociologist Elijah Anderson, who served as Goffman's undergraduate adviser at Penn and is now a professor at Yale, and Princeton sociologist Mitchell Duneier, who worked closely with Goffman as the chairman of her dissertation committee and conducted interviews with two of her subjects in front of his students. Both said they trusted Goffman. "She spent real time with real people—she did not make this stuff up," Anderson told me. Duneier said he spoke with her on a weekly basis about her findings, saw some of her field notes, and tested the plausibility of her conclusions by speaking to several of the people Goffman wrote about, including a warrant officer with the Philadelphia Police Department.           


When it came to the numerous accusations made against her in the anonymous letter, Goffman felt obliged to respond herself, in writing, to her university. The letter attacked Goffman from every conceivable angle, and while its sprawling, scattered nature doesn't make a great first impression, a handful of points do jump out as puzzling. Why was there an inconsistency in the number of funerals Goffman said she attended during her time studying 6th Street (at one point she says it was nine, but elsewhere she says it was 19)? Why were there two stories about a character named "Ronny" accidentally shooting himself in the leg, but one of them has him refusing to go to the hospital out of fear of being arrested, and the other describes his friend being arrested after dropping him off at the ER? And how could it be that Goffman's friend "Chuck" was killed in the summer of 2007, only to reappear in a different part of the book, in a field note dated 2009, driving a family member to a court hearing?


I put these issues to Goffman, who surprised me by saying she was glad I had singled them out. The first two were inadvertent errors, she said. Nineteen was a typo; there had been nine funerals. Ronny was in fact shot on two separate occasions, but only one of them was self-inflicted; Goffman said she had described both of them as such by mistake. Both errors will be fixed in future editions of the book, she told me.




The third discrepancy was of a different type, however. Goffman said that Chuck was indeed alive at the time of the court hearing. But the field note in which he is described giving his brother a ride was not written down in 2009—Goffman just labeled it that way in the book as part of the anonymization process. When it came to court hearings, she explained, she felt it was especially important to scramble dates because public records can be used with relative ease to identify cases and thus people. In this instance, Goffman said, her failure was in neglecting to make sure that the timeline as presented in the book was internally consistent. 


It's this last example that is most illuminating: unlike the other two, which seem to be innocent, if careless, mistakes, it illustrates the lengths to which Goffman went to change facts in order to protect her subjects. More to the point, that's exactly what she was supposed to do, according to the rules of ethnography.


* * *

So what are the rules of ethnography, and who enforces them?


To find out, I called several sociologists and anthropologists who had either done ethnographic research of their own or had thought about the methodology from an outside perspective. Ethnography, they explained, is a way of doing research on groups of people that typically involves an extended immersion in their world. If you're an ethnographer, they said, standard operating procedure requires you to take whatever steps you need to in order to conceal the identities of everyone in your sample population. Unless you formally agree to fulfill this obligation, I was told, your research proposal will likely be blocked by the institutional review board at your university.


Institutional review boards owe their authority to a 1974 law seeking to ensure that experimental medical research is conducted ethically. The law was passed in the wake of a scandal involving researchers who had studied the effects of syphilis on nearly 400 black Americans for 40 years without telling them they were infected or trying to help them get better. Though the law, and the federal regulations associated with it, were not primarily motivated by concerns about ethics in social science research, scholars in anthropology, history, political science, and sociology were nevertheless affected. As detailed by George Mason University historian Zachary Schrag in his 2010 book about the authority IRBs have come to exercise over the social sciences, academics who wanted to conduct research involving "human subjects" in any capacity—including just interviewing them—found themselves having to submit their research proposals for prior approval starting in the mid-'70s.


A campaign that began in late 1978 to grant social scientists certain exemptions from IRB oversight was temporarily successful, but during the mid- to late-1990s, federal regulators once again started applying the rules to social scientists, this time in an even more expansive way, roping in oral historians and folklorists. As Schrag reported in a 2009 paper, "Scholars who had been trained during the lull in regulations were astonished to learn that their work would now be subject to review."


Many academics who work in the social sciences see IRBs as needlessly restrictive and argue that they represent a challenge to academic freedom. These scholars don't like being told how to conduct their work and argue it's ridiculous that social scientists are being hamstrung by a system that was designed to protect participants in medical trials. "It is frustrating for many of us who want to write about particular individuals—because sometimes individuals want their names included," said Rena Lederman, an anthropologist at Princeton who has done ethnographic research in Papua New Guinea and now studies IRBs and ethical standards in academia.


The frustration is not merely a matter of academics resenting oversight out of principle. Many researchers think the uncompromising demand for total privacy has a detrimental effect on the quality of scholarship that comes out of the social sciences—in part because anonymization makes it impossible to fact-check the work.


"It makes it really hard to verify—you don't even know if the people exist," said Christopher Winship, a sociologist at Harvard University. He added, "The discipline thinks it's fine and that's probably totally wrong."


University of Chicago sociologist Richard Taub doesn't think it's fine and explained why: "Your honor—your word—is the only thing you have to make your stuff believable, because your job is to not let anyone track these people down," he told me. "It's a terrible problem."


Taub is among the ethnographers who would prefer not to anonymize their research to the extent IRBs oblige them to. He wanted to use actual place names in his 2006 book There Goes the Neighborhood, co-written with Harvard's William Julius Wilson, about four working- and lower-middle-class neighborhoods in Chicago, but decided not to because the authors knew the institutional review board at the University of Chicago wouldn't allow it.


Goffman's graduate school adviser, Mitchell Duneier, voiced a somewhat dissenting view about the inflexibility of IRBs when it comes to anonymity. "It's a case by case thing that IRBs decide," Duneier said, noting that he was able to use real names and places in two of his ethnographies, about street vendors in Greenwich Village and a restaurant in Chicago. The researchers who have the least amount of wiggle room when it comes to anonymization, he said, are the ones who study "deviant activity and criminal worlds," in which gaining access necessarily comes at the expense of transparency.


Goffman is one such researcher, and she readily embraced the anonymizing process. Indeed, she went further than she was required to by destroying her notes: According to Lederman, who sits on the IRB at Princeton, no such demands are placed on researchers at the school. When I asked Goffman if she regretted going as far as she did to protect her sources, she told me that, given how easy it is to triangulate small bits of information using public online resources, she wishes she had gone even further.


One confounding factor in the Goffman story is that On the Run both is and isn't an academic book; though it began as a dissertation and was originally published by the University of Chicago Press, it was later repackaged as a trade paperback by Picador with the intention of reaching general readers. One reason the book met with such success upon its publication is that it is extremely readable: Goffman tells stories and renders characters in engrossing detail, and the book largely lacks the jargon and disquisitions on theory you might expect from an academic text. Nevertheless, she adhered to the academic standards of anonymization to an extent that most readers of popular nonfiction would probably be surprised by. And while she does say explicitly in the book that she changed names and identifying characteristics, the fact that On the Run sits on the same shelf as rigorously reported and checkable works like Ghettoside: A True Story of Murder in America by Los Angeles Times journalist Jill Leovy and Random Family by Adrian Nicole LeBlanc (both of which are concerned with the nightmarish realities of inner city life) encourages readers to assume the books are following the same rules.


According to Goffman, her book is no less true than Leovy's or LeBlanc's. That's because, as she sees it, what sociologists set out to capture in their research isn't truths about specific individuals but general truths that tell us how the world works. In her view, On the Run is a true account because the general picture it paints of what it's like to live in a poor, overpoliced community in America is accurate.


"Sociology is trying to document and make sense of the major changes afoot in society—that's long been the goal," Goffman told me. Her job, she said, as a sociologist who is interested in the conditions of life in poor black urban America, is to identify "things that recur"—to observe systemic realities that are replicated in similar neighborhoods all over the country. "If something only happens once, [sociologists are] less interested in it than if it repeats," she wrote to me in an email. "Or we're interested in that one time thing because of what it reveals about what usually happens." This philosophy goes back to the so-called Chicago school of sociology, Goffman added, which represented an attempt by observers of human behavior to make their work into a science "by finding general patterns in social life, principles that hold across many cases or across time."




In light of that goal, you can see how the need to maintain strict adherence to real details could take a back seat. The point of On the Run isn't to tell the story of the individuals Goffman nicknamed Chuck and Ronny—it's to tell the story of all the people who are like them, who deal with what they deal with. As for how to know whether to believe her account of 6th Street's general patterns, Goffman said the best way to verify her work is to simply take a look around.


"The stuff I'm saying—it looks a lot like what people have been reporting from Ferguson, from New York, from Baltimore—for the past two years," she said. "When you read the Department of Justice's report on Ferguson—not just the violence but the level of police stops, the number of warrants that each family has ... it's exactly the same story. So in terms of fact-checking, you can say, 'Does this account look similar to what has been shown to be true for places like Ferguson or Baltimore?' I think that's the way to kind of figure out, when you're thinking about an ethnography, whether it holds up."


Harvard's Christopher Winship underscored the distinction sociologists make between particular facts and general truths: "If you told a sociologist they got a particular fact wrong, they'd say, 'Well, that doesn't matter—what's important is that it's true in a bigger sense.' We can talk about a piece of fiction as being true or not—as in, would real people actually act that way?—and I think sociologists and ethnographers fall back on that."


Should they though? Given that the work that goes into ethnography necessarily involves a close-up look at the lives of individuals, and the particulars of their experiences, shouldn't ethnographers feel compelled to report those particulars with precision and a strict commitment to reality? If those particulars aren't their data, then what is?


Princeton's Lederman doesn't see it that way. She pointed out that we don't expect survey researchers—scholars who conduct studies on large sample populations and present quantitative generalizations based on a statistical analysis of data—to know or care about the names of the individuals they question to generate their findings. Why should ethnographers be held to a standard of naming names? Lederman suggested that it is ethnographers' interest in developing qualitative generalizations—general truths—from the study of specific individuals that makes them similar to scholars whose work is more unambiguously recognizable as science.


And yet, works of ethnography like On the Run are very much focused on individuals. As Alice Dreger, a historian and the author of a recent book on ethics in science, put it to me: "If you want people to read for the big picture, then what you should write is the big picture. You should write, 'These are the general things I observed, these are the patterns I observed,' and not present it in the form of exquisite individual portraits as if those are true."


Goffman herself is the first to admit that she wasn't treating her "study subjects" as a mere sample population—she was getting to know them as human beings and rendering the conditions of their lives from up close. Her book makes for great reading precisely because it is concerned with specifics—it is vivid, tense, and evocative. At times, it reads less like an academic study of an urban environment and more like a memoir, a personal account of six years living under extraordinary circumstances. Memoirists often take certain liberties in reconstructing their lives, relying on memory more than field notes and privileging compelling narrative over strict adherence to the facts. Indeed, in a memoir I'm publishing next month, there are several moments I chose to present out of order in order to achieve a less convoluted timeline, a fact I flag for the reader in a disclaimer at the front of the book.


In retrospect, I regret moving those moments around—especially so after reporting on and thinking about Goffman's predicament. Though I know the changes I made were minor, my readers won't know what, exactly, I tweaked. Is that a price worth paying to make the story a little tighter? Upon reflection, I believe the answer is no: Even though my book is categorically different than Goffman's—mine is about my friendship with a struggling artist, hers is a work of social science about an urgent national problem—it seems to me that deviating from reality in a work of nonfiction inevitably leaves readers at a disadvantage.


* * *


One of the scenes in On the Run that feels most like memoir is also, now, one of the most controversial. It takes place—we're told—in the summer of 2007, shortly after Chuck, one of the closest friends Goffman makes during her fieldwork, has been killed by a member of a rival gang. In the aftermath of that tragedy, Goffman drives a character called "Mike" around Philadelphia, helping him look for Chuck's murderer. As she describes it in the book, she and Mike are in the car because they are seeking revenge; Mike is armed, and at one point he gets out of the car because he thinks (mistakenly) that he sees his target. Goffman describes the terrible feeling of knowing in her bones that she wants to see Chuck's killer dead and reflects on the profound effect of her immersion in the world of 6th Street. "At the time and certainly in retrospect, my desire for vengeance scared me," Goffman writes.


It was this scene that moved Steven Lubet to argue that Goffman had committed a felony while working on On the Run. In Lubet's eyes, participating in what sounded like an attempted murder plot represented a gross ethical violation on Goffman's part. (According to Pennsylvania law, Lubet noted, the fact that they never find or carry out their revenge against Chuck's killer would not absolve Goffman of the alleged crime.) As Lubet told me in an interview, "I think it has to be recognized that fieldwork should not include participating in an armed manhunt."


In a response to Lubet, Goffman rejected the notion that she had done anything illegal: The scene in question didn't depict a manhunt, she wrote—actually, it was a mourning ritual in honor of a friend, and at no point did she or anyone else in the car think they were really going to kill anyone. "Talk of retribution," she wrote, "was just that: Talk."


If Lubet had been setting a trap for Goffman by accusing her of a felony, it worked: With her response, Goffman seemed to confirm that she had dressed up the story in order to make it more dramatic. And in this instance, at least, she appeared to have done it more for narrative effect than to disguise her subjects' identities. If she had observed a mourning ritual practiced by the group she was writing about, why didn't she describe it as such? As Lubet put it in a follow-up article in the New Republic, "Goffman essentially admits that she embellished and exaggerated her account of a crucial episode, which should leave even the most sympathetic readers doubting her word."


I consider myself a sympathetic reader. I don't think Goffman is a fabulist, and intuitively, I believe that the stories she tells in her book do add up to a generally true picture of what life is like for young black men in inner city Philadelphia. But because Goffman's book can't really be fact-checked, I can't really know. And as a journalist accustomed to reading journalism, that leaves me confused about how I'm supposed to use the information in On the Run. What kind of information is it? What facts about the world is it telling me?


Of course, journalists sometimes withhold names or change identifying details in order to protect their sources, too. But unlike in ethnography, the default is not to anonymize: We try to deviate from reality rarely, and only after carefully weighing the benefits of telling a story and the costs of depriving the reader of essential facts and protecting sources from public scrutiny. The standards of the profession dictate that whatever scrambling of "data" is taking place, information is presented to the reader as accurately as possible.


I understand that it would have been harder—some might say impossible—for Goffman to tell the story she set out to tell if she had been forced to use real names and identifying details instead of changing them for the benefit of her subjects. But as it stands, the conventions of ethnography led her to produce a work that can't be trusted to reflect any reality, let alone a general one, and cannot be effectively defended against skeptics like Lubet and the author of the anonymous letter.

This is particularly sad because Goffman bore witness to a world that most of the people who read her book will never see firsthand, and the picture she paints of the criminal justice system's hold on the lives of young, black Americans in Philadelphia is disturbing. Indeed, as Elijah Anderson, Goffman's former adviser, pointed out to me, the findings of urban ethnographers are often doubted simply because to believe them is to confront some of society's most damning failures. That only makes their work more important—and makes it that much more imperative that their methods and findings be unimpeachable.

[Martinus]

I wish PDH was here to tell Tim that he is a goof.

[jimmy olsen]

I wish PDH was here to tell Tim that he is a goof.

I'd read an ethnography of Wyoming written by PDH any day.

[garbon]

So we disguise slate links now?

[jimmy olsen]

So we disguise slate links now?

The ethical guidelines of my field require it.

[Martinus]

So we disguise slate links now?

The ethical guidelines of my field require it.

If you followed ethical guidelines, you would stop teaching Koreans the language you don't speak properly yourself .

[jimmy olsen]

So we disguise slate links now?

The ethical guidelines of my field require it.

If you followed ethical guidelines, you would stop teaching Koreans the language you don't speak properly yourself .

Students of mine have made it to national competitions, my work speaks for itself.

[The Brain]

Not reading wall of text.

Many fields are largely pseudoscience. Gender studies etc etc etc. I'm sure ethnography has a lot of unscientific research going on, but I am not in a position to guess the exact proportions.

[PDH]

Ethnography has never been a hard science, it is as much art as it is science - the goal of describing culture means that the fluid nature of what is being described forces the researcher to both find commonalities/immersion at the same time as describing and being aloof.  This very dichotomy at the heart of ethnography means the the researcher must be trained, practiced, and willing to fail.

The biggest problem with such studies is that research is being done in a variety of fields that is being called ethnography, when in reality it is more akin to embedded reporting.  The mishmash of jargon, theory from various fields, and implied intellectual superiority makes some of these studies far less important.  Anthropology has always had the goal of multiple ethnographic studies by various workers on the same or similar groups - much of what happens today are micro-ethnographies that are non-repeatable.

When the right tools are used, when the proper intellectual rigor is demanded, and when the results are put out for peer review...then ethnography is a vital way of examining culture.  When they are not followed, then they are happy story time to get tenure.  Calling all ethnography a pseudoscience is neither worthwhile not intellectually honest.  But then it is a moron like Tim posting an article from an idiotic site.