Here are some tentative notes on methodological problems in quantitative approaches to literary history that I’ve been worrying over. I’m moved to post these because of two great discussions of quantitative literary studies by sociologists, a review by Ben Merriman of recent work by Franco Moretti and Matt Jockers, and a blog post by Tressie McMillan Cottom on quantitative text analysis across the disciplines. Monomaniacal proselytizer for closer relations between sociology and literary study that I am, first I say: go read those! They’re really great! Also, I’m leaping onto this bandwagon. Mostly I want to indicate how this very useful pressure from sociologists might help literary scholarship to refine its methods and clarify its research program. I’m afraid this is all very drafty, in the form of broad assertions with basically no citations. Blogging!
The trend towards quantitative methods in literary and historical scholarship now goes by the name “distant reading,” a term produced by the polemical wizardry of Franco Moretti. The label is meaningful, for good and ill. On the one hand, it signals (by design) a break with “close reading,” the methodological orthodoxy of literary study: almost all literary scholars take for granted that their task is to produce subtle interpretations of small selections of texts. To be a “distant reader” is to reject both this selectivity and the idea that it is whole individual texts that are the only suitable objects of study, rather than aggregates—as Moretti emphasizes, of units that are either larger or smaller than texts. On the other hand, “distant reading,” in its heterodoxy, also assents to the doxa of literary study, which is that scholarship consists in acts of expert “reading” (that is, interpretation) of texts of some kind. The result is what Merriman calls a “describe and interpret” approach to textual data analysis. As in the process of close reading, an object—most typically a visualization, sometimes a list, very rarely a table—is constructed from texts and then subjected to the hermeneutic circle, its details made salient and intelligible in relation to some context (the market for novels in the eighteenth century, the putative force of gender and nationality in shaping thematic choices, etc.).1
This ad hoc approach facilitates broad conjectures about wholes but makes it hard to give methodical answers to some crucial questions: how can we grasp the individual agent (micro) within large-scale structures (macro)? how do we give an adequate account of people’s experiences and interpretations of the social world and not just of their “objective” characteristics? and, the hardest problem of all, how do we explain the cultural-historical patterns we see? As Merriman points out, these problems are nothing other than the classic problems of sociology. I have been singing this tune for a while, so I was pleased to see matters looked similar from a position on the sociology side of the disciplinary divide. Though Merriman suggests that what is needed for “distant reading” to make progress on these problems is a general theory of literary change, it seems to me that nothing so grand (and so unattainable—a unified theory of literary change is about as plausible as a social physics) is needed. Rather, as Tressie McMillan Cottom argues in her post, a quantitative textual analysis has to take on board the social relations among people in which texts are produced and circulated, above all the relations of power which characterize all institutions of cultural production.
Trying to do that poses a methodological problem which “distant reading” struggles to solve. Speaking broadly, it needs further development in three areas:
explicit hypotheses about the mechanisms by which agents—I mean people, actual people—might produce the textual behaviors of interest;
systematic procedures for deciding whether the evidence favors some explanations over others;
disciplinary standards for judging whether an account of quantitative evidence is valid enough to be a starting point for further work.
Perhaps quantitative literary studies has so far done best with (1); the tradition of cultural studies has been particularly useful in teaching literary scholars how to generate speculative explanations for cultural change or continuity. But because the uptake of cultural studies has always been through the doxa of reading in literary studies, the only methodology (2) for assessing those explanations has been ad hoc judgments of interpretive power, not always easy to distinguish from sheer rhetorical élan. Here is where the generalization of close to distant “reading” tends to fail. A highly trained close reader has a strong intuition for what parts of an individual text need to be accounted for in an interpretation. When Moretti discusses Ulysses in Modern Epic, he knows very well that no account of that novel can omit a consideration of Joyce’s stylistic devices, above all the stream of consciousness. Whereas an interpretation of that novel’s relation to consumer culture that merely pointed out that Bloom sells advertisements would be hopelessly flatfooted, Moretti’s ingenious discussion of the relation between political economy and style answers the literary-critical imperative.
Neither Moretti nor anyone else in the field of distant reading systematically explains which features of a list, graph, or map need to be accounted for and which are negligible. The methods for doing so belong, of course, to the field of statistics, the “science of uncertainty.” There is just no getting around the fact that intuition is a poor judge when it comes to the behavior of aggregates and quantities, and all the fields of knowledge that deal with such aggregates apply a statistical corrective to intuition for good reason. I do not think, contrary to what is sometimes implied in DH discussions of “modeling,” that more elaborately “humanistic” representations of data can resolve the problem on their own. True, the more complex the categories in which the data is divided up, the more possible explanations for the data we can consider. So far, so good; without disaggregation, no explanation. But deciding among competing explanations for aggregate patterns will still be a task for a statistical analysis, not for intuition alone. This does not mean that everyone in literary studies needs to become an expert in statistics, though it probably does mean we should talk to statisticians from time to time. But it does mean learning which propositions are susceptible to statistical assessment and which are not, so that the kinds of hypotheses (1) we propose could be confirmed or challenged. Even that is a major task, and I cannot point to any work from within literary studies that shows how to solve the problem convincingly, though it seems to me that there is increasing interest in trying among those with quantitative leanings.
Yet though a statistical apparatus comparable to those developed in sociology or corpus linguistics (say) is indispensible for the progress of knowledge in this field, it will not suffice on its own to meet the third need. The question of validity (3) is not settled, though it may be clarified, by the use of some statistical technique or another. Rather, the judgment of validity will depend on the relation between the method of analysis, its theoretical presuppositions, and prior work.2 This is the place to say that qualitative literary studies (like other qualitative approaches in the social sciences) has its own ways to address all three of the demands I propose above, and part of the challenge for quantitative method is to build on what we already know. Validity judgments about distant reading are usually at a very high level of generality, the worst of which is “this can never tell us anything we don’t already know”; then again, “this has been an interesting ‘experiment’ and we should do more of them” is not much better. Rather, “can this work be the basis for further work?” is a better question: that is not just a question about the statistical power of the data but about the relation of one argument to past and future ones.
The issue of validity (3) has been frequently confounded with a rather different one, the political virtue or vice of “DH”: instead of asking whether valid knowledge has been produced about literary history (or some other domain of culture) by quantitative methods, one asks whether the attempt is compromised from the start. There is a long tradition in literary studies (even longer in criticism) of suspecting all quantification and aggregation of complicity with the destructive forces of commercial-industrial society—forces which, it flatters itself, it opposes. Needless to say I do not think this is a fruitful line of argument. Granted that there are political and moral dimensions to all pursuits of knowledge, including the most disinterested, it is nonetheless impossible to settle questions of method by a test of purity; we won’t find the right method by trying to find one which is never put to malign use. If it is necessary, we can remind ourselves of the fine reactionary heritage of literary hermeneutics, from Cleanth Brooks and Martin Heidegger to Paul de Man.
A more serious challenge aims at the questions being asked in the first place. McMillan Cottom, reading across Moretti’s body of work, notices that despite Moretti’s essays invoking World Systems Theory, neither Moretti’s quantitative work nor that of any other distant reader seems to have much to do with geopolitics or the global distribution of resources. Is this really, she asks, the theoretical “ground” of this research? If so, where has the impulse to study power (geopolitical or otherwise) gone in quantitative text analysis? The question takes on some urgency because many of the same people concerned with “DH” have also taken note of the information industries’ widespread neoliberal commitment to cloaking power in the supposed neutrality of measurement and the putative impersonality of markets. Then “DH” is suspiciously popular with administrators, funding bodies like the Mellon Foundation, and New York Times reporters. It is tempting to infer that popularity with the established powers is by design. As Adorno says about independently wealthy men in academic departments in Minima Moralia, “such suspicions are usually well-founded.”3 But if we bracket the hopelessly ill-defined umbrella term “DH,” then we might disentangle this suspicion from the methodological argument. It is not true that any effort to measure or model in the domain of literary history will necessarily promote the neoliberal ideology of measurement.4 A good counter-model could be found in Bourdieu’s use of large-scale survey data—while simultaneously criticizing the dominant-class ideology frequently lodged in survey categories—in Distinction. That work continues to generate further work of confirmation, qualification, and challenge by sociologists of culture, including rigorous quantitative analyses of possible causal mechanisms.5
And it would be good to dispense with the idea that it is literary studies’ (or the humanities’) special task to ritually repeat that all data are constructed by particular agents with particular interests and biases, as though this skeptical formula, a starting axiom of any serious research, were in itself a contribution to knowledge. On the contrary, one can and must measure and model the multiplicitous operations of power itself, under the same validity constraints as all other scholarly knowledge production. As I have said before, if understanding the relations of literary production to social power is one of the fundamental aims of recent literary study, quantification could be an important way to achieve that aim.
The real blockage lies elsewhere. I return again to my thesis that distant reading assents to the doxa of reading in literary study even as it contradicts the orthodoxy of close reading. It takes textual properties (for Moretti, preeminently stylistic ones) as the major objects of interpretation, and it considers that it has done its work when it says what a textual pattern means. This choice constructs a universe of texts which is only weakly connected to the institutional matrix that actually produces and circulates them. Where book historians speak of the communications circuit, distant reading has a short. It looks for patterns in texts instead of analyzing social systems in which texts (in their various material forms) play a role.6 Seeking to explain that systemic role is only “reading” in the most extended sense. The kind of analysis I have in mind deprives texts of much of their autonomy, simply by relegating the internal analysis of texts, on any scale, to the status of one piece of a different kind of puzzle. But whether literary scholars can organize themselves to solve that puzzle, at the cost of some of their long-standing claims to exceptional status, is another question.
- Peccavi. In my and Ted Underwood’s discussion of the history of literary study, our analysis leaned heavily on interpreting visualizations of the time series of topics in the topic model we constructed. ↩
- A useful comparison here might be to Piketty’s Capital. Though a certain amount of econometric modeling is buried in the notes and undergirds his “law” r > g (the details are beyond me), Piketty’s main claim to validity rests on the exacting construction of the right numbers for the comparative study of national and individual wealth over a long time span. The quantitative literary scholarship that has recently appeared has often been too vague about how it has constructed its data, and sometimes confused about the difference between documenting a “software carpentry” process and articulating what population the data are taken to represent, either as a sample or a census. Another example I often recur to is Stanley Lieberson’s A Matter of Taste, where again simple tabulations and time-series visualizations rather than elaborate models are enough to be evidence for Lieberson’s arguments. ↩
- In “For Marcel Proust.” Proust, and Adorno himself, were both the kind of person he was talking about. ↩
- Insofar as neoliberalizing ideology has effects in the domain of literary study, it is rather, I suspect, to be found at the organizational level. The grants-based funding structure of “DH,” with its imperative to specify deliverables on a relatively short time scale, produces isomorphic structures across all kinds of scholarly “projects,” which, whatever their putative political content, all have the same organizational design: a principal investigator, their subordinate non-faculty workers, their contingent networked organization, their web presence (in a design idiom which is often recognizable as that of the startup). Here Daniel Allington’s essay on digital humanities as the managerial humanities hits home. ↩
- An example I enjoyed: Edelmann and Vaisey’s Cultural Resources and Cultural Distinction in Networks, which uses some scary network-modeling statistics to show how musical taste works not just to facilitate certain kinds of friendship-by-homophily but also to maintain distinctions-by-aversion. ↩
- And “big data” is probably a red herring. A quantitative sociology of literature need not require huge datasets or fancy machine learning. Cf. Janice Radway, Wendy Griswold, John B. Thompson, and Pierre Bourdieu. (Bourdieu, ironically, would be something of an exception, since the multiple correspondence analysis which kinda-sorta supports his analysis of the cultural field was fairly computationally demanding.) Perhaps there are high-tech aids to content analysis that could be useful. But the adoption of the “digital humanities” rubric has had a kind of performative effect, so that aptitude with computers (programming, database engineering, website design, digital text encoding) has appeared a more important body of skills than statistical—or sociological—ones. I rebuke myself for this even more than others, having seen no other way than to put programming at the center of a course on literary data I taught this past semester. I would significantly revise that course if I returned to it—but I will say more about this on another occasion. ↩