Posts in category: Kludgetastic

I noticed that early this month John MacFarlane wrote a retrospective on Twenty Years of Pandoc. I’ve used this incredible tool almost every day at least since 2011, when I set out to write a syllabus and a website for one of my first courses from a common template. It warms the heart of this procrastinator to learn that it began as a “product of procrastination” that just kept snowballing. Now it can bring endless avoidance-through-fiddling to all kinds of writing and document-producing. In fact it seems likely to me that pandoc really gave TeX a new lease of life, spreading it far beyond its well-entrenched home ground of the mathematical sciences to everywhere PDFs are needed—and in my case allowing me to indulge my nostalgia-toned techie/type-y compulsions while writing, teaching, and pursuing all kinds of other distractions. At this point it is easier to imagine the end of the world than the end of TeX, and that’s largely thanks to the streamlining and flexibility made possible by pandoc’s TeX-generating capacities. So bravo John MacFarlane, and bravo pandoc. May you thrive for many decades more, keeping the flame of useful abstraction and playful rigor alight in dark times.

Now days are dragon-ridden, the nightmare rides upon sleep, so here I am back fiddling with my computer configuration out of pure escapism. When I prepare slides for class or a talk, I write in markdown, using pandoc to generate the slides via Beamer. Beamer, you say, still? In 2026? Well, what do you want? I read somewhere (Finnegans Wake?) that Vico says people only understand what they themselves have made. I made an elaborate Rube-Goldberg pile of Makefiles, pandoc templates, pandoc Lua filters, hacky LaTeX tweaks, and so on and so forth, and it suits my purposes. I’ve been using it ever for a decade and more, and I keep hacking on it to keep up with whatever changes pandoc updates bring. I hadn’t tidied things up for the public repository in a while. Now I have. I give you, once again, Easy Lecture Slides Made Difficult: available for whatever use on github.com/agoldst/elsmd.

This should work with recent pandoc to generate slides and accompanying speaker notes from a single markdown file, by default in my preferred dark-on-light “scuro” theme.

I haven’t yet updated my R package that uses this for R markdown slides in my preferred style, but I will soon. I certainly have plenty I want to escape from.

Manual pin: My article, “Origins of the US Genre-Fiction System, 1890–1856,” is just out in Book History. Read all about it.

I made an R package with some “cultural” datasets of various kinds that might be of pedagogical use. It is available on github as agoldst/dataculture. See the repository page for a summary of the datasets, which I used to teach introductory analyses of:

1. cultural tastes over time and social space (names, music genres, recipes)
2. textual/paratextual signs of fictional genre (text-mining science fiction and crime)
3. historical and fictional social networks (Hamlet characters and eighteenth-century Bostonian troublemakers)

If that sounds interesting, take a look at the package and the lecture slides and lab exercises from the course I created it for, “Data and Culture” (Fall 2022). I’m hoping to save someone somewhere a few steps of wheel-reinventing.

The other day I was talking with an innocent bystander about some of my past work in the digital humanities. It occurred to me to wonder what a person who went looking for that work would find. The abyss also looks into you. Anyhoo, once upon a time I spent a lot of time working with data from JSTOR’s Data for Research service, a thing that no longer exists, and I produced two fairly elaborate programming projects related to topic models of text: my dfrtopics R package and my dfr-browser topic-model visualization. I am writing this post to announce that those things are still available and continue to shamble on, zombie-like, into the coming apocalypse. But I don’t plan to develop them further.

In the unlikely event that you, Internet Reader, wish to write to me, please note that Rutgers’s university e-mail system, provided by our friends at Microsoft at who knows what cost, aggressively misidentifies almost all non-Rutgers-origin messages (and many Rutgers-origin messages!) as spam. Over the last few months I have regularly been finding important messages from former students in my spam folder. I try to check my university spam folder regularly, but occasionally I neglect that particular Augean stable for too long. It seems Microsoft’s advanced machine learning technology is tuned to annihilate literally every message originating from Gmail.

I can be reached at first name dot last name at gmail. I cannot be reliably reached on social media.

For a troublingly long while I have been almost finished with an update to my software for exploring topic models, which consists of an R package, dfrtopics, and a JavaScript thingy, dfr-browser. This update is focused on making it easier to look at a single topic model more than one way or to toggle between topic models of identical or related corpora. Experience suggests that producing and interpreting even one useful topic model of a corpus is so much work that many people, myself included, are tempted to stop short of comparing the outputs of multiple models, even though this is an important way of checking that patterns revealed by any model are not purely artifacts.

I really, really hate pointing and clicking.

In my line of work—and in my social-media universe—I seem to see a lot of ranking of categories: most frequent words in a text corpus, most frequent genres in a bibliographic listing, most frequent nationalities in a group of authors… I’ve often wondered how much uncertainty lies behind rankings but had no idea how to quantify this uncertainty. I caught a link (via @coulmont) to a short essay on the subject: Michael Höhle, “Rank Uncertainty: Why the ‘Most Popular’ Baby Names Might Not Be the Most Popular” (available in OA preprint), the basics of which I was able to follow, thanks to its clear exposition and accompanying R code. Höhle explains that even when one has a full census of the names of all the babies born in a year, there is still uncertainty associated with ranking the names: the true propensity in the population to give a particular name is not fully revealed by the actual total. Höhle suggests we can begin by modeling the uncertainty arising from the fact that (1) even if the birth rate is fixed, there are “chance” variations in the number of births and (2) even if the probabilities of names are fixed, each draw from the bag of names has some randomness (little Jacob might well have been William). The rank uncertainty can then be found by simulating a year’s worth of names many times and finding popularity ranks in each simulation.

This post is a discussion and partial replication of Ted Underwood and Jordan Sellers’s fascinating essay “How Quickly Do Literary Standards Change?” (available in preprint and discussed on Ted’s blog). Hats off to Ted and Jordan, who have contributed something really remarkable here, not just in their provocative arguments but in the data they have made usable by others. It’s one thing to circulate data and code so that someone can in principle re-run your scripts—though that is already a lot—and quite another to make the data accessible enough for fresh analyses. The latter is a very demanding standard, too demanding for everyone to meet, I think. But it is what is required to let others build directly on your results. Even more importantly, it’s what’s needed to make research results pedagogically available. As I argue in an essay I’m working on now, any quantitative methods pedagogy should—must—lean heavily on the results of research. In the immortal words of Miriam Posner, “It’s just awful trying to find a humanities dataset”: one of the best ways to address this challenge would be to make good research data available in recirculable, easily accessible form.

So consider this post partly pedagogical in intent: I want to show that Ted and Jordan’s replication repository is already an excellent dataset and could be the basis for a lesson in automatic classification. What I want to emphasize here is that their work allows us to breeze right past the data-wrangling and straight into the analytical substance. This may not be entirely obvious from their python code, so I’m going to try to make it clearer by doing the whole thing in R instead.

Rather than give all the technical detail, I’ll only show the R code where it makes a point about the analysis. The full R markdown source for this post is in this repository. Feel free to skim right past the R listings anyway. This is a literary theory and data analysis post that happens to have some software details in it, not a programming post. Here we go…

It’s gratifying, and a little frightening, when someone else uses your own code. Jonathan Goodwin has built on my dfrtopics and dfr-browser code-blobs to produce a fascinating visualization of topics in fiction 1920–1922, derived by modeling the genre-specific word frequencies data set from HathiTrust. He’s given a nice description of his process as well. In the process, Jonathan revealed some unnecessarily restrictive assumptions built into my code. He solved the problem by modifying my code: all praise to him! But then I felt bad and wanted to make it possible for others to go further without having to dig into the Area X that is my code. So I made a few adjustments to my new version of dfrtopics. Here are some notes on using the updated version to cope with the issues Jonathan found in processing and modeling the Hathi data.