Posts in category: Kludgetastic

I have spent a lot of time experimenting with and exploring topic models of text. Aside from an article, some blog posts, and a bunch of strongly held opinions, that time also produced quite a few lines of computer code for handling topic models from MALLET. I started out with a big file of R functions, then escalated to a folder full of R functions. The organization got ever more byzantine, even more so when I collaborated with others. Finally I bit the bullet and, following the gospel of Wickham, converted my pile o’ scripts into an R package, called dfrtopics because I was making models of data from JSTOR’s Data for Research. There it has sat, on github, accumulating bits now and then, plus some function documentation written in a fit of compulsion, but really not in a form that anyone but I could use (and, as time went on, becoming hard for me to use too). My website has had a note promising a tutorial demonstration of how to use my package for nigh-on two years, but no demonstration demonstrated itself.

The package was hard to use and document because of the messy and ad hoc way I represented pieces of the topic model. A hierarchical model is not easy to wrap your mind around, and different questions require different slices of the model to answer. And all the mess of code passing around random collections of data frames, lists, and who knows what else seemed like fertile ground for errors and glitches, even when the whole thing seemed more or less to do what I wanted most of the time.

So: in a questionable expenditure of energy, I’ve spent a few days applying some polish to the package. Herewith dfrtopics version 0.2. Install it from github with devtools::install_github("agoldst/dfrtopics").

Three things are new from a potential user’s perspective—and my hope is that the idea of a potential user is slightly less far-fetched than before. First, there is now an introductory tutorial in the form of a package vignette. Second, the whole package has been rewritten around the idea that a topic model is an object, stored in a single R variable. Third, I have tried to make everything as modular as possible, so that the usefulness of the package is not restricted to MALLET models of DfR data. If you have other textual data in wordcount-form—oh, let’s say, 180,000 18th-through-early-20th-century volumes—you might write up some variant file-loading code if necessary, then use the package functions to model those texts with MALLET. And if you have wordcount data that you want to analyze in R in some other way than with MALLET, there are still, I hope, some useful things here for converting those wordcounts into data frames or term-document matrices.

In Easy Lecture Slides Made Difficult, I showed how to use markdown to make slides while still retaining some aesthetic flexibility. All that was required was a handful of TeX Macros, a little python script, a few Makefiles, and a maniacal commitment to automation. That was an enjoyable trip into the abyss, but what if your slides regularly include the results of calculations, data visualizations, or computational examples? Then it is time to, how do they say it these days, dive deeper. Thanks to R Markdown and knitr, it is possible to build on the pandoc/beamer system I described before to incorporate program code and its results. Call it, uh, Presentationally Literate Programming.

I am spurred to describe my approach by two things: first, I have had a whole semester of teaching Literary Data to work out the kinks; in that graduate course I regularly used R markdown-based slides to present new material. (Here are a couple examples: slides on network visualization, April 23, 2015; slides on topic modeling and PCA for literary texts, April 16, 2015.) Second, I was just at a conference where almost everyone’s presentation had slides with R output in them, but it seemed like people might…benefit…from an example showing how to make the aesthetics of the R output and the slides more consistent with one another. I’ll also show how the same R markdown can be the basis of slides, speaker notes, and a handout for audience members to take notes on. And I’ll show the settings for incorporating program code into the slides, which I used in my teaching. At the end of this post, I imagine you will question my sanity. The whole setup can be found in this repository subdirectory on github.

[Update, November 2015. I’ve made a slightly easier-to-use version of the same setup available as an R package here.]

A new OS upgrade means new Java problems. I use MALLET, which depends on rJava, which depends on Java. Here’s what I did to get things working (apparently, for now).

Markdown for slides is a great idea. Don’t painfully lay out each slide, click by click: just write down an outline and let a program generate the slides for you. The almighty Pandoc does this very well. You write plain text, and a single pandoc command produces slides in one of a number of nice HTML formats (like reveal.js or slidy), ready to be uploaded, or in PDF.

I switched over to this way of doing things last year. It was a relief to stop fighting with Keynote,¹ which is nice as presentation programs go but still exhausting: it constantly makes wrong guesses about formatting; putting text where you want is a struggle, and inevitably imprecise. And, worst of all, adding notes for myself to slides is pretty hopeless. I always ended up having to create two parallel versions of each class: the slides, and my own notes. That involved a lot of redundant work. By contrast, once I started writing my slides and notes together, I got my materials for class together quite a bit faster. Whether the temptation to further procrastination/tinkering was really healthy is a separate issue.

Anyway, markdown for slides with pandoc is pretty simple and easy-to-use, but because I wanted some features that the HTML formats do not easily support, I was tempted into going with pandoc’s capacity to generate LaTeX slides using the beamer package. Getting what I want out of that setup is what makes easy slides difficult. (If you want easy slides made easy, consider Ben Schmidt’s suggestions.) Here’s what I do these days. All the bits and pieces for doing this can be found in my repository of TeX stuff on github, in the lecture-slides directory. [Edit, May 18, 2016: I now use a slightly tidier set-up, available in a new repository.] For a sample of the kind of results I get, see the slides from my Early Twentieth-Century Fiction course this semester, for example a lecture on Djuna Barnes.

The website for Ted Underwood’s and my Quiet Transformations of Literary Studies has just been updated. Though this doesn’t add much in the way of new functionality (just a slightly modified bibliography display), every single chronological datum on the site has been changed to correct an error. Importantly, this means:

1. Every time-series chart is slightly different (more explanation below).
2. These charts now match the visualizations in our preprint article, which were not subject to this error.
3. Article bibliographic information more closely matches JSTOR’s.

In not entirely unrelated news, I learned more about date encoding in Javascript and d3.js. After the jump, an explanation of the bug.

And if that title doesn’t grab you, I don’t know what will.

Nerdery: how I’m doing latex + biblatex → html, in order to make this syllabus page.

Here’s a funny thing. This is a LaTeX document class file for a Yale Ph.D. dissertation: mythesis.cls. When I submitted mine, I discovered an older document class other people had worked on, and tweaked it to bring it up to the 2006 spec. Might be useful for current Yalies (if they somehow find this page by googling) or others TeXing their dissertations. Invoke with \documentclass[12pt]{mythesis}.

I’m adding it to the small github repository in which I’m going to start keeping tex sources for people to play with.

Coming next: a document class/style file/sample source for a student paper. One day I am going to persuade a student to TeX a paper.

I can dream, anyway.

I don’t know what causes it, but every now and then biber gives up working on my system. Then I start getting error messages like data source /var/folders/m6/bn7r45zx6cx55rr9g4qh6s6w0000gn/T/par-agoldst/cache-87e533530c5239fb9f8e3ff008979f1f16ea0e5e//inc/lib/Biber/LaTeX/recode_data.xml not found in .

There is a magic incantation, which I found somewhere on the biber sourceforge forums, to get biber to clear the cobwebs by reinstalling itself using perl’s PAR tool, as it does the first time it is run on a new system. Just clear the PAR cache, whose location under /var is helpfully specified in the error message:

 sudo rm -rf /var/folders/m6/bn7r45zx6cx55rr9g4qh6s6w0000gn/T/par-agoldst

You will have different stuff in /var/folders.../par-<username>.

Since I’ve seen other people complaining of this, I’m convinced it’s not my fault but the result of some deeply buried sporadic bug in biber. I’d try harder to figure it out, but it’s just too easy to make this one go away when it pops up once every few months. TeX is hacking, people!

If you work with tex4ht to convert LaTeX to OpenDocument (for subsequent Word conversion in NeoOffice, say), you may find yourself wanting to doctor an .odt file. At least I did; sometimes tex4ht outputs an odt with problems or syntax errors. But here’s the nice thing, if you need some quick odt first aid: as I learned from this article by Maarten Wisse, an odt is really just a zip archive.

> unzip test.odt
Archive:  test.odt
   creating: META-INF/
  inflating: META-INF/manifest.xml
   creating: Pictures/
  inflating: content.xml
  inflating: meta.xml
  inflating: settings.xml
  inflating: styles.xml

Mirabile dictu, those xml files are pretty easy to read. All the good stuff is in content.xml and styles.xml. You can burrow into these files wth a text editor, modifying style parameters or the way tex4ht has tried to tag your content. And when you’re done:

> zip test.odt content.xml
updating: content.xml (deflated 81%)

That’s all! If NeoOffice gives you an error when it tries to open a generated odt, it will tell you the line number of the syntax problem, and you can just fix it by hand.

All right, I know, kludge city. But very, very useful in a pinch.