Of Sonorous Medieval Chinese Texts and NLP Model Training

A distinctive set of challenges arises when training machines to process a historical language, especially one that was last spoken two millennia ago. One of the core issues embedded in natural language processing (NLP) models for historical languages is the acute lack of annotated datasets, despite the long scholastic and exegetical traditions for some of these languages. This article focuses on a specific historical language with an extensive commentarial tradition: premodern forms of Chinese spanning the Warring States (476–221 BCE) and early imperial (221 BCE–220 CE) periods. By highlighting the challenges of this particular language and our approach to building an NLP model that aims to overcome these difficulties, we will argue how textual commentaries from the medieval period can be used for NLP model training purposes.

It may at first appear that the goal of an NLP model for historical languages consists in building the largest model and extracting the highest accuracy score from it. Conventional wisdom holds that larger is better when it comes to datasets, particularly for modern languages, in order to build a seemingly complete picture of the target language.¹ It may be equally tempting to rely on the currently dominant focus in NLP by building contextual representations of meaning in the form of word embeddings. (This is nowhere more apparent than in the currently ubiquitous assertion that “attention is all you need” in the Transformer architecture.²)

But when approaching a language like Old Chinese, the architecture and hyperparameters³ of the model itself are, at best, of secondary concern. In reality, the bulk of time and effort is devoted to questions of data curation and the painstaking labor necessary to identify and annotate the unique qualities of this language.⁴ As we will see, the question of meaning in context is far more complex than counting the collection of words surrounding the one we happen to be interested in, as the Transformer architecture would imply. Additional considerations include questions like: Who will curate the corpus of texts? How will this corpus be made machine readable? What portions will be used for the training and evaluation of the model?

In the project we introduce here⁵ — the result of a collaboration between a software engineer and a philologist — we generate an NLP model based on data derived from premodern Chinese annotations. In many ways, we have found that questions facing the twenty-first-century researcher equipped with large language models and Python libraries echo those of the sixth-century scholar using brush and ink.

I.

Let’s begin by describing some of the distinctive features of Old Chinese, a language that survives in a corpus of ancient texts that can be dated to the centuries preceding and during the first dynasties of Imperial China, or roughly 476 BCE to 220 CE. These written texts survive either as documents that were transmitted and copied through the millennia, or as recently excavated or otherwise surfaced manuscripts.⁶

For heuristic purposes, we use the term “Old Chinese” for the underlying language, and like other stages of the Chinese language family, it is marked by the usage of Chinese characters or glyphs. As a writing system, Chinese glyphs have remained largely stable from the Han dynasty (202 BCE–220 CE) to the present day, with the greatest change occurring in 1956 in the form of the People’s Republic of China’s script reform and the introduction of simplified characters. A text from the early twentieth century may thus on the surface appear indistinguishable from a genuinely ancient piece of writing. This is in particular the case due to the venerated status of a few classical texts, largely from pre-imperial China, which served as models for later literary forms of writing up until the twentieth century. Existing NLP models for premodern Chinese assume a seemingly enduring and unchanging use of the written language, grouped under the notions of “Literary” or “Classical Chinese” (wen yan 文言 and gudian Hanyu 古典漢語).⁷ But this understanding of a never-changing and static language is not just ahistorical and incorrect; it also misses the point of what Chinese glyphs inherently represent: like other forms of writing, they are a conventionalized system used to represent the dynamic utterances of a language.

In this sense, sound is crucial to reading early Chinese texts, and the content of these texts is often inextricably linked to the topic of sound: the Chinese classics and texts of the genre of Masters literature (zi shu 子書) are astoundingly rich in puns, rhymes, and wordplay.⁸

Often, these texts present themselves as representing speech, and many of them may have had performative functions and poetic features that remain largely hidden, simply due to the fact that the Chinese writing system only poorly reflects its phonological features.⁹ The conceit of continuous access to written heritage is thus a mirage.

The sound system of the language underwent significant changes even as its graphemes remained static.

So how can we access the sounds of a language that lost its last native speakers millennia ago? Chinese — like other languages — has changed over time. More specifically, the sound system of the language underwent significant changes even as its graphemes remained static. Old Chinese phonology is therefore a reconstructed system, derived from documentary evidence. In this context, heteronyms pose an issue also present in modern forms of Chinese: characters whose pronunciation and meaning change based on context, defying attempts to convert them directly into phonemes.¹⁰ One of the best-known examples of such a homograph is the glyph 樂, which can be read — depending on context — as either lè (“joy”) or yuè (“music”) in modern Standard Chinese, which can be reconstructed as *[rˤ]awk or *[ŋ]ˤrawk in Old Chinese.

Another problem is the graphic variation common in early forms of Chinese, especially prior to the large-scale standardization of orthography during the third to first centuries BCE. In other words, while one glyph could be pronounced multiple ways, the same word could additionally be written with different glyphs in ancient texts. The same textual passage — say, a line of a poem — could hence be written with rather different glyphs.¹¹ Thus without a model that incorporates homographs, graphic variation, and other contextual understandings of phonology, the (often quite complex) puns, rhymes, and wordplays of ancient texts remain hidden below their surface forms. More crucially, however, unstandardized recurrences — not just of named entities, but of entire parallel passages — may go unnoticed.¹²

Fortunately, previous scholarship has grappled with many of these issues, and historical secondary sources offer an intriguing source of semi-structured data in the form of commentaries, dictionaries, and other scholarly work. The Qieyun 切韻, a rhyme dictionary compiled in 601 CE, is an important example: it records normative reading practices that represent a compromise between then-current Northern and Southern styles of pronouncing of glyphs in the classical texts from early China. Modern reconstructions of Old Chinese phonology draw heavily on the Qieyun and its later redactions, as these texts provide a formal structure and closed system of phonological distinctions for the underlying Middle Chinese.

The often quite complex puns, rhymes, and wordplays of ancient texts remain hidden below their surface forms.

The stability of the script also led to problems over time. In China, generations of medieval and early modern writers were trained to memorize the Odes of the ancient Shi jing 詩經 — called the Classic of Poetry for a good reason — but they soon grappled with the fact that these texts largely did not to rhyme when read aloud. The underlying phonology had changed.¹³

II.

Instead of annotating early Chinese texts manually to disambiguate these problems in reading a given passage, we decided to rely entirely on traditional Chinese scholarship on the ancient classics as a data source. In order for such premodern scholarship to be fruitfully utilized as a data source for NLP training purposes, and for such scholarship to address the problem of disambiguating different readings, several criteria must be met. Initially, the source text must provide phonological data as explicitly as possible — after all, our aim is to assemble a language model that meaningfully reflects not just the glyphs used, but also their underlying sounds. Secondly, older scholarly sources are better; however, unlike Middle Chinese, which offers a convenient textual starting point in the form of the Qieyun, approaching Old Chinese inevitably includes Middle Chinese data, given the significantly closer temporal proximity of medieval scholars — like the Qieyun’s compilator Lu Fayan 陸法言 (ca. 581–618) — to the ancient classics. Thirdly, data is needed on pronunciations given in realistic contexts in order to overcome the problems of homographs and graphic variation. That way, a future model of ours can use this contextual information to improve its accuracy.

One early work appears to fulfill all of these criteria: the Jingdian Shiwen 經典釋文, completed in 583 CE by the scholar Lu Deming 陸德明 (d. 630). This monumental commentary provides tens of thousands of phonological, semantic, and bibliographic notes across a representative selection of sixteen ancient classical texts. The material being annotated is broad, ranging from poetic odes (the Shi jing) and historical chronicles (the Chunqiu 春秋 and its commentaries) to an ancient dictionary (the Erya 爾雅). Lu Deming’s analysis draws from some 230 sources, some of which are not attested in any other work.¹⁴

Lu Deming’s meticulous attention to detail produced what is effectively a machine-readable dataset millennia before such machines would exist.

The Jingdian Shiwen wrestles with some of the same problems we face today, as its commentarial style disambiguates homographs and the inconsistencies presented by graphic variation in the texts it studies. While Lu Deming lived during a tumultuous period that spanned multiple imperial dynasties and spawned many competing schools of thought on how the classics should be read, his explanatory annotations eventually received official recognition and earned him posthumous fame and a commendation from emperor Taizong 太宗 (598–649) of the Tang dynasty 唐 (618–907 CE).¹⁵

The Jingdian Shiwen utilizes a relatively novel form of commentary: rather than reproducing the source text in full, it instead lists only what we call headwords, which are short excerpts, ranging from single glyphs to short passages. Each of these short sequences of glyphs is paired with a corresponding annotation. Each headword is distinctive enough to be matched to its location in the full text of the original source. The Jingdian Shiwen is thus a semi-structured text that provides sequences of glyphs that can be located in specific contexts in the source texts, and supplies annotations for a specific glyph in the relevant sequence.¹⁶ By essentially compressing the source texts in this way, the Jingdian Shiwen manages to cover almost 900,000 characters of primary-source material in just over 100,000 characters of excerpt. The resulting “compression ratio” is 13:1.¹⁷

While earlier dictionaries primarily glossed glyphs by providing similar-sounding glyphs to indicate their reading, the Jingdian Shiwen employed a rather novel way of indicating pronunciation: the fanqie 反切 system.¹⁸ This method of noting a glyph’s phonology separates a syllable into its initial consonant on the one hand, and its rhyme and tone on the other. No longer constrained to providing pronunciations by finding a word that overlapped exactly in sound, the fanqie system allowed scholars such as Lu Deming to instead choose common graphs for the initial and rhyme plus tone independently. Given the reliance on the Chinese script, both initial and rhyme plus tone are expressed through a common glyph.

In this way, the Jingdian Shiwen is both comprehensive and concise in the way it provides phonological data in context. We believe it provides enough data to train an NLP model. The key question is then how to extract this data; while Lu Deming’s meticulous attention to detail produced what is effectively a machine-readable dataset millennia before such machines would exist, adjusting the specific format still necessitates significant labor on our part.

III.

The first obstacle we face in turning the Jingdian Shiwen into phonological training data is the need for a digitized version of this text. Fortunately, the Kanseki Repository, an online database of premodern Chinese texts, offers digitized editions of over nine thousand works, including the Jingdian Shiwen and the source texts it annotates.¹⁹ This repository usually offers multiple editions for each work, in addition to interpretive versions that attempt to merge the editions into a state-of-the-art copy of the text. In keeping with the repository’s permissive licensing and spirit of “electronic texts by researchers, for researchers,” our own work is available under an open license and published on GitHub.²⁰

Only with such a digitized version of the Jingdian Shiwen and related texts can we approach the key question of how to extract useful phonological data from Lu Deming’s commentary. A look at the content of this text highlights why this question is crucial: while close to one-third of the roughly 55,000 notes in the Jingdian Shiwen consist solely of a reading gloss (thus providing a reading aid to indicate pronunciation), the remainder is more complex and addresses multiple concerns. Some annotations feature semantic glosses, and others highlight instances in which additional works reproduce a glyph differently or include citations to the interpretations of other scholars. Many annotations combine these different elements. More importantly, the Jingdian Shiwen contains yet another form of abridgement: instead of reproducing the same annotation multiple times, the text attaches qualifiers to indicate that a given reading applies every time a human reader encounters the given string of glyphs in a specific section of the source text. These qualifiers act as multipliers for the data, effectively extending the commentary to cover whole swaths of text not explicitly noted elsewhere.

Our approach to handling these complexities is to train a special-purpose model equipped to parse the terse style of the Jingdian Shiwen’s highly structured annotations. We use the Prodigy annotation tool to note parts of speech and syntactic relationships in the commentary, and pair it with the spaCy NLP library to create a custom processing pipeline.²¹ By applying this micro-model to the annotation corpus, each individual reading gloss can be extracted and paired with qualifying data. A notable side effect of this approach is that it simultaneously produces a citation network dataset: references that the Jingdian Shiwen makes to other texts and authors can be extracted from the text along with phonological data.

Once we have extracted all of the relevant phonological data, the task still remains to transform it into reconstructed forms of first Middle Chinese, and then Old Chinese. This is a process involving a few considerations: while the reading glosses of the Jingdian Shiwen — given directly and in fanqie form — reflect Middle Chinese, these glosses can also be used to make some inferences about the earlier Old Chinese. In order to strengthen these inferences, we use the reading glosses provided by the Jingdian Shiwen as disambiguation data and cross-reference these glosses with both contemporary rhyme dictionaries like the Qieyun and modern historical linguistic data (primarily William H. Baxter and Laurent Sagart’s 2014 reconstruction of Old Chinese²²). The overall goal is to stay as true as possible to the source material: if the rhyming portion of a syllable can be determined, but its initial consonant is represented ambiguously in the Jingdian Shiwen, we attempt to capture that ambiguity in the data.

We borrow a strategy from the modern science of bioinformatics: specifically, genetic algorithms designed to find the best overall “alignment” of sequences of amino acids (or, in our case, sequences of glyphs).

Finally, in order to create a fully machine-readable dataset, we reunite the annotations of the Jingdian Shiwen with the full text of the classical works that it annotates. Because of the Jingdian Shiwen’s tendency to economize on space by attaching comments to excerpted headwords, identifying the precise locations in the text where the headwords occur is necessary to match the annotations to their original context. To do this, we borrow a strategy from the modern science of bioinformatics: specifically, genetic algorithms designed to find the best overall “alignment” of sequences of amino acids (or, in our case, sequences of glyphs).²³ Once the headwords have been matched to their original positions in the source text, we copy over their corresponding phonological data to obtain a dataset of reading glosses in their historical context.

In using historical secondary sources for training an NLP model, we have found that even sources compiled millennia ago can still be considered by their very nature semi-structured data of the type needed to construct a model. The constrained, formulaic syntax of dictionaries and commentaries — an unnatural language designed to be quickly parsed by a human reader — lends itself equally well to parsing by relatively simple algorithms. For historical languages, secondary sources may even form a substantial part of the entire known body of the language, providing invaluable contemporary context to their literary counterparts.

Our project continues the practice of reading classical texts as data, but with a different aim than previous iterations of this practice: our goal is to produce a machine-learning algorithm. Some of the outlined steps that we have used to parse the Jingdian Shiwen are still works in progress, and additionally, some of the most difficult work remains to be approached. Constructing a statistical model that can represent all the complexities of current hypotheses regarding Old Chinese syllables will strain the limits of contemporary NLP platforms, most of which have no concept whatsoever of phonology. Our aim here is to persuade our peers that this task, and others like it in other languages, is not just possible but wholly worthwhile for researchers in the humanities. For this reason, we believe that further collaboration — with digital humanists, philologists, and others interested in expanding the debates around ancient texts to incorporate sound — is one of the most generative approaches to making use of NLP frameworks in the study of ancient texts.

Acknowledgments

This work was made possible through our participation in the “New Languages for NLP: Building Linguistic Diversity in the Digital Humanities,” a National Endowment for Humanities Institute for Advanced Topics in the Digital Humanities. Our thanks to the organizers, Natalia Ermolaev and Andrew Janco, and to Toma Tasovac, Quinn Dombrowski, and David Lassner. We also thank Gissoo Doroudian and Rebecca Sutton Koeser for their design and implementation of the figures of this article, in a lively exchange with Nick Budak. Figure 4 was further based on the work of Jeffrey R. Tharsen, whose work more generally has inspired many of the thoughts in this article. Lastly, thank you to Grant Wythoff for his erudite editorial work.