Previous studies have shown that it is possible to detect macroscopic patterns of cultural change over periods of centuries by analyzing large textual time series, specifically digitized books. This method promises to empower scholars with a quantitative and data-driven tool to study culture and society, but its power has been limited by the use of data from books and simple analytics based essentially on word counts.
This study addresses these problems by assembling a vast corpus of regional newspapers from the United Kingdom, incorporating very fine-grained geographical and temporal information that is not available for books. The corpus spans 150 years and is formed by millions of articles, representing 14% of all British regional outlets of the period. Simple content analysis of this corpus allowed us to detect specific events, like wars, epidemics, coronations, or conclaves, with high accuracy, whereas the use of more refined techniques from artificial intelligence enabled us to move beyond counting words by detecting references to named entities. These techniques allowed us to observe both a systematic underrepresentation and a steady increase of women in the news during the 20th century and the change of geographic focus for various concepts. We believe that these data-driven approaches can complement the traditional method of close reading in detecting trends of continuity and change in historical corpora.
Here are some of the findings we came across in this study. For more details on results, data sources and technical aspects please refer to our publication in PNAS given in the link at the end of the post.
A starting point for our study was to compare some results for our corpus with those for the Google books corpus, showing the similarities and differences between using a corpus of books and one of newspapers and highlighting that we can find the same trends.
Fig. 2. n-Gram trends showing (A) the steady decline of steam and the rise of electricity, (B) the waning popularity of horses and the increase in trains, (C) the rate of uptake for different communication technologies, (D) “the economy” as a concept beginning in late Victorian times after a decline in “political economy,” and (E) that the four largest peaks for “panic” correspond with negative market movements linked to banking crises in 1826, 1847, 1857, and 1866.
Fig. 1. Comparison between (A, C, E, and G) our corpus of British periodicals and (B, D, F, and H) the Google books corpus (1) using n-gram trends identifying (A and B) major wars, (C and D) coronations, (E and F) conclaves, and (G and H) epidemics between 1800 and 1950 in the United Kingdom. Events are clearly identifiable in the periodical corpus, whereas it is more difficult to distinguish exact years of events in the books corpus.
Fig 4: (A) Replicating the study (1) on famous personalities by occupation using all extracted entities associated with a Wikipedia entry, we found that politicians and writers are most likely to achieve notoriety within their lifetimes, whereas scientists and mathematicians are less likely to achieve fame but decline less sharply. (B) We computed the probability that a given reference to a person is to a male or a female person. We find that, although males are more present than females during the entire period under investigation, there is a slow but steady increase of the presence of women after 1900, although it is difficult to attribute this to a single factor at the time.
Fig. 3: n-Gram trends showing that (A) “unrest” corresponds with well-known periods of social tension, whereas “revolt” corresponds with tension in British colonies; (B) the suffragette movement falls within a delimited time interval; (C) “slavery” includes the movement for abolitionism and the American Civil War, whereas “anarchist” corresponds to the heyday of concern over anarchist direct action before the rise of fascism and bolshevism; (D) the gender gap in mentions of men and women is closing, with women making advances during the two wars; (E) the gender gap is also closing when measured using the pronouns he and she; (F) actors, singers, and dancers begin to increase in the 1890s, rising significantly thereafter, whereas references to politicians, by contrast, gradually decline from the early 20th century; and (G) football is more prominent than cricket from 1909 on.
The key aim of this study was to show an approach to understanding continuity and change in history based on the distant reading of vast news corpora, which is complementary to the traditional close reading by historians. We showed that changes and continuities detected in newspaper content can reflect properties of culture, biases in representation, or actual real-world events.
In particular, we showed that computational approaches can establish a meaningful relationship between a given signal in large-scale textual corpora and verifiable historical moments, and that newspapers provide increased clarity to the analysis of these events that may not be possible in other cultural forms, such as books. We further showed that the approach can reveal or confirm ways in which news media represent particular people or issues over time, as evidenced by the existence of a gender bias that is still present in the media today, and that historical trends in public discourse can be made accessible through the same means.
Publication
- Content Analysis of 150 Years of British Periodicals by Thomas Lansdall-Welfare, Saatviga Sudhahar, James Thompson, Justin Lewis, The FindMyPast Newspaper Team and Nello Cristianini in Proceedings of the National Academy of Sciences of the United States of America (PNAS).
