Big Data

Max Kemman
University of Luxembourg
October 11, 2015

Online slides optimised for Full-HD screens in full-screen mode
Download PDF here

Doing Digital History: Introduction to Tools and Technology

Recap from last time

What were aspects of an archive?

What are the three steps of digitisation?

What is the difference between data & metadata?

What meta/data do we have of letters?


  • Are digital libraries big data?
  • N=ALL
  • Messy data
  • From causality to correlation
  • Radical contextualisation
  • Next time

Are digital libraries big data?

Last week we discussed digital libraries/archives

Europeana contains about 53M digital objects

Is this big data?

What is "data" anyway?

Term has rhetorical function: "that which is given prior to argument" (Gitelman, 2014)

Common description: "raw data"

But creating data requires vast amount of work (as we saw last week)

Interpretive work into creating data

What is "big data"?

Metaphors used to describe big data give different interpretations (Awati & Shum, 2014)

  • Food: raw or cooked
  • Resource: oil, gold
  • Liquid: ocean, tsunami

What is "big data" anyway?

'Classic' definition by V's:

  • Volume: size
  • Velocity: accumulation
  • Variety: heterogeneous

Another definition: too much data to handle

Is this new?

Andrew Prescott (2015):

  • Domesday book
  • US Census 1890

What is "big data" anyway?

What is the difference between "lots of data" and "big data"? (Lagoze, 2014)

  • "Large" is historical: computers change
  • Big data makes us rethink what science is

Are digital libraries big data?

Or, does History have big data?

From the definitions so far:

  • Size: not so much (compared to CERN)
  • Velocity: not so much
  • Variety: yes!
  • Too much data to handle: probably
  • Makes us rethink what science/scholarship is: maybe

Is our collection of Hillary Clinton emails 'big data'?

Some say History/Humanities do not have big data

Why is big data interesting

BUT, why are we concerned with big data, but not with particle physics? (Wallach, 2014)

What are the 2 reasons she gives?

  • Social: big data are about people
  • Granularity: individual people and their activities

Here maybe History/Humanities do have interest in big data

Big data is a big topic

Another definition of big data (Mayer-Schönberger & Cukier, 2014)

  • N=ALL
  • Messy
  • From causality to correlation

Let's discuss these features


"N" refers to the number of observations done as part of the sample size

Sample: a group that represents the entire population

So N=ALL refers to measuring everything, rather than a representative smaller group

All historical sources?

A difference between "a lot of data" and "all data"

Remember Rosenzweig from week 1:

The injunction of traditional historians to look at “everything” cannot survive in a digital era in which “everything” has survived

Rosenzweig (2003)

Is size that interesting?

If big data is merely a quantitative difference, what's the interest?

But, quantitive can lead to qualitative difference (Mayer-Schönberger, 2014)

Quantitative to qualitative

Longue durée

Rather than focusing on a very short timespan, see development over ages

(Manning, 2013)

Messy data

Big data has Variety

A heterogeneous dataset

  • Different data-types
  • Different variables

Too much data to manually check

Can we use messy data?

Mayer-Schönberger & Cukier: size makes up for messiness

Exactness is from the age of spare information

The noise can be smoothed out


One way of trying to get someone to look at the data

Need to trust anonymous people

Does big data reflect the world?

With N=ALL, big data = reality, right?

But (big) data incorporates choices of what to measure

Twitter/Facebook are biased reflections of the world

Biases in language

Big data word-pairs (MIT Technology review)

  • Man - Woman
  • King - Queen
  • Brother - Sister
  • Computer programmer - Homemaker
  • Doctor - Midwife
  • Coward - Whore
  • etc

How big data is 'unfair'

The average person is a fiction

Hitchcock: it is the exceptions we are interested in!

Looking at the exceptions

Wallach agrees: use the granularity of big data to study minorities & exceptions

How do we discover the minorities & exceptions of interest?

To repeat; cannot look at all cases individually

Some statistical analysis is required

From causality to correlation

Correlation: two variables show a statistical relation

  • Positive: when A increases, B increases
  • Negative: when A increases, B decreases

Causation: one variable explains the second

  • Example: when it rains, more people take umbrellas with them

Correlation found

A nice example is Google Flu Trends:

  • Took flu data from national health center for number of years
  • Investigated which keyword searches occurred shortly before or during flu outbreaks
  • Use keyword searches to predict outbreak of flu

Correlation and causation

Important to remember: correlation does not equal causation

The keyword searches do not cause the flu!

Sometimes you don't know which variable comes first

Maybe a third variable explains the two measured ones

Meaningful correlation

Does the correlation mean anything?

Google Flu Trends later found not to produce accurate results

Spurious correlations

Spurious correlations

Spurious correlations

Find a correlation yourself:

Meaningful correlation

We cannot only use the statistics, we need to interpret them

But still we do not want to manually check all the possible correlations

Machine learning

Wallach describes herself as machine learning researcher

A simple introduction to machine learning (Geitgey, 2014)

Rather than telling the computer what to do, it learns what to do

  • Supervised
  • Unsupervised

Supervised learning

Provide enough answers to learn to give a new answer

Computer figures out how to go from data to the answer

Supervised learning

Or beat masters at chess or Go

Unsupervised learning

No given answer

Are there patterns? Outliers?

Train without knowing the rules

What do pregnant women buy?

How are sentences translated to different languages? (MIT Technology Review)

Biased algorithms?

Issues of biased algorithms:

  • Diversity in job applications
  • School drop outs
  • Predictive profiling of criminality
"We have no idea how these predictions are made"

Often criticism of algorithm, but where does bias come from?

Rethinking science/scholarship

How does this require a rethinking of scholarship?

Ways of reasoning (Dixon, 2012)

  • Induction: from the specific to the general
  • Deduction: from the general to the specific
  • Abduction: patterns


Rens Bod: discovery of patterns with tools is Humanities 2.0

Hermeneutic interpretation of these patterns is Humanities 3.0

Fickers: context more interesting than the data

Radical contextualisation

What is the context of each datapoint?

Hitchcock - contextualize using the big data


If content is king, context is its crown

Your search keywords make sense in your context

Radical context

Remember from week 1: what does this tweet mean as part of 31M?

Or actually: what does this tweet mean outside of Twitter?


Hitchcock describes the macroscope quoting Katy Börner

Macroscopes provide a "vision of the whole," helping us "synthesize" the related elements and detect patterns, trends, and outliers while granting access to myriad details. Rather than make things larger or smaller, macroscopes let us observe what is at once too great, slow, or complex for the human eye and mind to notice and comprehend.

Zooming in on people

If today we have a public dialogue that gives voice to the traditionally excluded and silenced – women, and minorities of ethnicity, belief and dis/ability – it is in no small part because we now have beautiful histories of small things. In other words, it has been the close and narrow reading of human experience that has done most to give voice to people excluded from ‘power’ by class, gender and race.

Close reading

Hitchcock argues for interchange of close and distant reading

Distant reading? That's the next lecture

For next time

18 October

Distant Reading

  • Aiden, E. L., & Michel, J.-B. (2013). The sound of silence. In Uncharted (pp. 69–83). Penguin.
  • Moretti, F. (2009). Style, Inc. Reflections on Seven Thousand Titles (British Novels, 1740–1850). Critical Inquiry, 36(1), 134–158.