Week6-5Language Modelling2

Smoothing

• If the vocabulary size is ∣V∣=1M
  
  • Too many parameters to estimate even a unigram model
  • MLE assigns value of 0 to unseen data, let alone bigram and trigram.
• Smoothing(regularization)
  
  • Reassign some probability mass to some unseen data

How to model novel words?
-Distribute some of the probability mass to allow novel events

Add-one(Laplace) smoothing

• Bigrams: P(wi∣wi−1)=c(wi−1,Wi)+1cwi−1+V
• reassign too much probability mass to unseen data
• possible to add k instead of add 1

Advanced smoothing

• Good-turing
  
  • try to predict the probabilities of unseen events based on the probabilities of seen events
  • Kneser-Ney
  • Class-based n-grams

Good-turing

• Actual count c,
• Nc: total number of n-grams that occur exactly c times in the corpus
• N0: total number of n-grams in the corpus
• Revised count c∗=(c+1)Nc+1Nc

How do we deal with the sparse data?

Backoff

• Going back to the lower-order n-gram model if the higher-order model is sparse.
• Learning the parameter

Interpolation

• If P′(wi∣wi−1,wi−2) is sparse:
  
  • Use λ1P′(wi∣wi−1,wi−2)+λ2P′(wi∣wi−1)+λ3P′(wi)
• See [Chen and Goodman, 1998]