Deep learning with word2vec and gensim

posted on September 17, 2013 by Radim | 11 Comments

Neural networks have been a bit of a punching bag historically: neither particularly fast, nor robust or accurate, nor open to introspection by humans curious to gain insights from them. But things have been changing lately, withdeep learning becoming a hot topic in academia with spectacular results. I decided to check out one deep learning algorithm via gensim.

Word2vec: the good, the bad (and the fast)

The kind folks at Google have recently published several new unsupervised, deep learning algorithms in this article.

Selling point: “Our model can answer the query “give me a word like king, likewoman, but unlike man” with “queen“. Pretty cool.

Not only do these algorithms boast great performance, accuracy and a theoretically-not-so-well-founded-but-pragmatically-superior-model (all three solid plusses in my book), but they were also devised by my fellow country and county-man, Tomáš Mikolov from Brno! The googlers have also released anopen source implementation of these algorithms, which always helps with uptake of fresh academic ideas. Brilliant.

Although, in words of word2vec’s authors, the toolkit is meant for “research purposes”, it’s actually optimized C, down to cache alignments, memory look-up tables, static memory allocations and a penchant for single letter variable names. Somebody obviously spent time profiling this, which is good news for people running it, and bad news for people wanting to understand it, extend it orintegrate it (as researchers are wont to do).

In short, the spirit of word2vec fits gensim’s tagline of topic modelling for humans, but the actual code doesn’t, tight and beautiful as it is. I therefore decided to reimplement word2vec in gensim, starting with the hierarchical softmax skip-gram model, because that’s the one with the best reported accuracy. I reimplemented it from scratch, de-obfuscating word2vec into a less menial state. No need for a custom implementation of hashing, lists, dicts, random number generators… all of these come built-in with Python.

Free, fast, pretty — pick any two. As the ratio of clever code to comments shrank and shrank (down to ~100 Python lines, with 40% of them comments), so did the performance. About 1000x. Yuck. I rewrote the explicit Python loops in NumPy, speeding things up ~50x (yay), but that means it’s still ~20x slower than the original (ouch). I could optimize it further, using Cython and whatnot, but that would lead back to obfuscation, beating the purpose of this exercise. I may still do it anyway, for selected hotspots. EDIT: Done, see Part II: Optimizing word2vec in Python — performance of the Python port is now on par with the C code, and sometimes even faster.

For now, the code lives in a git branch, to be merged into gensim proper once I’m happy with its functionality and performance. In the meanwhile, the gensim version is already good enough to be unleashed on reasonably-sized corpora, taking on natural language processing tasks “the Python way”.

So, what can it do?

Distributional semantics goodness; see here and the original article for more background. Basically, the algorithm takes some unstructured text and learns “features” about each word. The neat thing is (apart from it learning the features completely automatically, without any human input/supervision!) that these features capture different relationships — both semantic and syntactic. This allows some (very basic) algebraic operations, like the above mentioned “king-man+woman=queen“. More concretely:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

>>> # import modules and set up logging

>>> from gensim.models import word2vec

>>> import logging

>>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

>>> # load up unzipped corpus from http://mattmahoney.net/dc/text8.zip

>>> sentences = word2vec.Text8Corpus('/tmp/text8')

>>> # train the skip-gram model; default window=5

>>> model = word2vec.Word2Vec(sentences, size=200)

>>> # ... and some hours later... just as advertised...

>>> model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)

[('queen', 0.5359965)]

 

>>> # pickle the entire model to disk, so we can load&resume training later

>>> model.save('/tmp/text8.model')

>>> # store the learned weights, in a format the original C tool understands

>>> model.save_word2vec_format('/tmp/text8.model.bin', binary=True)

>>> # or, import word weights created by the (faster) C word2vec

>>> # this way, you can switch between the C/Python toolkits easily

>>> model = word2vec.Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True)

 

>>> # "boy" is to "father" as "girl" is to ...?

>>> model.most_similar(['girl', 'father'], ['boy'], topn=3)

[('mother', 0.61849487), ('wife', 0.57972813), ('daughter', 0.56296098)]

>>> more_examples = ["he his she", "big bigger bad", "going went being"]

>>> for example in more_examples:

...     a, b, x = example.split()

...     predicted = model.most_similar([x, b], [a])[0][0]

...     print "'%s' is to '%s' as '%s' is to '%s'" % (a, b, x, predicted)

'he' is to 'his' as 'she' is to 'her'

'big' is to 'bigger' as 'bad' is to 'worse'

'going' is to 'went' as 'being' is to 'was'

 

>>> # which word doesn't go with the others?

>>> model.doesnt_match("breakfast cereal dinner lunch".split())

'cereal'

This already beats the English of some of my friends :-)

Python, sweet home

Having deep learning available in Python allows us to plug in the multitude of NLP tools available in Python. More intelligent tokenization/sentence splitting, named entity recognition? Just use NLTK. Web crawling, lemmatization? Trypattern. Removing boilerplate HTML and extracting meaningful, plain text?jusText. Continue the learning pipeline with k-means or other machine learning algos? Scikit-learn has loads.

Needless to say, better integration with gensim is also under way.

Part II: Optimizing word2vec in Python