Collapsed Gibbs Sampling

(The contents of this post are largely due to a conversation with Percy Liang at ACL.)

I’m a big fan of Gibbs sampling for Bayesian problems, just because it’s so darn easy. The standard setup for Gibbs sampling over a space of variables a,b,c (I’ll assume there are no exploitable independences) is:

Draw a conditioned on b,c
Draw b conditioned on a,c
Draw c conditioned on a,b
This is quite a simple story that, in some cases, be “improved.” For instance, it is often possible to jointly draw a and b, yielding:
Draw a,b conditioned on c
Draw c conditioned on a,b
This is the “blocked Gibbs sampler.” Another variant, that is commonly used in our community, is when one of the variables (say, b) can be analytically integrated out, yielding:
Draw a conditioned on c
Draw b conditioned on a,c
Draw c conditioned on a,b
This is the “collapsed Gibbs sampler.” In fact, we can often collapse b out entirely and, in cases where we don’t actually care about it’s value, we get:
Draw a conditioned on c
Draw c conditioned on a