Mean-Field Variational Inference Made Easy
来源:互联网 发布:spider python 下载 编辑:程序博客网 时间:2024/04/28 08:38
I had the hardest time trying to understand variational inference. All of the presentations I’ve seen (MacKay, Bishop, Wikipedia, Gelman’s draft for the third edition of Bayesian Data Analysis) are deeply tied up with the details of a particular model being fit. I wanted to see the algorithm and get the big picture before being overwhelmed with multivariate exponential family gymnastics.
Bayesian Posterior Inference
In the Bayesian setting (see my earlier post, What is Bayesian Inference?), we have a joint probability model for data and parameters , usually factored as the product of a likelihood and prior term, . Given some observed data , Bayesian predictive inference is based on the posterior density of the the unknown parameter vector given observed data vector . Thus we need to be able to estimate the posterior density to carry out Bayesian inference. Note that the posterior is a whole density function—we’re not just after a point estimate as in maximum likelihood estimation.
Mean-Field Approximation
Variational inference approximates the Bayesian posterior density with a (simpler) density parameterized by some new parameters . The mean-field form of variational inference factors the approximating density by component of , as
.
I’m going to put off actually defining the terms until we see how they’re used in the variational inference algorithm.
What Variational Inference Does
The variational inference algorithm finds the value for the parameters of the approximation which minimizes the Kullback-Leibler divergence of from ,
.
The key idea here is that variational inference reduces posterior estimation to an optimization problem. Optimization is typically much faster than approaches to posterior estimation such as Markov chain Monte Carlo (MCMC).
The main disadvantage of variational inference is that the posterior is only approximated (though as MacKay points out, just about any approximation is better than a delta function at a point estimate!). In particular, variational methods systematically underestimate posterior variance because of the direction of the KL divergence that is minimized. Expectation propagation (EP) also converts posterior fitting to optimization of KL divergence, but EP uses the opposite direction of KL divergence, which leads to overestimation of posterior variance.
Variational Inference Algorithm
Given the Bayesian model , observed data , and functional terms making up the approximation of the posterior , the variational inference algorithm is:
- .
The inner expectation is a function of returning a single non-negative value, defined by
Despite the suggestive factorization of and the coordinate-wise nature of the algorithm, variational inference does not simply approximate the posterior marginals independently.
Defining the Approximating Densities
The trick is to choose the approximating factors so that the we can compute parameter values such that . Finding such approximating terms given a posterior is an art form unto itself. It’s much easier for models with conjugate priors. Bishop or MacKay’s books and the Wikipedia present calculations for a wide range of exponential-family models.
What if My Model is not Conjugate?
Unfortunately, I almost never work with conjugate priors (and even if I did, I’m not much of a hand at exponential-family algebra). Therefore, the following paper just got bumped to the top of my must understand queue:
- Wang, Chong and David M. Blei. 2012–2013. Variational Inference in Nonconjugate Models. arXiv 1209.4360.
It’s great having Dave down the hall on sabbatical this year — one couldn’t ask for a better stand in for Matt Hoffman. They are both insanely good at on-the-fly explanations at the blackboard (I love that we still have real chalk and high quality boards).
- Mean-Field Variational Inference Made Easy
- 变分推断(Variational Inference)-mean field
- Variational Inference入门:Mean field for the Ising model
- variational inference
- Variational Inference
- Variational Inference入门:variational bayesian EM
- 机器学习课程 Variational Inference
- 转--Approximate Inference(近似推断,变分推断,KL散度,平均场, Mean Field )
- Gaussian mixture model. EM VS variational inference
- Learning Variational Inference and Latent Dirichlet Allocation
- Optimizing Latent Dirichlet Allocation with Variational Inference
- 变分推断(variational inference)
- 变分推断(variational inference)
- PRML读书会第十章 Approximate Inference(近似推断,变分推断,KL散度,平均场, Mean Field )
- PRML读书会第十章 Approximate Inference(近似推断,变分推断,KL散度,平均场, Mean Field )
- Chapter 10.1 Variational Inference <Pattern Recognition and Machine Learning>
- 变分贝叶斯推断(Variational Bayes Inference)简介
- 变分贝叶斯推断(Variational Bayes Inference)简介
- Jquery autocomplete
- 滚动加载页面
- C++单例模式
- oracle中的sql%rowcount
- 智能家居趋势
- Mean-Field Variational Inference Made Easy
- 新建工程 Sloppy Joe's Menu 无法调试
- 1163 访问艺术馆
- oracle11g 静默安装(要点记录)
- WebSphere修改默认端口
- 20130910-STM8官方STVD环境编译链设置
- 转:IE和FF在对js支持的不同及解决方法
- Python下载网页图片
- 注解@PostConstruct与@PreDestroy