Bloom Filters by Example
来源:互联网 发布:通道网络花店 编辑:程序博客网 时间:2024/05/20 16:44
http://billmill.org/bloomfilter-tutorial/
Bloom Filters by Example
A Bloom filter is a data structure designed to tell you, rapidly and memory-efficiently, whether an element is present in a set.
The price paid for this efficiency is that a Bloom filter is a probabilistic data structure: it tells us that the element eitherdefinitely is not in the set or may be in the set.
The base data structure of a Bloom filter is a Bit Vector. Here's a small one we'll use to demonstrate:
Each empty cell in that table represents a bit, and the number below it its index. To add an element to the Bloom filter, we simply hash it a few times and set the bits in the bit vector at the index of those hashes to 1.
It's easier to see what that means than explain it, so enter some strings and see how the bit vector changes. Fnv and Murmur are two simple hash functions:
Enter a string:
murmur:
Your set: []
When you add a string, you can see that the bits at the index given by the hashes are set to 1. I've used the color green to show the newly added ones, but any colored cell is simply a 1.
To test for membership, you simply hash the string with the same hash functions, then see if those values are set in the bit vector. If they aren't, you know that the element isn't in the set. If they are, you only know that itmight be, because another element or some combination of other elements could have set the same bits. Again, let's demonstrate:
Test an element for membership:
murmur:
Is the element in the set? no
Probability of a false positive: 0%
And that's the basics of a bloom filter!
Advanced Topics
Before I write a bit more about Bloom filters, a disclaimer: I've never used them in production. Don't take my word for it. All I intend to do is give you general ideas and pointers to where you can find out more.
In the following text, we will refer to a Bloom filter with k hashes,m bits in the filter, and n elements that have been inserted.
Hash Functions
The hash functions used in a Bloom filter should be independent anduniformly distributed. They should also be as fast as possible (cryptographic hashes such as sha1, though widely used therefore are not very good choices).
Examples of fast, simple hashes that are independent enough3 includemurmur, thefnv series of hashes, andJenkins Hashes.
To see the difference that a faster-than-cryptographic hash function can make,check out this story of a ~800% speedup when switching a bloom filter implementation from md5 to murmur.
In a short survey of bloom filter implementations:
How big should I make my Bloom filter?
It's a nice property of Bloom filters that you can modify the false positive rate of your filter. A larger filter will have less false positives, and a smaller one more.
Your false positive rate will be approximately (1-e-kn/m)k, so you can just plug the numbern of elements you expect to insert, and try various values of k andm to configure your filter for your application.2
This leads to an obvious question:
How many hash functions should I use?
The more hash functions you have, the slower your bloom filter, and the quicker it fills up. If you have too few, however, you may suffer too many false positives.
Since you have to pick k when you create the filter, you'll have to ballpark what range you expectn to be in. Once you have that, you still have to choose a potential m (the number of bits) and k (the number of hash functions).
It seems a difficult optimization problem, but fortunately, given an m and ann, we have a function to choose the optimal value of k: (m/n)ln(2)2, 3
So, to choose the size of a bloom filter, we:
- Choose a ballpark value for n
- Choose a value for m
- Calculate the optimal value of k
- Calculate the error rate for our chosen values of n, m, andk. If it's unacceptable, return to step 2 and change m; otherwise we're done.
How fast and space efficient is a Bloom filter?
Given a Bloom filter with m bits and k hashing functions, both insertion and membership testing areO(k). That is, each time you want to add an element to the set or check set membership, you just need to run the element through thek hash functions and add it to the set or check those bits.
The space advantages are more difficult to sum up; again it depends on the error rate you're willing to tolerate. It also depends on the potential range of the elements to be inserted; if it is very limited, a deterministic bit vector can do better. If you can't even ballpark estimate the number of elements to be inserted, you may be better off with a hash table or a scalable Bloom filter4.
What can I use them for?
I'll link you to wiki instead of copying what they say. C. Titus Brown also has an excellent talk on an application of Bloom filters to bioinformatics.
References
1: Network Applications of Bloom Filters: A Survey, Broder and Mitzenmacher. An excellent overview.
2: Wikipedia, which has an excellent and comprehensive page on Bloom filters
3: Less Hashing, Same Performance, Kirsch and Mitzenmacher
4: Scalable Bloom Filters, Almeida et al
- Bloom Filters by Example
- Bloom Filter 介绍(Bloom Filters by Example)
- Bloom Filters布鲁姆过滤器
- 论文推荐 Bloom Filters
- 原译:使用Bloom Filters
- HowTo do filters (Yahoo example)
- How are bloom filters used in HBase?
- How are bloom filters used in HBase?
- Retinex by Two Bilateral Filters
- Unity Shader Example 12 (Bloom 高光)
- Unity Shader Example 13 (边缘 Bloom )
- 正则表达式 by example
- JavaScript by Example
- Jini by Example
- JavaScript by Example
- XML by Example
- Perl by Example
- Scala By Example: 拍卖会
- (七)洞悉linux下的Netfilter&iptables:如何理解连接跟踪机制?【下】
- 数字图像处理5--边缘检测探究(内容较多,持续更新)
- Gradle入门系列(4):创建二进制发布版本
- 一纸沙华,半掩尘埃
- pat1041考试座位号
- Bloom Filters by Example
- 【Java基础】JPA入门例子(采用JPA的hibernate实现版本)
- Machine Learning Library for Python
- lintcode刷体--树的最小深度
- (八)洞悉linux下的Netfilter&iptables:状态防火墙
- Error in Microsoft Visual Studio 2012 :No exports were found that match the constraint
- (九)洞悉linux下的Netfilter&iptables:网络地址转换原理之DNAT
- Gradle入门系列(5):创建多项目构建
- 对象与指针的转换