查找相似项目
来源:互联网 发布:java crontrigger 编辑:程序博客网 时间:2024/05/21 20:22
Finding similar items (including LSH)
Given a large set of text documents, find pairs that are near duplicates.
Problems:
A. Many small pieces of one document can appear out og order in another.
B. Too many documents to compare all pairs.
C. Documents are so large or so many that they cannot fit in main theory.
Locality sensitive hashing
Three steps:
1.Shingling: convert documents to sets.
(used hashing to assign each shingle an ID)
2.convert large sets to short signatures, while preserving similarity
We used similarity preserving hashing to generate signature.
We used hashing to get around generating random permutations.
3.locality-sensitive hashing: Focus on pairs of signatures likely to be from similar documents.
We used hashing to find candidate pairs of
similarity >=s
1.Shingles example:
Hash the singles in order to reduce the space.
In order to find similar items(such as docs, pic.)We usually compare all of the items . So the time is vast. This is the motivation for the minhash/LSH
2.with the problem of 1.we:
A. Encode the set of shingles using 0/1 vectors.
B. Build the boolean matrix
The matrix is
We can finding similar columns using the following formula
But comparing all paris may take too much time, so hash columns(signatures) is put forward.
C.hash the columns. The result is also called signatures.
Minhashing example
So we achieve the goal. Compressed long bit vectors into short signatures.And the note is :
3.LSH
Goal: find documents with jaccard similarity at least s.
LSH(general idea): use a function f(x,y) that tells wether x and y is a candidate pair: a pair of elements whose similarity must be evaluated.
Method:
Split signatures into bands and hash them. Documents with similar signatures get hashed into same buckets: candidate pairs.
Here we use bands:
A. Partition M into bands.
B. Hashing the bands.
Example:
Assumption :
the pairs in the same bucket is called candidate pair.
The method can cause the false positive and false negatives.
1.false negative
2. False positive
So there is a tradeoff:
- 查找相似项目
- 相似图像查找
- LSH相似网页查找
- OpenCV2应用Meanshift查找相似物体
- 相似文档查找算法之 simHash
- 通过人脸识别查找相似图片
- 中文文本相似度---项目实践
- GitHub 更新:更新 timeline & 相似项目推荐
- 基于图片距离实现相似人脸的查找
- 海量数据相似度计算之simhash短文本查找
- 相似文档查找算法之 simHash 简介及其 java 实现
- 海量数据相似度计算之simhash短文本查找
- 基于图片距离实现相似人脸的查找
- 海量数据相似度计算之simhash短文本查找
- 海量数据相似度计算之simhash短文本查找
- 海量数据相似度计算之simhash短文本查找
- 海量数据相似度计算之simhash短文本查找
- 微软认知服务 人脸识别 API 之 查找相似
- WinCE的USB Device功能实现(Serial,RNDIS,Mass_Storage)
- 申请IDP图文教程
- C/C++的一点技巧:如何在控制台工程中使用CString类
- 几款富文本web编辑器的用后感
- Linux内存参数参考-与oracle相关
- 查找相似项目
- C#浅复制和深复制
- ORACLE编码修改
- 情商低是硬伤
- HEVC/H.265与AVC/H.264对比总结
- 解决Window桌面上的图标都变成白底问题
- JVM性能优化, Part 2 ―― 编译器
- 使用 CDO 发送传递状态通知 For VBScript
- #pragma once