Cloud Computing(4)_Basic MapReduce Algorithm Design_Computing Relative Frequencies&Secondary Sorting
来源:互联网 发布:mac版本photoshop破解 编辑:程序博客网 时间:2024/06/06 16:41
How do we estimate relative frequencies from counts?
f(B|A):”Stripes”
Computing relative frequencies with the stripes approach is straightforward.
a->{b1:3, b2:12, b3:7, b4:1, ….}
- One pass to compute (a, *)
f(B|A):”Pairs” - “Order Inversion”
- What’s the issue?
- Computing relative frequencies requires marginal counts
- But the marginal cannot be computed until you see all counts
- Buffering is a bad idea!
- Solution
- 在本地统计(a,*),然后作为第一个数据发送给reducer,a是key
- Example
- For this to work
- Must emit extra (a,*) for every bn in mapper
- Must make sure all a’s get sent to same reducer(use parititioner)
- Must make sure (a,*) comes first (define sort order)
- Must hold state in reducer across diffrrent key-value pairs
Synchronization: Pairs vs. Stripes
- Pairs: turn synchronization into an ordering problem
- Sort keys into correct order of computation
- (a, *) is holded in the memory of reducer
- Stripes: construct data structures that bring partial results together
- Each reducer recieves all the data it needs to complete the computation
Secondary Sorting: Solutions
However, what if in addition to sorting by key, we also need to sort by value?
Example:
There are m sensors each taking readings on continuous basis.
Suppose we wish to reconstruct the activity at each individual sensor over time.
However, since MapReduce makes no guarantees about the ordering of values associated with the same key, the sensor readings will not likely be in temporal order.
Solution 1: “buffer and in-memory sort”
- 数据形式:m1 ->(t1, r80521), sensor id m1 is the key
- Buffer values in memory, then sort.
- It’s a bad idea: any in-memory buering of data introduces a potential scalability bottleneck.
Solution 2: “value-to-key conversion”
- 数据形式:(m1, t1) ->(r80521), sensor id m1 and the timestamp t1 are the keys
- Basic idea: move part of the value into the intermediate key to form a composite key, and let the MapReduce execution framework handle the sorting.
0 0
- Cloud Computing(4)_Basic MapReduce Algorithm Design_Computing Relative Frequencies&Secondary Sorting
- Cloud Computing(2)_Basic MapReduce Algorithm Design_Local Aggregation
- Cloud Computing(3)_Basic MapReduce Algorithm Design_Pairs&Stripes
- Data-Intensive Text Processing with MapReduce第三章(3)——COMPUTING RELATIVE FREQUENCIES
- Data-Intensive Text Processing with MapReduce第三章(4)-SECONDARY SORTING
- Cloud Computing(1)_Introduction to MapReduce
- Cloud Computing
- Cloud computing
- Cloud computing
- Cloud Computing
- Cloud Computing
- Cloud Computing
- Cloud Computing
- Sorting Algorithm
- Sorting Algorithm
- Sorting Algorithm
- hadoop map reduce secondary sorting
- Cloud Computing(0)_What is Cloud Computing?
- 数据压缩实验一——彩色空间转换
- 【LeetCode】488. Zuma Game
- 关于Apache服务器不能访问,无权限,forbidden的各种问题
- 初识jsp之登录—登录成功—登录失败页面
- 初识Struts 2
- Cloud Computing(4)_Basic MapReduce Algorithm Design_Computing Relative Frequencies&Secondary Sorting
- freebasic fbc 运行权限问题
- 在Java中动态传参调用Python脚本
- 加密ic 及 aes256 加密app
- 思维导图使用介绍
- 怎样使用android创建一个手机信息页面
- Linux内核分析(三)GDB调试工具分析Linux内核启动
- PAT L1-030. 一帮一 数组标记下标啊啊啊啊啊啊
- Android 绘制手势路线制作图画板并把View保存为图片