Euro-par 2013 计算机体系结构相关国际会议 review评审意见

Euro-par 2013 计算机体系结构相关国际会议 review评审意见 （信息供参考学习）

我投的一篇与NUMA服务器上虚拟机性能评测相关的论文，录用结果是reject。供大家学习经验教训吧！

----------------------- REVIEW 1 ---------------------
PAPER: 134
TITLE: The Impact of Virtual Machine Scheduling on NUMA multicore Systems
AUTHORS: Yuxia Cheng, Wenzhi Chen and Shaoyu Zhang

OVERALL EVALUATION: -2 (reject)
Relevance to the topic: 3 (fair)
Technical merit: 2 (poor)
Potential impact: 2 (poor)
Originality and novelty: 3 (fair)
Quality of presentation: 2 (poor)

----------- SUMMARY -----------
The paper describes some experimental results of running NAS benchmark applications on  virtualized NUMA machines.

----------- STRENGTHS -----------
Experimental results are presented.

----------- WEAKNESSES -----------
The addressed problem is not novel and no general solution is proposed.

----------- DETAILED COMMENTS TO AUTHOR(S) -----------
The paper describes some experimental results of running NAS benchmark applications on a virtualized NUMA machine. The conclusion is that the scheduler of an OS shall be optimized for being aware of the data locality. 
The title of the paper concerns “virtual machine scheduling” but the content of the paper is actually more about the scheduling of OS than VM scheduling.  The experiments in Section 3, for example, use a single virtual machine of 8 VCPUs. Hence, the problem addresses where (to which core) a thread shall be scheduled and not on which host not a VM shall be scheduled. Only 4.3 is related to VM scheduling. Therefore, the title of the paper is not precise. 
The scheduling problem is a common one for NUMA architectures and not specifically for virtual machines. As for NUMA, a lot of research works have been performed for data locality and as a result, general solutions have been proposed. The paper hence does not handle a novel topic.
 The paper gives some conclusions without reasoning. For example, the first paragraph of Section 3 claims that “This observation indicates that the virtual machine scheduling (actually the thread scheduling) has great impact on the performance”. It may be true but there is no any fact to show this. The experimental result shows a poor performance on the VM; but it can be caused by the overhead of virtualization. Remote access is surely a reason, but this reason also is true for the physical run, if the 4 threads of an application are executed on different nodes. Therefore, it cannot be declared that the scheduling scheme is the cause of performance loss on the VM.
The paper describes many experiment cases in a way that it is very difficult to follow all of them. For a clear description I suggest to use tables for the cases and make the text shorter.
For experiments in Section 4, VCPUs can be manually bund to a specific node. How was this done?
Overall, authors performed several experiments to study the performance of virtual machines. However, the investigated problem is not novel. Furthermore, no general solutions were proposed.


----------------------- REVIEW 2 ---------------------
PAPER: 134
TITLE: The Impact of Virtual Machine Scheduling on NUMA multicore Systems
AUTHORS: Yuxia Cheng, Wenzhi Chen and Shaoyu Zhang

OVERALL EVALUATION: 1 (weak accept)
Relevance to the topic: 3 (fair)
Technical merit: 3 (fair)
Potential impact: 4 (good)
Originality and novelty: 2 (poor)
Quality of presentation: 2 (poor)

----------- SUMMARY -----------
This paper deals with the evaluation of NUMA-specific characteristics of the KVM CFS scheduler. The authors briefly describe the NUMA architecture, the relevant background information on the CFS scheduler and present their performance analysis on two multicore platforms.

----------- STRENGTHS -----------
Virtualization (and the Cloud in general) seem like a great trend to increase elasticity and flexibility in application deployment. To this end, the paper validates previous findings on the overheads imposed by non-NUMA-aware vCPU schedulers.

----------- WEAKNESSES -----------
The authors do not succeed in providing sufficient solutions to the problem they describe. Although they mention current VM schedulers are anaware of the NUMA mutlicore topology, much work has been done towards this direction [1,2,3]. The presentation also needs considerable work, as there are several grammar and syntactic errors, that render the text incomprehensible at times (mainly in sections 3 and 4).

[1] http://blog.xen.org/index.php/2012/04/26/numa-and-xen-part-1-introduction/
[2] http://wiki.xen.org/wiki/Xen_NUMA_Introduction
[3] http://www.linux-kvm.org/wiki/images/5/59/Kvm-forum-2011-performance-improvements-optimizations-D.pdf

----------- DETAILED COMMENTS TO AUTHOR(S) -----------
In this paper the authors analyze the overheads imposed by the CFS scheduling policy (KVM) on NUMA nodes. They state that the KVM scheduler is NUMA-unaware and, thus, the NPB benchmarks suffer from degraded performance due to memory pressure from multiple VMs, and guest kernel scheduling (within a multi-core VM).

Point 1: the authors should present insights on why such performance degradation exists in Section 3 (4-thread case) (vCPU mappings, overcommitment factor, finetuning options used or unused in the CFS/KVM setup)

Point 2: Performance results are packed into 4 pages -- the text is highly repetitive without providing significant information on *why* these results make sense, or on *what* can be done to improve this degradation.

Overall, the authors should provide a clear indication of their findings -- stating that VM performance regarding memory accesses depends on the VM kernel and other VMs on the same container is not sufficient. I would prefer less text in the evalutation section (strip down all the repetitive sentences) and describe what can be done to overcome these limitations. I would not expect any implementation, but at least provide a discussion section, presenting possible solutions for this issue.

What could help, is juxtaposing section 3 figures with section 4 and explaining the difference in (say) one single benchmark.


----------------------- REVIEW 3 ---------------------
PAPER: 134
TITLE: The Impact of Virtual Machine Scheduling on NUMA multicore Systems
AUTHORS: Yuxia Cheng, Wenzhi Chen and Shaoyu Zhang

OVERALL EVALUATION: 1 (weak accept)
Relevance to the topic: 4 (good)
Technical merit: 3 (fair)
Potential impact: 3 (fair)
Originality and novelty: 3 (fair)
Quality of presentation: 3 (fair)

----------- SUMMARY -----------
The paper studies the impact of VM scheduling on NUMA systems, good mapping from VM to processors, and the contention among VMs.

----------- STRENGTHS -----------
* The paper is well written and easy to understand.
* The experiments are well designed to serve the ideas.

----------- WEAKNESSES -----------
* The findings may not be that novel.

----------- DETAILED COMMENTS TO AUTHOR(S) -----------
The paper misses some important related work so it is unclear how new the findings are compared with the state of the arts.  An important finding of the paper is on data locality, caching performance and resource contention.  However, to schedule jobs on a NUMA system, the tradeoff between data locality and contention is quite well-known. One related paper is "Performance analysis of thread mappings with a holistic view of the hardware resources".  The related work of the paper shall be improved to identify the novelty of the work.


----------------------- REVIEW 4 ---------------------
PAPER: 134
TITLE: The Impact of Virtual Machine Scheduling on NUMA multicore Systems
AUTHORS: Yuxia Cheng, Wenzhi Chen and Shaoyu Zhang

OVERALL EVALUATION: -2 (reject)
Relevance to the topic: 4 (good)
Technical merit: 3 (fair)
Potential impact: 2 (poor)
Originality and novelty: 3 (fair)
Quality of presentation: 4 (good)

----------- SUMMARY -----------
This work investigates the impact of virtual machine (VM) scheduling on multicore NUMA systems. Direct 
VM mapping combinations is alaysed in this work and concludes that the best VM scheduling strategy depends 
on both data sharing and memory characteristics as well as the resource contention and dynamic behaviour 
of co-running VMs.

----------- STRENGTHS -----------
The paper is well written and provide sufficient information regarding the background of the topic and existing
 body of knowledge. It also setup a proper performance evaluation and comparison using different benchmarks.

----------- WEAKNESSES -----------
This work tries to investigate and evaluate the performance of exisitng scheduling algorithms, when utilized 
on NUMA-based systems. However, if we assume that NUMA systems are trying to address a known performance issue, 
the more valid study would be to investigate how to improve the performance of the scheduling algorithms, when 
they are utilized on top of NUMA architecture. Even in the existing research, authors have failed to identify 
the features that should be improved in the existing scheduling algorithms, when they are used on NUMA systems.

----------- DETAILED COMMENTS TO AUTHOR(S) -----------
"This work investigates the impact of virtual machine (VM) scheduling on multicore NUMA systems. Direct VM mapping combinations is alaysed in this work and concludes that the best VM scheduling strategy depends on both data sharing and memory characteristics as well as the resource contention and dynamic behaviour of co-running VMs. There are some issues that should be addressed 
1) P2, 2nd paragraph: Since NUMA multicore systems should be considered as given. The reveiewer beleives that instead of investigating the impact of NUMA on VM scheduling, the other way around should be further investigated.
2)P2, 3rd paragraph: CFS is not defined. The acronyms should be extended at least in their first occurance. 
3)P3, 2nd paragraph: IOH/PCH chips are not defined. At least the acronyms should be properly expanded.
4) p4, 2nd paragraph: It seems that the authors are trying to investigate the impact of scheduling on the NUMA architecture. However, reviewer beleives that the reason behind the NUMA architecture is to address the bandwidth bottleneck and access of cores to the memory and therefore it makes sense logically to come up with algorithms which can benefit and perform properly using this architecture. While the authors are trying to investigate the impact of existing algorithms on NUMA architecture.
5) P4, last sentence: ...And 3 benchmarks --> ...And three benchmarks"