性能测试的问题…



性能测试的问题… 

 Neil Cameron

       最近有一次和客户打交道非常愉快，但从中也凸显出性能测试的一些问题所在，尤其是用测试软件做的测试。现今，测试软件大家都经常用到，这个工具也确实非常有效，能够深入探索一组特定的性能指标，监测存储系统在这些方面的特定反应。

       但是…这种情形下，关键问题在于要准确找到这些参数，让测试的参数确实与数据相匹配。

       回到我的这个客户，他想要了解的是maxcache——我们的RAID卡提供的SSD缓存功能，该软件可以利用连接到81605ZQ控制器的SSD硬盘，为某个存储阵列添加读和写缓存。

       用测试软件来测试时，发现改善不明显（至少该客户是这样讲）。我们讨论了测试参数的问题，以及测试时间长短（一分钟显然远远不够），在客户原来的测试中，也确实发现了显著改善（这些客户对自己的系统显然有深刻的了解）。

      所以呢，在多次测试以后，大家决定将系统放到真实客户那里，在虚拟的桌面环境下（运行一整套虚拟桌面的openE）进行测试。猜猜结果怎样？——客户（和最终用户）快乐得犹如….

       结果是，真实的数据非常适合缓存（正如系统构建者所想），但测试软件无法真正反映出现实世界中服务器上数据的特点。最终，所有人（系统构建者、数据中心运营者及最终用户）都对系统的性能非常满意，且啧啧称奇。

       这个小故事的意图何在？很简单，就是想要用测试软件来产生出能够近似现实世界中服务器的数据十分困难。有没有解决办法呢？可能没有，但我建议大家可以估且看看性能测试软件及其导出的结果，但还是要在真实的环境中进行测试，或者，哪怕进行近似的模拟测试。

       得到的结果可能非常惊人。

  

The problem with performancetesting …

 

   Had a really good experience with a customer recently, but ithighlighted the problems with performance testing, especially using iometer.Now, we use iometer a lot, and it’s a great tool to drill down into a specificset of performance characteristics to show a specific response from a storagesystem.

   However … the problem with such a situation is getting the parametersright so that you are testing the right parameters that match your data.

   So this customer was looking at maxcache – our SSD caching functionalitythat uses SSD drives attached to the 81605ZQ controller to add read and writecaching to an array.

   Testing with iometer didn’t show that much of an improvement (at leastaccording to the customer). Discussion regarding the test parameters and howlong to run a test for (1 minute won’t cut the mustard) saw a big improvementover their original testing (and yes, these guys know what they are doing withtheir systems so I’m not having a go at any individual system builder here).

   So after much testing, it was decided to put the machine into test withreal-world customers in a virtual desktop environment (believe is was openErunning a whole stack of virtual desktops). Guess what – customers (end users)were as happy as pigs in …

   Turns out the real world data is perfectly suited to caching (assuspected by the system builder), but that iometer was not able to accuratelyreflect the data characteristics of the real-world server. End result: everyone(system builder, datacenter operator, end users) – all happy and amazed at theperformance of the system.

   So where is the moral in this story? Simply that it’s difficult to playwith a test software and come up with something that will closely match the endresult of a server used in the real world. Is there an answer to this? Probablynot, but I’m suggesting that everyone take performance testing software and theresults they get with a grain of salt, and look at testing in the real world,or at least a close simulation.

   The results can be very surprising.