内存测试Stream程序分析

Stream测试是内存测试中业界公认的内存带宽性能测试基准工具。作为一个服务器工程师在衡量和评比服务器的性能时，该如何从简单的一个源码使之编译成适合自己的可用的测试工具呢？让我们一起来学习这个基本过程吧。
首先我们看一下Linux下最简单的编译过程：

点击(此处)折叠或打开

1. gcc -O stream.c -o stream.o

上述的编译使用了程序和编译器的默认参数，生成的stream.o即可执行，执行结果如下：

点击(此处)折叠或打开

1. # ./stream.o
2. -------------------------------------------------------------
3. STREAM version $Revision: 5.10 $
4. -------------------------------------------------------------
5. This system uses 8 bytes per array element.
6. -------------------------------------------------------------
7. Array size = 45000000 (elements), Offset = 0 (elements)
8. Memory per array = 343.3 MiB (= 0.3 GiB).
9. Total memory required = 1030.0 MiB (= 1.0 GiB).
10. Each kernel will be executed 10 times.
11. The *best* time for each kernel (excluding the first iteration)
12. will be used to compute the reported bandwidth.
13. -------------------------------------------------------------
14. Your clock granularity/precision appears to be 1 microseconds.
15. Each test below will take on the order of 122074 microseconds.
16. (= 122074 clock ticks)
17. Increase the size of the arrays if this shows that
18. you are not getting at least 20 clock ticks per test.
19. -------------------------------------------------------------
20. WARNING -- The above is only a rough guideline.
21. For best results, please be sure you know the
22. precision of your system timer.
23. -------------------------------------------------------------
24. Function Best Rate MB/s Avg time Min time Max time
25. Copy: 5376.3 0.134082 0.133922 0.134222
26. Scale: 5139.4 0.140465 0.140093 0.140754
27. Add: 5680.4 0.190304 0.190128 0.190547
28. Triad: 5476.8 0.197417 0.197195 0.197598
29. -------------------------------------------------------------
30. Solution Validates: avg error less than 1.000000e-13 on all three arrays
31. -------------------------------------------------------------

以上测试结果是在Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz  6Cores CPU及1333MHz*8的内存。以上测试中主要的影响结果的默认值为单线程，Array size = 45000000。以下是源文件全部可以定义的参数。

点击(此处)折叠或打开

1. #ifdef _OPENMP     定义开启多处理器运行环境，
   extern int omp_get_num_threads();
   #endif
   
   
2. 
   
3. #ifndef STREAM_ARRAY_SIZE   定义计算的数组大小（成员个数）
4. # define STREAM_ARRAY_SIZE 45000000
5. #endif
6. 
   
7. #ifdef NTIMES  定义值非法时，若定义计算次数，10
8. #if NTIMES<=1
9. # define NTIMES 10
10. #endif
11. #endif
12. 
    
13. #ifndef NTIMES   若定义计算次数,10
14. # define NTIMES 10
15. #endif
16. 
    
17. #ifndef OFFSET  定义数组偏移量
18. # define OFFSET 0
19. #endif
20. 
    
21. #ifndef STREAM_TYPE  定义数组为双精度，64bit,8Bytes
    #define STREAM_TYPE double
    #endif
    
    

以上参数是源码中的预定义参数，也可以在编译或运行时动态指定。 为方便理解，下面展示以下我编译的多核多内存机器的编译方法

1. gcc -mtune=native -march=native -O3 -mcmodel=medium -fopenmp -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=30 -DOFFSET=4096 stream.c -o stream.o

解释： 
 -mtune=native -march=native； 针对CPU指令的优化，此处由于编译机即运行机器。故采用native的优化方法。更多编译器对CPU的优化参考：http://gcc.gnu.org/onlinedocs/gcc-4.5.3/gcc/i386-and-x86_002d64-Options.html
 -O3 ； 编译器编译优化级别；
-mcmodel=medium ；当单个Memory Array Size 大于2GB时需要设置此参数。
-fopenmp； 适应多处理器环境；开启后，程序默认线程为CPU线程数，也可以运行时也可以动态指定运行的进程数 ：export OMP_NUM_THREADS=12   #12为自定义的要使用的处理器
-DSTREAM_ARRAY_SIZE=100000000；指定计算中a[],b[],c[]数组的大小，
-DNTIMES=30 ；执行的次数，并且从这些结果中选最优值。
-DOFFSET=4096 ；数组的偏移，一般可以不定义。

其中STREAM_ARRAY_SIZE对测试结果影响较大，源码中也为数组大小的选取进行了经验说明。

点击(此处)折叠或打开

1. You should adjust the value of 'STREAM_ARRAY_SIZE' (below)
2. * to meet *both* of the following criteria:
3. * (a) Each array must be at least 4 times the size of the
4. * available cache memory. I don't worry about the difference
5. * between 10^6 and 2^20, so in practice the minimum array size
6. * is about 3.8 times the cache size.
7. * Example 1: One Xeon E3 with 8 MB L3 cache
8. * STREAM_ARRAY_SIZE should be >= 4 million, giving
9. * an array size of 30.5 MB and a total memory requirement
10. * of 91.5 MB.   \\STREAM_ARRAY_SIZE的值最好能4倍于CPU Cache, 注意Array Size占的存储空间的换算，double 64bit 8Byte的转换。
11. * Example 2: Two Xeon E5's with 20 MB L3 cache each (using OpenMP)
12. * STREAM_ARRAY_SIZE should be >= 20 million, giving
13. * an array size of 153 MB and a total memory requirement
14. * of 458 MB.
15. * (b) The size should be large enough so that the 'timing calibration'
16. * output by the program is at least 20 clock-ticks.
17. * Example: most versions of Windows have a 10 millisecond timer
18. * granularity. 20 "ticks" at 10 ms/tic is 200 milliseconds.
19. * If the chip is capable of 10 GB/s, it moves 2 GB in 200 msec.
20. * This means the each array must be at least 1 GB, or 128M elements.  \ 这个可以参考运行时的提示，考虑修改

-DOFFSET=4096 ；数组的偏移，为了对齐数组内存空间和内存中物理空间，因为CPU在读取内存时不是1bit一bit的读，一般可以不定义，定义后也不一定有效果。主要是由于影响内存分配和物理存储单元的对齐。以下是解释和代码中仅有的OFFSET参数出现的地方

点击(此处)折叠或打开

1. Users are allowed to modify the "OFFSET" variable, which *may* change the
2. * relative alignment of the arrays (though compilers may change the
3. * effective offset by making the arrays non-contiguous on some systems).
4. * Use of non-zero values for OFFSET can be especially helpful if the
5. * STREAM_ARRAY_SIZE is set to a value close to a large power of 2.
6. * OFFSET can also be set on the compile line without changing the source
7. * code using, for example, "-DOFFSET=56".

点击(此处)折叠或打开

1. #ifndef STREAM_TYPE
2. #define STREAM_TYPE double
3. #endif
4. 
   
5. static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET],
6. b[STREAM_ARRAY_SIZE+OFFSET],
7. c[STREAM_ARRAY_SIZE+OFFSET];   //可以看出设置该值后，数组的长度变长了