Cache Coherent 非一致性内存解决与验证

来源：互联网发布：人工智能利弊作文编辑：程序博客网时间：2024/06/04 18:50

1 摘要

使用J6平台开发项目中，使用DMA拷贝数据，针对DMA物理内存和Cache的一致性进行测试和验证。

2 解决方法和目的

方法目的：使用dma_sync_single_for_device();dma_sync_single_for_cpu()对内存进行一致性管理，通过下文的介绍可以正确使用这两个函数。

方法验证：

A. 验证Cache与DMA存在不一致的问题。

B. dma_sync_single_for_device();dma_sync_single_for_cpu()对物理内存和Cache的实际操作内容。

C. 验证函数效率。

2.1 INV和CLEAN概念

（invalidate）清除cache的意思是清除cache中存储的全部数据。对处理器而言，清除操作只要清零相应cache行的有效位即可。当存储器配置上有变化时，整体或部分cache可能需要进行清除操作。
（clean）清理cache的意思是把脏的（即被改写过的）cache行强制写到主存，并把cache行中的脏位清零。清理cache可以重建cache与主存之间的一致性，它只用在使用写回策略的D-cache上。

2.2 测试使用程序列表

名称

功能

Dmatest

测试应用程序

omap_ds_dma.ko

测试对应驱动程序

2.3 测试程序命令列表

命令名称

功能

@cc <n>

Set cache on(1)/off(0)驱动内DMA的cache管理功能

@ccsrc [size]

Make src cached

@ccdst [size]

Make dst cached

@ccsrcn [size]

Make src cached write 0x66

@ccdstn [size]

Make src cached write 0x33

@up

Update src dynamic val

@dmacp

Dma Cp test 0x1000 DMAsrc->dst拷贝4K

@dmacp1

Dma Cp test max DMAsrc->dst拷贝8M

@dmacpn <n>

Dma Cp test n times

@showm <addr>

Show Mem 0x---- test 显示物理地址内数据

@shows

Show Mem 0xAC000000

@showd

Show Mem 0xAC800000

@coset <val>

Set coherent data 设置Coherent驱动内缓冲区的值

@coshow

Show coherent data显示Coherent驱动内缓冲区的值

@setwb <paddr>

Set write back 设置指定区域的clean功能WB

@setinv <paddr>

Set invalide 设置指定区域的invalid功能

@cotocc

Coherent to cache Coherent to dst(cached) DMA拷贝

@cctoco

Cache to Coherent dst(cached) to Coherent 的DMA拷贝

@exit

@help

2.4 验证条件

a. 驱动程序提供一块一致性内存;

b. 使用物理内存0xAC000000~0xAC800000为src带cache内存。

c. 使用物理内存0xAC800000~0xAD000000为dst带cache内存

2.5 验证1一致性问题存在

使用DMA将no cache数据传送到cached内存，需要将目的内存进行cache，然后启动DMA，再查看目的内存的内容，结果会导致CPU与物理内存不一致

<UT>:coset 0x65

[61901.806422]

[D]Paddr:0xAB859000 Paddr:0xEB859000

[D]coset 0x65

=============BUFF MSG============

[D] 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65

[D]

[D]=============BUFF END============

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E0C000

=============BUFF MSG============

22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22

=============BUFF END============

<UT>:cotocc

cotocc

func_get_coherent_arg ioctl paddr 0xab859000, size 0x00001000,

Dma Cp [0xAB859000] -> [0xAC800000] oK,Use time: 1009 us

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E0C000

=============BUFF MSG============

65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65 65

22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22

=============BUFF END============

<UT>:

已验证，会导致数据不一致。

2.6 验证2 invalidate功能清除cache

验证cache invalidate清除功能，使用函数

dma_sync_single_for_cpu(NULL,paddr, size,DMA_FROM_DEVICE);

参数如设置成DMA_TO_DEVICE，该函数什么都不做。

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E0C000

=============BUFF MSG============

88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88

=============BUFF END============

<UT>:coset 0x43

[62750.244259]

[D]Paddr:0xAB859000 Paddr:0xEB859000

[D]coset 0x43

=============BUFF MSG============

[D] 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43

[D]

[D]=============BUFF END============

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E0C000

=============BUFF MSG============

88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88

=============BUFF END============

<UT>:cotocc

cotocc

func_get_coherent_arg ioctl paddr 0xab859000, size 0x00001000,

Dma Cp [0xAB859000] -> [0xAC800000] oK,Use time: 1009 us

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E0C000

=============BUFF MSG============

88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88

=============BUFF END============

<UT>:setinv 0xac800000

[62763.252850]

[62763.252850] [D]Paddr:0xAC800000 Size:0x00001000

setinv 0xac800000

INVALIDE addr is 0xAC800000

func_get_coherent_arg ioctl paddr 0xab859000, size 0x00001000,

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E0C000

=============BUFF MSG============

43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43

=============BUFF END============

<UT>:

已验证cache invalidate将cache中的dirty位清除。当再次读取时，会重内存中读取数据。

2.7 验证3 clean清理功能将cache写入内存

验证Clean功能，使用函数

dma_sync_single_for_device(NULL,paddr, size, DMA_TO_DEVICE);

<UT>:showd

showd

pMemDstVirtAddr is 0xB5EA8000

=============BUFF MSG============

00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F

20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F

72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91

92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1

=============BUFF END============

<UT>:ccdstn

ccdstn

<UT>:showd

showd

pMemDstVirtAddr is 0xB5EA8000

=============BUFF MSG============

32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51

52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71

72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91

92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1

=============BUFF END============

<UT>:showd

showd

pMemDstVirtAddr is 0xB5EA8000

=============BUFF MSG============

32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51

52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71

72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91

92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1

=============BUFF END============

<UT>:cctoco

cctoco

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

Dma Cp [0xAC800000] -> [0xAB85A000] oK,Use time: 1008 us

<UT>:coshow

[71527.657295]

[D]Paddr:0xAB85A000 Paddr:0xEB85A000

[D]coshow

=============BUFF MSG============

[D] 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F

[D] 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F

[D] 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F

[D] 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F

[D]

[D]=============BUFF END============

<UT>:setwb 0xac800000

[71532.627374]

[71532.627374] [D]Paddr:0xAC800000 Size:0x00001000

setwb 0xac800000

WRITE_CB addr is 0xAC800000

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

<UT>:cctoco

cctoco

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

Dma Cp [0xAC800000] -> [0xAB85A000] oK,Use time: 1008 us

<UT>:coshow

[71541.280669]

[D]Paddr:0xAB85A000 Paddr:0xEB85A000

[D]coshow

=============BUFF MSG============

[D] 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51

[D] 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71

[D] 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91

[D] 92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1

[D]

[D]=============BUFF END============

2.8 验证4 clean清理功能cache改写才能写入内存

<UT>:showd

showd

pMemDstVirtAddr is 0xB5EA8000

=============BUFF MSG============

63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63

=============BUFF END============

<UT>:coset 0

[74406.875787]

[D]Paddr:0xAB85A000 Paddr:0xEB85A000

[D]coset 0

=============BUFF MSG============

[D] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[D]

[D]=============BUFF END============

<UT>:showd

showd

pMemDstVirtAddr is 0xB5EA8000

=============BUFF MSG============

63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63

=============BUFF END============

<UT>:cotocc

cotocc

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

Dma Cp [0xAB85A000] -> [0xAC800000] oK,Use time: 1008 us

<UT>:showd

showd

pMemDstVirtAddr is 0xB5EA8000

=============BUFF MSG============

63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63

=============BUFF END============

<UT>:setwb 0xac800000

[74412.795170]

[74412.795170] [D]Paddr:0xAC800000 Size:0x00001000

setwb 0xac800000

WRITE_CB addr is 0xAC800000

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

<UT>:showd

showd

pMemDstVirtAddr is 0xB5EA8000

=============BUFF MSG============

63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63

=============BUFF END============

<UT>:cctoco

cctoco

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

Dma Cp [0xAC800000] -> [0xAB85A000] oK,Use time: 1010 us

<UT>:coshow

[74422.187135]

[D]Paddr:0xAB85A000 Paddr:0xEB85A000

[D]coshow

=============BUFF MSG============

[D] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[D]

[D]=============BUFF END============

。。。。。。。。。。。

<UT>:showd

showd

pMemDstVirtAddr is 0xB5EA8000

=============BUFF MSG============

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

=============BUFF END============

验证到cache的clean功能，在cache未由cpu改写的情况下，并不能把物理内存变化的cache部分进行强制写入。

验证改写部分

<UT>:

<UT>:coset 0

[75962.455990]

[D]Paddr:0xAB85A000 Paddr:0xEB85A000

[D]coset 0

=============BUFF MSG============

[D] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[D]

[D]=============BUFF END============

<UT>:dmacp

dmacp

Dma Cp [0xAC000000] -> [0xAC800000] oK,Use time: 1010 us

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E4F000

=============BUFF MSG============

FF 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E

1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E

3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E

5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E

=============BUFF END============

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E4F000

=============BUFF MSG============

FF 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E

1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E

3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E

5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E

=============BUFF END============

<UT>:cotocc

cotocc

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

Dma Cp [0xAB85A000] -> [0xAC800000] oK,Use time: 1007 us

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E4F000

=============BUFF MSG============

FF 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E

1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E

3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E

5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E

=============BUFF END============

<UT>:ccdstn

ccdstn

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E4F000

=============BUFF MSG============

32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51

52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71

72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91

92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1

=============BUFF END============

<UT>:setwb 0xac800000

[75993.171957]

[75993.171957] [D]Paddr:0xAC800000 Size:0x00001000

setwb 0xac800000

WRITE_CB addr is 0xAC800000

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

<UT>:cctoco

cctoco

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

Dma Cp [0xAC800000] -> [0xAB85A000] oK,Use time: 1009 us

<UT>:coshow

[76001.389677]

[D]Paddr:0xAB85A000 Paddr:0xEB85A000

[D]coshow

=============BUFF MSG============

[D] 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51

[D] 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71

[D] 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91

[D] 92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1

[D]

[D]=============BUFF END============

验证结果

当cache被改写，后执行setwb 0xac800000会强制写入内存。

2.9 验证5 clean清理功能将cache改写后数据相同也有效

[D]coshow

=============BUFF MSG============

[D] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[D]

[D]=============BUFF END============

<UT>:showd

showd

pMemDstVirtAddr is 0xB5E4F000

=============BUFF MSG============

32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51

52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71

72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91

92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1

=============BUFF END============

<UT>:ccdstn

ccdstn

<UT>:setwb 0xac800000

[76396.223846]

[76396.223846] [D]Paddr:0xAC800000 Size:0x00001000

setwb 0xac800000

WRITE_CB addr is 0xAC800000

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

<UT>:cctoco

cctoco

func_get_coherent_arg ioctl paddr 0xab85a000, size 0x00001000,

Dma Cp [0xAC800000] -> [0xAB85A000] oK,Use time: 1009 us

<UT>:coshow

[76401.636119]

[D]Paddr:0xAB85A000 Paddr:0xEB85A000

[D]coshow

=============BUFF MSG============

[D] 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51

[D] 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71

[D] 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91

[D] 92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1

[D]

[D]=============BUFF END============

进一步验证得到cache对应的内容只要被再次写入，尽管是相同的数据，Clean也会强制写入内存。

2.10验证效率与flush_cache_all()对比

A. 使用上面函数dma_sync_single_for_cpu和dma_sync_single_for_device测试效率

入口位置Start前使用1次src清理和1次dst清理，DMA 完成使用1次dst清除。

B. flush_cache_all() 两次清理和一次清除所花费的时间

C. 入口位置Start前使用1次all，DMA 完成使用1次all。

3 其他

无。

0 0