Demystifying ASM REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB

Demystifying ASM REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB

Posted by Harald van Breederode on January 3, 2013

A recurring question in my classes is how Oracle Automatic Storage Management (ASM) calculates theREQUIRED_MIRROR_FREE_MB andUSABLE_FILE_MB disk group values. As usual the answer is: It depends! ;-)

In short: the REQUIRED_MIRROR_FREE_MB value in V$ASM_DISKGROUP indicates how much free space is required in an ASM disk group to restore redundancy after the failure of an ASM disk or ASM failure group. TheUSABLE_FILE_MB value indicates how much space is available in an ASM disk group considering the redundancy level of the disk group. The question is: How does ASM calculate these values?

The answer can be found by creating ASM disk groups in different configurations and take a look at theREQUIRED_MIRROR_FREE_MB andUSABLE_FILE_MB values in V$ASM_DISKGROUP. All we need are a bunch of small disks.

ASM> select label,os_mb from v$asm_disk

  2  where label like 'SAN%' order by label;

 

LABEL                                OS_MB

------------------------------- ----------

SAN01                                  255

SAN02                                  255

SAN03                                  255

SAN04                                  255

SAN05                                  255

SAN06                                  255

REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB in external redundancy disk groups

Let’s get started by creating an external redundancy disk group on top of the above 6 disks:

ASM> create diskgroup demo external redundancy

  2  disk 'ORCL:san01'

  3  disk 'ORCL:san02'

  4  disk 'ORCL:san03'

  5  disk 'ORCL:san04'

  6  disk 'ORCL:san05'

  7  disk 'ORCL:san06'

  8  attribute 'compatible.asm' = '11.2.0.0.0';

 

Diskgroup created.

Now we can explore this disk group by querying several columns from V$ASM_DISKGROUP:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     EXTERN       1530    1468        0   1468

The above output shows that this disk group size is 1530MB (TOTAL_MB ) of which 1468MB (FREE_MB)is free space. Because this is an external redundancy disk group, all free space is available for storing files as indicated by theUSABLE_FILE_MB column. The REQUIRED_MIRROR_FREE_MB column is zero because ASM does not mirror user data in external redundancy disk groups.

Note: The “missing” 62MB is used for storing ASM metadata.

REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB in normal redundancy disk groups

Next we create a normal redundancy disk group with six failure groups of one disk each.

ASM> create diskgroup demo normal redundancy

  2  failgroup FG1 disk

  3  'ORCL:san01'

  4  failgroup FG2 disk

  5  'ORCL:san02'

  6  failgroup FG3 disk

  7  'ORCL:san03'

  8  failgroup FG4 disk

  9  'ORCL:san04'

 10  failgroup FG5 disk

 11  'ORCL:san05'

 12  failgroup FG6 disk

 13  'ORCL:san06'

 14  attribute 'compatible.asm' = '11.2.0.0.0';

 

Diskgroup created.

ASM uses failure groups to provide redundancy inside disk groups. Data stored in one failure group is also stored in another failure group to protect it against failures. In a normal redundancy disk group the loss of one failure group can be tolerated without affecting data availability. ASM will restore redundancy by re-creating the lost data in another failure group. But this requires that a certain amount of disk group space is available within the same disk group to store this data. Because of this, it is useful to know how big a single failure group actually is.

ASM> select failgroup,sum(total_mb) from v$asm_disk

  2  where label like 'SAN%' group by failgroup order by failgroup;

 

FAILGROUP                      SUM(TOTAL_MB)

------------------------------ -------------

FG1                                      255

FG2                                      255

FG3                                      255

FG4                                      255

FG5                                      255

FG6                                      255

The above output tells us that the size of a single failure group is 255MB. Let’s see how this influences the free space requirements and how it affects usable file space:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     NORMAL       1530    1365      255    555

The above output indicates that there should be 255MB free space available in order to allow ASM to restore redundancy after a failure. The value is 255 because it is the size of one failure group. In case failure groups are not equally sized, ASM automatically selects the biggest failure group for this calculation.

The above output also indicates that we can use 555MB for storing files. This value is calculated as follows:

ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable

  2  from v$asm_diskgroup where name='DEMO';

 

   USEABLE

----------

       555

In plain English: The amount of usable file space is equal to the amount of free disk space subtracted by the amount of disk space that is required to restore redundancy after a failure divided by the redundancy level of the disk group which is two in this case.

Thus based on the amount of free space, the amount of required free space and the redundancy level ASM calculates the amount of space available for storing files. If the amount of free space changes, because a file gets created, resized or deleted, this value will also change. This can be demonstrated by creating a data file in this disk group.

SQL> create tablespace foo datafile '+DEMO' size 200m;

 

Tablespace created.

 

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     NORMAL       1530     939      255    342

Because we created a 200MB data file the amount of usable file space is reduced as shown above. The reduction is slightly more than the file size because ASM allocated a bit more space for its metadata. Let’s add another file to our tablespace:

SQL> alter tablespace foo add datafile '+DEMO' size 200m;

 

Tablespace altered.

 

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     NORMAL       1530     534      255    139

As expected the amount of usable file space is reduced by slightly more than the actual file size. Can we create another 200MB data file?

SQL> alter tablespace foo add datafile '+DEMO' size 200m;

 

Tablespace altered.

We could indeed create another 200MB data file despite the fact that ASM indicated that there was only 139MB usable file space available. It is important to note that ASM does not enforce the amount of free space indicated byREQUIRED_MIRROR_FREE_MB. Let’s take a look at the value of USABLE_FILE_MB now:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     NORMAL       1530     129      255    -63

The value has turned negative! This means that if we encounter a failure ASM will be unable to restore redundancy because there is not enough space to do so. It is very important thatUSABLE_FILE_MB is monitored! If it becomes negative for a particular disk group, you should either increase the size of the disk group or free up some space in it. For now we will just drop the tablespace ;-)

SQL> drop tablespace foo;

 

Tablespace dropped.

Now let’s create another normal redundancy disk group, but this time with only three failure groups each with two disks instead of six failure groups each with only one disk:

ASM> create diskgroup demo normal redundancy

  2  failgroup FG1 disk

  3  'ORCL:san01',

  4  'ORCL:san02'

  5  failgroup FG2 disk

  6  'ORCL:san03',

  7  'ORCL:san04'

  8  failgroup FG3 disk

  9  'ORCL:san05',

 10  'ORCL:san06'

 11  attribute 'compatible.asm' = '11.2.0.0.0';

 

Diskgroup created.

Now that the disk group is created we again take a look at the sizes of its failure groups:

ASM> select failgroup,sum(total_mb) from v$asm_disk

  2  where label like 'SAN%' group by failgroup order by failgroup;

 

FAILGROUP                      SUM(TOTAL_MB)

------------------------------ -------------

FG1                                      510

FG2                                      510

FG3                                      510

All three failure groups are equally sized at 510MB. Let’s see how this is reflected in the space administration columns inV$ASM_DISKGROUP:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     NORMAL       1530    1365      510    427

As expected REQUIRED_MIRROR_FREE_MB is now 510 which is of course equal to the size of the biggest failure group. This results in 427MB of usable file space. As shown before this value is calculated as follows:

ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable

  2  from v$asm_diskgroup where name='DEMO';

 

   USEABLE

----------

       427

Next we create yet another normal redundancy disk group, but this time with only two failure groups each with three disks instead of three failure groups each with two disks:

ASM> create diskgroup demo normal redundancy

  2  failgroup FG1 disk

  3  'ORCL:san01',

  4  'ORCL:san02',

  5  'ORCL:san03'

  6  failgroup FG2 disk

  7  'ORCL:san04',

  8  'ORCL:san05',

  9  'ORCL:san06'

 10  attribute 'compatible.asm' = '11.2.0.0.0';

 

Diskgroup created.

Again we determine the size of all failure groups:

ASM> select failgroup,sum(total_mb) from v$asm_disk

  2  where label like 'SAN%' group by failgroup order by failgroup;

 

FAILGROUP                      SUM(TOTAL_MB)

------------------------------ -------------

FG1                                      765

FG2                                      765

Both failure groups are equally sized at 765MB. What will the impact be on the space administration?

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     NORMAL       1530    1416      255    580

Hmmm, REQUIRED_MIRROR_FREE_MB is only 255 (the size of one disk) instead of 765 (the size of one failure group) as one might expect. Apparently ASM only anticipates on the failure of a single disk and no longer anticipates on the failure of a complete failure group.

When you think about this it actually makes sense! As stated at the beginning of this posting, ASM performs mirroring by storing the same data in different failure groups. A normal redundancy disk group requires at least two failure groups and a high redundancy disk group requires at least three failure groups. How can ASM restore redundancy in a normal redundancy disk group when there is only one failure group left? It can’t! Therefore ASM anticipates only on the loss of a single disk.

For completeness, this is how ASM calculated the USABLE_FILE_MB value of 580:

ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable

  2  from v$asm_diskgroup where name='DEMO';

 

   USEABLE

----------

       580

REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB in high redundancy disk groups

Now that we covered external and normal redundancy disk groups, we will take a look at high redundancy disk groups. So, let’s get started by creating a high redundancy disk group with six failure groups of one disk each and determine the size of its failure groups:

ASM> create diskgroup demo high redundancy

  2  failgroup FG1 disk

  3  'ORCL:san01'

  4  failgroup FG2 disk

  5  'ORCL:san02'

  6  failgroup FG3 disk

  7  'ORCL:san03'

  8  failgroup FG4 disk

  9  'ORCL:san04'

 10  failgroup FG5 disk

 11  'ORCL:san05'

 12  failgroup FG6 disk

 13  'ORCL:san06'

 14  attribute 'compatible.asm' = '11.2.0.0.0';

 

Diskgroup created.

 

ASM> select failgroup,sum(total_mb) from v$asm_disk

  2  where label like 'SAN%' group by failgroup order by failgroup;

 

FAILGROUP                      SUM(TOTAL_MB)

------------------------------ -------------

FG1                                      255

FG2                                      255

FG3                                      255

FG4                                      255

FG5                                      255

FG6                                      255

As expected all failure groups are equally sized at 255MB. Could there be a difference between the required free and usable disk space for a high redundancy disk group and a normal redundancy disk group with the same number of disks and failure groups? Let’s have a look:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     HIGH         1530    1365      510    285

REQUIRED_MIRROR_FREE_MB is 510, which is equal to the sum of the size of two failure groups (or the sum of the sizes of the two biggest failure groups if they are not equally sized). This is because a high redundancy disk group can tolerate the loss of two failure groups without affecting data availability.

USABLE_FILE_MB is 285 and is calculated as follows:

ASM> select trunc((free_mb - required_mirror_free_mb) / 3) as useable

  2  from v$asm_diskgroup where name='DEMO';

 

   USEABLE

----------

       285

Finally we create another high redundancy disk group but this time with three failure groups each with two disks instead of six failure groups of one disk each, followed by querying the failure group sizes:

ASM> create diskgroup demo high redundancy

  2  failgroup FG1 disk

  3  'ORCL:san01',

  4  'ORCL:san02'

  5  failgroup FG2 disk

  6  'ORCL:san03',

  7  'ORCL:san04'

  8  failgroup FG3 disk

  9  'ORCL:san05',

 10  'ORCL:san06'

 11  attribute 'compatible.asm' = '11.2.0.0.0';

 

Diskgroup created.

 

ASM> select failgroup,sum(total_mb) from v$asm_disk

  2  where label like 'SAN%' group by failgroup order by failgroup;

 

FAILGROUP                      SUM(TOTAL_MB)

------------------------------ -------------

FG1                                      510

FG2                                      510

FG3                                      510

Hopefully the above output isn’t surprising because we have created quite a number of disk groups already on the disks used to write this posting. If you managed to follow and understand everything so far the next output should match your expectations:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb

  2  from v$asm_diskgroup where name = 'DEMO';

 

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB

---------- ----------- ------ ---------- ------- -------- ------

DEMO       MOUNTED     HIGH         1530    1365      510    285

The 510 listed for REQUIRED_MIRROR_FREE_MB is the size of two disks and not the size of a single failure group. The reason for this is almost the same as for a normal redundancy disk group with only two failure groups, but this time we have a high redundancy disk group with only three failure groups. If one or two failure groups are lost, ASM cannot restore redundancy as that would require at least three working failure groups.

Below is the calculation for the 285 listed as USABLE_FILE_MB:

ASM> select trunc((free_mb - required_mirror_free_mb) / 3) as useable

  2  from v$asm_diskgroup where name='DEMO';

 

   USEABLE

----------

       285

In summary

REQUIRED_MIRROR_FREE_MB is the amount of free space required to restore redundancy after a failure that ASM can tolerate without affecting data availability. This amount depends on the redundancy level and the number of failure groups in the disk group.

Normal redundancy disk groups with at least three failure groups require an amount of free space that is equal to the size of a single failure group. Normal redundancy disk groups with only two failure groups require an amount of free space that is equal to the size of a single disk.

High redundancy disk groups with at least four failure groups require an amount of free space that is equal to the sum of the size of two failure groups. High redundancy disk groups with only three failure groups require an amount of free space that is equal to the sum of the size of two disks.

USABLE_FILE_MB is the amount of disk space available for storing user data. This amount depends on the total size of the disk group, the disk group redundancy and the amount of required free disk space that ASM needs to restore redundancy after a failure. USABLE_FILE_MB is calculated as follows:.

USABLE_FILE_MB = (FREE_MB – REQUIRED_MIRROR_FREE_MB) / [2|3]

-Harald