The Dutch Prutser's Blog

By: Harald van Breederode

  • Disclaimer

    The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.
  • Subscribe

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 238 other followers

Demystifying ASM REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB

Posted by Harald van Breederode on January 3, 2013

A recurring question in my classes is how Oracle Automatic Storage Management (ASM) calculates the REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB disk group values. As usual the answer is: It depends! ;-)

In short: the REQUIRED_MIRROR_FREE_MB value in V$ASM_DISKGROUP indicates how much free space is required in an ASM disk group to restore redundancy after the failure of an ASM disk or ASM failure group. The USABLE_FILE_MB value indicates how much space is available in an ASM disk group considering the redundancy level of the disk group. The question is: How does ASM calculate these values?

The answer can be found by creating ASM disk groups in different configurations and take a look at the REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB values in V$ASM_DISKGROUP. All we need are a bunch of small disks.

ASM> select label,os_mb from v$asm_disk
  2  where label like 'SAN%' order by label;

LABEL                                OS_MB
------------------------------- ----------
SAN01                                  255
SAN02                                  255
SAN03                                  255
SAN04                                  255
SAN05                                  255
SAN06                                  255

REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB in external redundancy disk groups

Let’s get started by creating an external redundancy disk group on top of the above 6 disks:

ASM> create diskgroup demo external redundancy
  2  disk 'ORCL:san01'
  3  disk 'ORCL:san02'
  4  disk 'ORCL:san03'
  5  disk 'ORCL:san04'
  6  disk 'ORCL:san05'
  7  disk 'ORCL:san06'
  8  attribute 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

Now we can explore this disk group by querying several columns from V$ASM_DISKGROUP:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     EXTERN       1530    1468        0   1468

The above output shows that this disk group size is 1530MB (TOTAL_MB ) of which 1468MB (FREE_MB)is free space. Because this is an external redundancy disk group, all free space is available for storing files as indicated by the USABLE_FILE_MB column. The REQUIRED_MIRROR_FREE_MB column is zero because ASM does not mirror user data in external redundancy disk groups.

Note: The “missing” 62MB is used for storing ASM metadata.

REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB in normal redundancy disk groups

Next we create a normal redundancy disk group with six failure groups of one disk each.

ASM> create diskgroup demo normal redundancy
  2  failgroup FG1 disk
  3  'ORCL:san01'
  4  failgroup FG2 disk
  5  'ORCL:san02'
  6  failgroup FG3 disk
  7  'ORCL:san03'
  8  failgroup FG4 disk
  9  'ORCL:san04'
 10  failgroup FG5 disk
 11  'ORCL:san05'
 12  failgroup FG6 disk
 13  'ORCL:san06'
 14  attribute 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

ASM uses failure groups to provide redundancy inside disk groups. Data stored in one failure group is also stored in another failure group to protect it against failures. In a normal redundancy disk group the loss of one failure group can be tolerated without affecting data availability. ASM will restore redundancy by re-creating the lost data in another failure group. But this requires that a certain amount of disk group space is available within the same disk group to store this data. Because of this, it is useful to know how big a single failure group actually is.

ASM> select failgroup,sum(total_mb) from v$asm_disk
  2  where label like 'SAN%' group by failgroup order by failgroup;

FAILGROUP                      SUM(TOTAL_MB)
------------------------------ -------------
FG1                                      255
FG2                                      255
FG3                                      255
FG4                                      255
FG5                                      255
FG6                                      255

The above output tells us that the size of a single failure group is 255MB. Let’s see how this influences the free space requirements and how it affects usable file space:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     NORMAL       1530    1365      255    555

The above output indicates that there should be 255MB free space available in order to allow ASM to restore redundancy after a failure. The value is 255 because it is the size of one failure group. In case failure groups are not equally sized, ASM automatically selects the biggest failure group for this calculation.

The above output also indicates that we can use 555MB for storing files. This value is calculated as follows:

ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable
  2  from v$asm_diskgroup where name='DEMO';

   USEABLE
----------
       555

In plain English: The amount of usable file space is equal to the amount of free disk space subtracted by the amount of disk space that is required to restore redundancy after a failure divided by the redundancy level of the disk group which is two in this case.

Thus based on the amount of free space, the amount of required free space and the redundancy level ASM calculates the amount of space available for storing files. If the amount of free space changes, because a file gets created, resized or deleted, this value will also change. This can be demonstrated by creating a data file in this disk group.

SQL> create tablespace foo datafile '+DEMO' size 200m;

Tablespace created.

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     NORMAL       1530     939      255    342

Because we created a 200MB data file the amount of usable file space is reduced as shown above. The reduction is slightly more than the file size because ASM allocated a bit more space for its metadata. Let’s add another file to our tablespace:

SQL> alter tablespace foo add datafile '+DEMO' size 200m;

Tablespace altered.

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     NORMAL       1530     534      255    139

As expected the amount of usable file space is reduced by slightly more than the actual file size. Can we create another 200MB data file?

SQL> alter tablespace foo add datafile '+DEMO' size 200m;

Tablespace altered.

We could indeed create another 200MB data file despite the fact that ASM indicated that there was only 139MB usable file space available. It is important to note that ASM does not enforce the amount of free space indicated by REQUIRED_MIRROR_FREE_MB. Let’s take a look at the value of USABLE_FILE_MB now:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     NORMAL       1530     129      255    -63

The value has turned negative! This means that if we encounter a failure ASM will be unable to restore redundancy because there is not enough space to do so. It is very important that USABLE_FILE_MB is monitored! If it becomes negative for a particular disk group, you should either increase the size of the disk group or free up some space in it. For now we will just drop the tablespace ;-)

SQL> drop tablespace foo;

Tablespace dropped.

Now let’s create another normal redundancy disk group, but this time with only three failure groups each with two disks instead of six failure groups each with only one disk:

ASM> create diskgroup demo normal redundancy
  2  failgroup FG1 disk
  3  'ORCL:san01',
  4  'ORCL:san02'
  5  failgroup FG2 disk
  6  'ORCL:san03',
  7  'ORCL:san04'
  8  failgroup FG3 disk
  9  'ORCL:san05',
 10  'ORCL:san06'
 11  attribute 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

Now that the disk group is created we again take a look at the sizes of its failure groups:

ASM> select failgroup,sum(total_mb) from v$asm_disk
  2  where label like 'SAN%' group by failgroup order by failgroup;

FAILGROUP                      SUM(TOTAL_MB)
------------------------------ -------------
FG1                                      510
FG2                                      510
FG3                                      510

All three failure groups are equally sized at 510MB. Let’s see how this is reflected in the space administration columns in V$ASM_DISKGROUP:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     NORMAL       1530    1365      510    427

As expected REQUIRED_MIRROR_FREE_MB is now 510 which is of course equal to the size of the biggest failure group. This results in 427MB of usable file space. As shown before this value is calculated as follows:

ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable
  2  from v$asm_diskgroup where name='DEMO';

   USEABLE
----------
       427

Next we create yet another normal redundancy disk group, but this time with only two failure groups each with three disks instead of three failure groups each with two disks:

ASM> create diskgroup demo normal redundancy
  2  failgroup FG1 disk
  3  'ORCL:san01',
  4  'ORCL:san02',
  5  'ORCL:san03'
  6  failgroup FG2 disk
  7  'ORCL:san04',
  8  'ORCL:san05',
  9  'ORCL:san06'
 10  attribute 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

Again we determine the size of all failure groups:

ASM> select failgroup,sum(total_mb) from v$asm_disk
  2  where label like 'SAN%' group by failgroup order by failgroup;

FAILGROUP                      SUM(TOTAL_MB)
------------------------------ -------------
FG1                                      765
FG2                                      765

Both failure groups are equally sized at 765MB. What will the impact be on the space administration?

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     NORMAL       1530    1416      255    580

Hmmm, REQUIRED_MIRROR_FREE_MB is only 255 (the size of one disk) instead of 765 (the size of one failure group) as one might expect. Apparently ASM only anticipates on the failure of a single disk and no longer anticipates on the failure of a complete failure group.

When you think about this it actually makes sense! As stated at the beginning of this posting, ASM performs mirroring by storing the same data in different failure groups. A normal redundancy disk group requires at least two failure groups and a high redundancy disk group requires at least three failure groups. How can ASM restore redundancy in a normal redundancy disk group when there is only one failure group left? It can’t! Therefore ASM anticipates only on the loss of a single disk.

For completeness, this is how ASM calculated the USABLE_FILE_MB value of 580:

ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable
  2  from v$asm_diskgroup where name='DEMO';

   USEABLE
----------
       580

REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB in high redundancy disk groups

Now that we covered external and normal redundancy disk groups, we will take a look at high redundancy disk groups. So, let’s get started by creating a high redundancy disk group with six failure groups of one disk each and determine the size of its failure groups:

ASM> create diskgroup demo high redundancy
  2  failgroup FG1 disk
  3  'ORCL:san01'
  4  failgroup FG2 disk
  5  'ORCL:san02'
  6  failgroup FG3 disk
  7  'ORCL:san03'
  8  failgroup FG4 disk
  9  'ORCL:san04'
 10  failgroup FG5 disk
 11  'ORCL:san05'
 12  failgroup FG6 disk
 13  'ORCL:san06'
 14  attribute 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

ASM> select failgroup,sum(total_mb) from v$asm_disk
  2  where label like 'SAN%' group by failgroup order by failgroup;

FAILGROUP                      SUM(TOTAL_MB)
------------------------------ -------------
FG1                                      255
FG2                                      255
FG3                                      255
FG4                                      255
FG5                                      255
FG6                                      255

As expected all failure groups are equally sized at 255MB. Could there be a difference between the required free and usable disk space for a high redundancy disk group and a normal redundancy disk group with the same number of disks and failure groups? Let’s have a look:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     HIGH         1530    1365      510    285

REQUIRED_MIRROR_FREE_MB is 510, which is equal to the sum of the size of two failure groups (or the sum of the sizes of the two biggest failure groups if they are not equally sized). This is because a high redundancy disk group can tolerate the loss of two failure groups without affecting data availability.

USABLE_FILE_MB is 285 and is calculated as follows:

ASM> select trunc((free_mb - required_mirror_free_mb) / 3) as useable
  2  from v$asm_diskgroup where name='DEMO';

   USEABLE
----------
       285

Finally we create another high redundancy disk group but this time with three failure groups each with two disks instead of six failure groups of one disk each, followed by querying the failure group sizes:

ASM> create diskgroup demo high redundancy
  2  failgroup FG1 disk
  3  'ORCL:san01',
  4  'ORCL:san02'
  5  failgroup FG2 disk
  6  'ORCL:san03',
  7  'ORCL:san04'
  8  failgroup FG3 disk
  9  'ORCL:san05',
 10  'ORCL:san06'
 11  attribute 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

ASM> select failgroup,sum(total_mb) from v$asm_disk
  2  where label like 'SAN%' group by failgroup order by failgroup;

FAILGROUP                      SUM(TOTAL_MB)
------------------------------ -------------
FG1                                      510
FG2                                      510
FG3                                      510

Hopefully the above output isn’t surprising because we have created quite a number of disk groups already on the disks used to write this posting. If you managed to follow and understand everything so far the next output should match your expectations:

ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
  2  from v$asm_diskgroup where name = 'DEMO';

NAME       STATE       TYPE     TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO       MOUNTED     HIGH         1530    1365      510    285

The 510 listed for REQUIRED_MIRROR_FREE_MB is the size of two disks and not the size of a single failure group. The reason for this is almost the same as for a normal redundancy disk group with only two failure groups, but this time we have a high redundancy disk group with only three failure groups. If one or two failure groups are lost, ASM cannot restore redundancy as that would require at least three working failure groups.

Below is the calculation for the 285 listed as USABLE_FILE_MB:

ASM> select trunc((free_mb - required_mirror_free_mb) / 3) as useable
  2  from v$asm_diskgroup where name='DEMO';

   USEABLE
----------
       285

In summary

REQUIRED_MIRROR_FREE_MB is the amount of free space required to restore redundancy after a failure that ASM can tolerate without affecting data availability. This amount depends on the redundancy level and the number of failure groups in the disk group.

Normal redundancy disk groups with at least three failure groups require an amount of free space that is equal to the size of a single failure group. Normal redundancy disk groups with only two failure groups require an amount of free space that is equal to the size of a single disk.

High redundancy disk groups with at least four failure groups require an amount of free space that is equal to the sum of the size of two failure groups. High redundancy disk groups with only three failure groups require an amount of free space that is equal to the sum of the size of two disks.

USABLE_FILE_MB is the amount of disk space available for storing user data. This amount depends on the total size of the disk group, the disk group redundancy and the amount of required free disk space that ASM needs to restore redundancy after a failure. USABLE_FILE_MB is calculated as follows:.

USABLE_FILE_MB = (FREE_MBREQUIRED_MIRROR_FREE_MB) / [2|3]

-Harald

About these ads

12 Responses to “Demystifying ASM REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB

  1. Hi Harald,

    Great article. Very easy to read and to understand.
    two years ago I also wrote a blog about ASM, but then more about a view that shows on which disk ASM has actually stored the data.
    It is not completely related to the above, but anyway here is the link: http://managingoracle.blogspot.nl/2010/10/under-hood-of-asm.html

    Thanks again

    • Harald van Breederode said

      Hi Frank-Jan,

      Thanx for your kind words.
      I’ve read your posting and found it to be interesting. Very useful query.
      -Harald

  2. [...] Harald van Breederode is demystifying ASM REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB. [...]

  3. Hi Harald,

    Thanks for putting it nicely and sharing with us.

    I was exploring the same issue on our Exadata machine last week :)

    Regards,
    Saurabh

  4. MOHAMMED AJAZ said

    Hi Harald,

    I had a same issue in our production environment and resolved through this article. Nice job. Keep it up good work.

    Regards,
    MOHAMMED.

  5. Hi Herald,

    This is surely a very descriptive and nice article.

    But if I look at current configuration we have, the required_mirror_free_mb is something different, would appreciate if you can look into this and clarify my doubts.

    Below is the command outputs as you have given in this article:

    SQL> select failgroup,sum(total_mb/1024/1024) TOTAL_TB from v$asm_disk group by failgroup order by failgroup;

    FAILGROUP TOTAL_TB
    —————————— ———-
    DW01CEL01 6.47998047
    DW01CEL02 6.47998047
    DW01CEL03 6.47998047
    DW01CEL04 6.47998047
    DW01CEL05 6.47998047
    DW01CEL06 6.47998047
    EX01ECEL01 32.6750793
    EX01ECEL02 32.6750793
    EX01ECEL03 32.6750793
    EX01ECEL04 32.6750793

    10 rows selected.

    SQL> select name,total_mb/1024 Total_GB,free_mb/1024 Free_GB,cold_used_mb/1024 Cold_GB,required_mirror_free_mb/1024 Req_GB,usable_file_mb/1024 Usable_GB from v$asm_diskgroup;

    NAME TOTAL_GB FREE_GB COLD_GB REQ_GB USABLE_GB
    —————————— ———- ———- ———- ———- ———-
    DATA 30960 6151.31641 24808.6836 5160 495.658203
    SYSTEMDG 1747.5 1745.89063 1.609375 291.25 727.320313
    DATA_EXP 76800 58168.6328 18631.3672 19200 19484.3164
    RECO_EXP 35905.5 19704.3672 16201.1328 7200 6252.18359
    DATA_DEV 19200 18492.0977 707.902344 4800 6846.04883
    RECO_DEV 7685.25 7576.875 108.375 1921.3125 2827.78125
    DBFS_EXP 1351.875 1351.21094 0.6640625 337.96875 506.621094

    7 rows selected.

    Here I see that the required_free_mb space is too high and consuming almost 38 TB of storage. We have mixed disk types in our configuration (600 GB and 3 TBs). Diskgroups which suffixes with EXP are having 3 TBs disks.

    ASMCMD>lsdg

    Name Total_MB Free_MB Req_mirror_free_mb Usable_file_MB
    DATA/ 31703040 6298956 5283840 507558
    DATA_DEV/ 19660800 18935908 4915200 7010354
    DATA_EXP/ 78643200 59564680 19660800 19951940
    DBFS_EXP/ 1384320 1383640 346080 518780
    RECO_DEV/ 7869696 7758720 1967424 2895648
    RECO_EXP/ 36767232 23764236 7372800 8195718
    SYSTEMDG/ 1789440 1787792 298240 744776

    Thanks in advance.

    Regards
    Saurabh

    • Harald van Breederode said

      Hi Saurabh,

      I am quite busy this and next week, so please allow me some time to answer your question.

      Thanx, -Harald

  6. LSC said

    Hi Harald

    In your example with 2 FG where ASM anticipates one disk failure instead of entire failgroup because there is only one failgroup left when something goes wrong, in my tests I see that each disk reserves space for restoring redundancy when 1 disk fails.

    For example

    FG1 has 2 disks, 1016MB each
    FG2 has 2 disks, 1016MB each

    REQUIRED_MIRROR_FREE_MB is 1016MB which is 254MB per disk, in fact we can see in a FULL disk group where USABLE_FILE_MB is 0 each disk has 254MB in FREE_MB. So suppose one disk in FG2 dies which contains 762MB (1016-254) the reserved 254 in other 3 disks can be used to restore this data but hey arent we using failgroup FG1 to restore redundancy? Arent the data from FG2 stored in FG1 as well?

    • Harald van Breederode said

      Hi LSC,

      No, because ASM mirrors across failgroups and not within failgroups. The whole point abount a failgroup is that ASM can tolerate the loss of one, or two in case of high redundancy, failgroup without affecting data availability. If ASM would mirror inside a failgroup it would defeat its purpose.

      Hope this clarifies it,
      -Harald

  7. LSC said

    Hi Harald

    I got the point after doing several tests and realized that having two FG in normal redundancy doesnt really make much sense reserving one disk space since losing a disk the redundancy cannot be restored unless the surviving disks have plenty of space but guess that is how it is designed

    Thans

  8. […] Harald van Breederode already explains it in detail into this blog post. […]

  9. Zafar Rashid said

    Herald, this is an excellent article. it is tough explaining to non techies the diffarances between usable, free, req, negative usable file MB mean especially when they see 72 Tb of storage disappear on a 3/4 rack. this article helped.
    regards,
    zafar

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 238 other followers

%d bloggers like this: