March 19

VMWARE: Recreating a missing VMFS datastore partition in VMware vSphere 5.0/5.1/5.5

Recreating a missing VMFS datastore partition in VMware vSphere 5.0/5.1/5.5 (2046610)

Symptoms

A datastore has become inaccessible.
A VMFS partition table is missing.

Purpose

The partition table is required only during a rescan. This means that the datastore may become inaccessible on a host during a rescan if the VMFS partition was deleted after the last rescan. The partition table is physically located on the LUN, so all vSphere hosts that have access to this LUN can see the change has taken place. However, only the hosts that do a rescan will be affected.

This article provides information on:

Determining whether this is the same problem.
Resolving the problem.

Cause

This issue occurs because the VMFS partition can be deleted by deleting the datastore from the vSphere Client. This is prevented by the software, if the datastore is in use. It can also happen if a physical server has access to the LUN on the SAN and does an install, for example.

Resolution

To resolve this issue, you must compare partedUtil output to determine whether the partition is missing.

The first output shows the partition as intact:

# partedUtil getptbl /vmfs/devices/disks/naa.6006016045502500c20a2b3ccecfe011
gpt
52216 255 63 838860800
1 2048 838850039 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

The second output shows the partition as missing:

# partedUtil getptbl /vmfs/devices/disks/naa.6006016045502500c20a2b3ccecfe011
gpt
52216 255 63 838860800

Note: This means the partition is not listed.

To recreate the partition:

Find the beginning and end blocks where the VMFS partition is supposed to be. To find the beginning of the partition, copy and paste this command (one line script):

# offset=”128 2048″; for dev in `esxcfg-scsidevs -l | grep “Console Device:” | awk {‘print $3’}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`; do echo “Checking offset found at $i:”; hexdump -n4 -s $((0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo “———————“; done

Note: The above script checks all of the storage devices, so the list may be lengthy. This script is not applicable for local disks.

An example of the output::

/vmfs/devices/disks/naa.60060160455025009839a9ed4cfee011
msdos
78325 255 63 1258291200
1 128 1258291124 251 0
Checking offset found at 128:
0110000 d00d c001
0110004
1310000 f15e 2fab
1310004
0131001d 46 43 5f 53 68 61 72 65 64 00 45 76 65 72 5f 47 |old_VMFS3…….|
0131002d 65 74 74 69 6e 67 5f 55 70 00 00 00 00 00 00 00 |…………….|
———————
/vmfs/devices/disks/naa.6006016045502500c20a2b3ccecfe011
gpt
52216 255 63 838860800
Checking offset found at 2048:
0200000 d00d c001
0200004
1400000 f15e 2fab
1400004
0140001d 4a 55 50 48 41 4d 5f 53 52 4d 35 00 00 00 00 00 |new_VMFS5…….|
0140002d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |…………….|
———————

The above output has two example storage devices. The first example was created on an ESX version prior to version 5, so it reports:

Checking offset found at 128.

The second example was created on vSphere 5+, so it reports:

Checking offset found at 2048.

Note: In this example, you are using the second device, so the beginning of the partition is 2048.
To get the end block for the partition, run this command:
# partedUtil getUsableSectors /vmfs/devices/disks/naa.6006016045502500c20a2b3ccecfe011
34 838860766

Notes:
- If you do not get the expected output of two numbers and you get an Unknown partition table on disk error run this command to label the table as a GPT partition table:
  # partedUtil mklabel /vmfs/devices/disks/naa.6006016045502500c20a2b3ccecfe011 gpt
  
  Rerun the partedUtil getUsableSectors command, if you do not get the expected output of 2 numbers, run the partition type identification commands in the next bullet also.
- If you do not get the expected output of two numbers and receive an error message stating partition table invalid,unable to satisfy all constraints on the partition or a similar error, run this command:
  
  # partedUtil setptbl /vmfs/devices/disks/naa.6006016045502500c20a2b3ccecfe011 gpt “1 2048 4123456 AA31E02A400F11DB9590000C2911D1B8 0″T
  
  This creates a temporary partition. Once applied, the read disk information can be read, and the correct result from the above query is returned, which then allows you to calculate the correct last usable block
  
  The partition type identifies the purpose of a partition, and may be represented by either a decimal identifier (for example, 251) or a GUID (for example, AA31E02A400F11DB9590000C2911D1B8). See the list of commonly used partitions in the Additional Information section in Using the partedUtil command line utility on ESXi and ESX (1036609). Partitions created on ESXi 5.x and higher with the gpt disklabel must be specified using the GUID.
  
  The number in red indicates the last usable block, so the end of the partition cannot be any higher. It is unknown whether this was the number used when the datastore was created, so you can try it and adjust if necessary.
  
  With the relevant values replaced for your environment, run this command to set the correct value for last usable block:
  
  # partedUtil setptbl /vmfs/devices/disks/naa.6006016045502500c20a2b3ccecfe011 gpt “1 2048 838860766 AA31E02A400F11DB9590000C2911D1B8 0″
Run this command to attempt to mount the VMFS datastore:
# vmkfstools -V

Note: If it mounts, the numbers are correct. If the numbers are correct, values do not require adjustment, so you can skip step 4.
If it does not mount, you may see a message in /var/log/vmkernel.log that is similar to:
… cpu0:44828)LVM: 2891: [naa.6006016045502500c20a2b3ccecfe011:1] Device expanded (actual size 838858719 blocks, stored size 838847992 blocks)

In this case, add 2047 to the stored size to get the true end, … 838847992 + 2047 = 838850039

Run the command with the new end value:

# partedUtil setptbl /vmfs/devices/disks/naa.6006016045502500c20a2b3ccecfe011 gpt “1 2048 838850039 AA31E02A400F11DB9590000C2911D1B8 0″

Now you have the correct partition. Run the VMFS rescan again:

# vmkfstools -V

After the datastore is successfully mounted on one host, you can expect that the same VMFS rescan command will mount the VMFS datastore when run on other hosts that have access to this LUN.

Alternatively, you can run a full cluster rescan from the vCenter Server using the vSphere Client.

Posted March 19, 2015 by Timothy Conrad in category "Virtualization

About the Author

If I were to describe myself with one word it would be, creative. I am interested in almost everything which keeps me rather busy. Here you will find some of my technical musings. Securely email me using - PGP: 4CB8 91EB 0C0A A530 3BE9 6D76 B076 96F1 6135 0A1B View all posts by Timothy Conrad