May 26

VMWARE: Datastore fails to unmount due to a locked file

When trying to unmount a datastore in VMWARE an error occurs due to a file still existing on the datastore. The file will not delete in the GUI or from the cli.

1. Discovering a file is lock
# rm VDI-Desktop-028-vdm-disposable-4f70ddb4-6699-4649-85e0-d1fce33f6e0a.vmdk
rm: can’t remove ‘VDI-Desktop-028-vdm-disposable-4f70ddb4-6699-4649-85e0-d1fce33f6e0a.vmdk’: Device or resource busy

2. Look to see what has the file locked
# lsof | grep -i VDI-Desktop-028-vdm-disposable-4f70ddb4-6699-4649-85e0-d1fce33f6e0a.vmdk
/vmfs/volumes/51e5bda0-d77e7dbd-c71c-00215e5d489c/VDI-Desktop-028

3. In my case, there is nothing showing up as having the file locked

4. Now determine the state of the file with vmkfstools. For the purposes of this write-up we will cd into the needed directory.

# vmkfstools -D VDI-Desktop-028-vdm-disposable-4f70ddb4-6699-4649-85e0-d1fce33f6e0a.vmdk

Lock [type 10c00001 offset 92305408 v 246409, hb offset 4186112
gen 857, mode 2, owner 00000000-00000000-0000-000000000000 mtime 148164482
num 1 gblnum 0 gblgen 0 gblbrk 0]
RO Owner[0] HB Offset 3842048 587d159e-9ab8bf0c-687b-0025b5001b29
Addr <4, 196, 79>, gen 246048, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 576, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 8192

5.Refer to the above UUID of the system that has it locked: 0025b5001b29

6. You can look for the UUID of each system by logging onto the with the CLI or use VCENTER

CLI
# esxcfg-info -y | grep “System UUID”
|—-System UUID………………………………………….5880b297-eadd-89ba-c3c4-0025b5001c0a

Web GUI
Go to: Host and Clusters – Choose an ESX Host – Networking – Physical Adapters – MAC Address

7. Once you have determined which ESX host has locked the file you will need to reboot the server.
8. Assuming that your server is in a cluster with HA and DRS running, put the ESX server in maintenance mode
9. Once the virtual machines have vacated your server right click on the server and reboot.
10. At this point you can log onto another ESX host in the same cluster and delete your previously locked file.

# rm VDI-Desktop-028-vdm-disposable-4f70ddb4-6699-4649-85e0-d1fce33f6e0a.vmdk -f
/vmfs/volumes/51e5bda0-d77e7dbd-c71c-00215e5d489c/VDI-Desktop-028

11. After your server comes back up exit maintenance mode
12. You can now unmount your data store.

If you were examining th file from a different ESX server, be certain to cd out of the volume on the datastore in order to be able to delete the datastore from Vcenter

By: Timothy Conrad

Category: Virtualization | Comments Off on VMWARE: Datastore fails to unmount due to a locked file
September 1

V7000: Updating Storwize V7000 drive code

**************************************************************************
Update 17/12/2011:

Until the flash is updated showing how to avoid this issue, only update drive firmware when installing a new machine or if all hosts are offline.

**************************************************************************

IBM recently released new drive firmware for the Storwize V7000, so I thought I would share the process of how I update that firmware.    I recommend you perform the drive update before you next update your Storwize V7000 microcode.

I want to be clear that one of the central goals of the Storwize V7000 is to ensure that performing drive firmware updates can be done online without host disruption.    This is possible because each drive can be updated in less than around 4 seconds.   The scripts I share below leave a 10 second delay between drives just to be safe.  I would still prefer that you did the update during a quiet period.

We need to perform this procedure using the command line as there is no way to do this procedure from the GUI (yet).

There are four steps:

  1. Upload the Software Upgrade Test Utility to determine which drives need updating.
  2. Upload the drive microcode package.
  3. Apply the drive software.
  4. Confirm all drives are updated.

Step 1:  Upload and run the upgrade utility

  • You will need the upgrade test utility.
  • You will need the Putty utility PSCP.
  • You will need to have created a public/private key pair and assigned it to a user.  In all the examples the user name I use is anthonyv.  You need to use your own user-id, although you could also use admin.   Place the PPK file into the putty folder along with the upgrade test utility.

From the Putty folder we need to upload the test utility.  You will need to change the key file name, userid and IP address (all highlighted in red) to suit your installation.

NOTE:  The following command is being run in a Windows command prompt.  You need to be in the C:Program FilesPutty or C:Program Files (x86)Putty folder.

pscp -i anthonyv.ppk IBM2076_INSTALL_upgradetest_6.15 anthonyv@9.191.0.78:/home/admin/upgrade

Having uploaded the file, now start PuTTY and SSH to your Storwize V7000.   Logon and issue the following two commands.  You are using SSH commands now, not the Windows Command Prompt:

svcservicetask applysoftware -file IBM2076_INSTALL_upgradetest_6.15
svcupgradetest -f -d

If you get a warning window like the one shown below, indicating we have down-level drives, we need to proceed to the next step (note that the enclosure and slot numbers are not the same as drive IDs).  If you have a lot of drives, you can drop the -d from the svcupgradetest command to get a summary list.

******************* Warning found *******************
+----------------------+-----------+------------+------------------------------------------+
| Model                | Latest FW | Current FW | Drive Info                               |
+----------------------+-----------+------------+------------------------------------------+
| HK230041S            | 2920      | 291E       | Drive in slot 24 in enclosure  1         |
|                      |           |            | Drive in slot 23 in enclosure  1         |
| ST9450404SS          | B548      | B546       | Drive in slot 22 in enclosure  1         |
|                      |           |            | Drive in slot 21 in enclosure  1         |
|                      |           |            | Drive in slot 20 in enclosure  1         |
|                      |           |            | Drive in slot 19 in enclosure  1         |
|                      |           |            | Drive in slot 18 in enclosure  1         |
|                      |           |            | Drive in slot 17 in enclosure  1         |
|                      |           |            | Drive in slot 16 in enclosure  1         |
|                      |           |            | Drive in slot 15 in enclosure  1         |
|                      |           |            | Drive in slot 14 in enclosure  1         |
|                      |           |            | Drive in slot 13 in enclosure  1         |
|                      |           |            | Drive in slot 12 in enclosure  1         |
|                      |           |            | Drive in slot 11 in enclosure  1         |
|                      |           |            | Drive in slot 10 in enclosure  1         |
|                      |           |            | Drive in slot  9 in enclosure  1         |
|                      |           |            | Drive in slot  8 in enclosure  1         |
|                      |           |            | Drive in slot  5 in enclosure  1         |
|                      |           |            | Drive in slot  6 in enclosure  1         |
+----------------------+-----------+------------+------------------------------------------+

Step 2:  Upload the drive microcode package

Download the drive update package. Put it into the PuTTY folder.
From a Windows command prompt we need to upload the package using the following command.   You will need to change the key file name, userid and IP address (all highlighted in red) to suit your installation.    Note yet again that you are running this in a Windows command prompt from the PuTTY folder (not from inside an SSH session):

pscp -i anthonyv.ppk IBM2076_DRIVE_20110928 anthonyv@9.191.0.78:/home/admin/upgrade

Step 3:  Apply the drive software

I have written some scripts to help you list the drive IDs that need to be updated and perform the updates.   You can upgrade the drives one at a time, or in bulk, depending on how you want to do this.  All the remaining commands are all run in a PuTTY session.

Firstly run this script to list all the drive IDs and current firmware levels.  We need the drive IDs if we want to update individual drives.

svcinfo lsdrive -nohdr |while read did error use;do svcinfo lsdrive $did |while read id value;do if [[ $id == "firmware_level" ]];then echo $did"   "$value;fi;done;done

The output will look something like this, showing the drive ID and that drive’s current firmware level.   From step 1 we know what the latest firmware level is, so we can compare to the current firmware level:

0   291E
1   291E
2   B546
3   B546
4   B546
5   B546
6   B546
7   B546
8   B546
9   B546
10   B546
11   B546
12   B546
13   B546
14   B546
15   B546
16   B546
17   B546
18   B546
19   B546
20   B546
21   B546
22   B546
23   B546

Now we can update individual drives with this command, which will update drive ID 23.   Just keep changing the drive IDs, using the list of down-level drives, until every drive has been updated:

svctask applydrivesoftware -file IBM2076_DRIVE_20110928 -type firmware -drive 23

However you may have a lot of drives and want to upgrade them in bulk. So you could use this command, which updates drive ID 19 and 20 (highlighted in red).  You could change and also add extra drives to the list as required:

for did in 19 20;do echo "Updating drive "$did;svctask applydrivesoftware -file IBM2076_DRIVE_20110928 -type firmware -drive $did;sleep 10s;done

If we just wanted to upgrade every single drive in the machine (regardless of their level), we could run this command:

svcinfo lsdrive -nohdr |while read did name IO_group_id;do echo "Updating drive "$did;svctask applydrivesoftware -file IBM2076_DRIVE_20110928 -type firmware -drive $did;sleep 10s;done

When updating multiple drives, I have inserted a 10 second sleep between updates, just to ensure the process runs smoothly.  This means each drive takes about 13-15 seconds.

Once we have upgraded every drive, it is time for a final check.

Step 4: Confirm all drives are updated

You have two ways to confirm this.   Firstly run the following command to list the firmware level of each drive.  Is each drive reflecting the levels reported in Step 1?

svcinfo lsdrive -nohdr |while read did error use;do svcinfo lsdrive $did |while read id value;do if [[ $id == "firmware_level" ]];then echo $did"   "$value;fi;done;done

Now run the software upgrade test utility again:

svcupgradetest -f -d

Provided you receive no warnings about drives not being at the recommended levels, you are now finished with the drive updates.   Of course you could now proceed to install 6.2.0.4 firmware, but you can do that from the GUI.

By: anthonyv

Category: SAN, Virtualization | Comments Off on V7000: Updating Storwize V7000 drive code
August 17

Virtualization: Virtualbox command line options and control

Controlling the VirtualBox VM

Now that we have VirtualBox installed and a VM guest created we need to control and modify the VM

How to List VM information

How to show the VirtualBox VM info

vmadmin$ VBoxManage showvminfo <vmname>

How to show the VM Harddrive info

vmadmin$ VBoxManage showhdinfo <filename>

How to list running VM

vmadmin$ VBoxManage list runningvms

How to list available VM

vmadmin$ VBoxManage list vms

How to list available VM Harddrives

vmadmin$ VBoxManage list hdds

How to list available ISO’s

vmadmin$ VBoxManage list dvds

How to Control VM

How to Start VM

nohup and & to place process in background, so VM continues to run after closing console.

vmadmin$ nohup VBoxHeadless –startvm <vmname> &

How to pause VM

vmadmin$ VBoxManage controlvm <vmname> pause

How to resume VM

vmadmin$ VBoxManage controlvm <vmname> resume

How to reset VM

vmadmin$ VBoxManage controlvm <vmname> reset

How to poweroff VM (hard poweroff eg. pull the plug)

vmadmin$ VBoxManage controlvm <vmname> poweroff

How to poweroff VM (Save State)

vmadmin$ VBoxManage controlvm <vmname> savestate

How to send poweroff single to VM (tells VM OS to shutdown)

vmadmin$ VBoxManage controlvm <vmname> acpipowerbutton

* Be certain to set your power options inside of your guest OS to respond to the power button.

How to attach a DVD / CD to a running vm

vmadmin$ VBoxManage controlvm <vmname> dvdattach <filename>

How to de-attach a DVD / CD from a running vm

vmadmin$ VBoxManage controlvm <vmname> dvdattach none

By: Fabio Milano, additions by Tim Conrad

Category: Virtualization | Comments Off on Virtualization: Virtualbox command line options and control
August 13

DS3200: Storage Manager commands

When in the IBM DS Storage Manager Enterprise management client you can run scripts to check the status and execute commands on the DS series storage systems.

1. Login
2. Go to Tools – Execute scripts
3. Type in your command in the upper script editor window
4. Click Tools – Execute Only

Command Examples:

Command to check status on the DS3200 – show logicalDrive [“drivenumber”] actionProgress;
eg. show logicalDrive [“5”] actionProgress;
Command to force a drive replacement on the DS3200 – replace Drive [chassis number, slot] replacementDrive=chassis number,slot;
eg. replace Drive [82,2] replacementDrive=85,2;

Category: Virtualization | Comments Off on DS3200: Storage Manager commands