November 18

Linux: Fixing a VDO Volume due to running out of space

If you fill the physical space of a VDO volume it will corrupt the data and you will see I/O errors
dmesg will probably look something like this:

[ 3150.967303] XFS (dm-2): writeback error on sector 10950872
[ 3153.384860] XFS (dm-2): metadata I/O error in "xlog_iodone" at daddr 0x1200068 len 64 error 28
[ 3153.384873] XFS (dm-2): xfs_do_force_shutdown(0x2) called from line 1261 of file fs/xfs/xfs_log.c.  Return address = 000000000d11b9e8
[ 3153.384926] XFS (dm-2): Log I/O Error Detected.  Shutting down filesystem
[ 3153.384928] XFS (dm-2): Please umount the filesystem and rectify the problem(s)

The only way to fix this as of 8/21/2018 is to expand the raid.  This should be kept in mind when building VDO volumes.  If you "land-lock" yourself out of the ability to add drive space then you be relying on any backups you might have to recover this data.

Process for recovery:
In this "mini" example we have 4 drives in a raid 5
2 TB - drives
8 TB - Raw
6 TB - Usable after the raid

VDO is setup for 18 TB --name=vdo_data

We artificially create files us in dd if=/dev/urandom and if=/dev/zero.
After the space is consumed we begin to get I/O errors on our mounted share.

Process for fixing:
Check for new vdo size:
vdostats --human-readable
[root@localhost mnt]# vdostats --human-readable
Device                    Size      Used Available Use% Space saving%
/dev/mapper/vdo_data      6.0T      6.0T      0.0B 100%            0%

Unmount your broken clock level mount:
umount /mnt/vdo_data

Add another 2 TB drive to the raid5

Use fdisk -l to make certain you know the device id
In this example our new and 5 drive is: /dev/sdf
Also, take note of the raid size for /dev/md0

Add drive:
mdadm --add /dev/md0 /dev/sdf

Watch process:
watch cat /proc/mdstat 

Grow raid group:
mdadm --grow /dev/md0 --raid-devices=5

Watch process:
watch cat /proc/mdstat

Check raid size:
fdisk -l
You should see /dev/md0 is larger than before:

Grow the Logical volume size of the vdo volume
vdo growLogical --name=vdo_data --vdoLogicalSize=24G

Grow the physical size of vdo
vdo growPhysical --name=vdo_data

Check for new vdo size:
vdostats --human-readable
You should now see available freespace

Repair your file system:
mount -t xfs /dev/mapper/vdo_data /mnt/vdo_data
ummount /mnt/vdo_data
xfs_repair /dev/mapper/vdo_data 
mount -t xfs /dev/mapper/vdo_data /mnt/vdo_data

The final file that filled the drive will be missing data.

Copyright 2021. All rights reserved.

Posted November 18, 2021 by Timothy Conrad in category "Linux

About the Author

If I were to describe myself with one word it would be, creative. I am interested in almost everything which keeps me rather busy. Here you will find some of my technical musings. Securely email me using - PGP: 4CB8 91EB 0C0A A530 3BE9 6D76 B076 96F1 6135 0A1B