Linux: Fixing a VDO Volume due to running out of space
If you fill the physical space of a VDO volume it will corrupt the data and you will see I/O errors dmesg will probably look something like this: [ 3150.967303] XFS (dm-2): writeback error on sector 10950872 [ 3153.384860] XFS (dm-2): metadata I/O error in "xlog_iodone" at daddr 0x1200068 len 64 error 28 [ 3153.384873] XFS (dm-2): xfs_do_force_shutdown(0x2) called from line 1261 of file fs/xfs/xfs_log.c. Return address = 000000000d11b9e8 [ 3153.384926] XFS (dm-2): Log I/O Error Detected. Shutting down filesystem [ 3153.384928] XFS (dm-2): Please umount the filesystem and rectify the problem(s) The only way to fix this as of 8/21/2018 is to expand the raid. This should be kept in mind when building VDO volumes. If you "land-lock" yourself out of the ability to add drive space then you be relying on any backups you might have to recover this data. Process for recovery: In this "mini" example we have 4 drives in a raid 5 2 TB - drives 8 TB - Raw 6 TB - Usable after the raid VDO is setup for 18 TB --name=vdo_data We artificially create files us in dd if=/dev/urandom and if=/dev/zero. After the space is consumed we begin to get I/O errors on our mounted share. Process for fixing: Check for new vdo size: vdostats --human-readable [root@localhost mnt]# vdostats --human-readable Device Size Used Available Use% Space saving% /dev/mapper/vdo_data 6.0T 6.0T 0.0B 100% 0% Unmount your broken clock level mount: umount /mnt/vdo_data Add another 2 TB drive to the raid5 Use fdisk -l to make certain you know the device id In this example our new and 5 drive is: /dev/sdf Also, take note of the raid size for /dev/md0 Add drive: mdadm --add /dev/md0 /dev/sdf Watch process: watch cat /proc/mdstat Grow raid group: mdadm --grow /dev/md0 --raid-devices=5 Watch process: watch cat /proc/mdstat Check raid size: fdisk -l You should see /dev/md0 is larger than before: Grow the Logical volume size of the vdo volume vdo growLogical --name=vdo_data --vdoLogicalSize=24G Grow the physical size of vdo vdo growPhysical --name=vdo_data Check for new vdo size: vdostats --human-readable You should now see available freespace Repair your file system: mount -t xfs /dev/mapper/vdo_data /mnt/vdo_data ummount /mnt/vdo_data xfs_repair /dev/mapper/vdo_data mount -t xfs /dev/mapper/vdo_data /mnt/vdo_data The final file that filled the drive will be missing data.