Recently I had spun up a new vm that I was using as “tempstorage”. Basically I set the size to 16tb which was the overall servers max space. I had 3 other vms up and running before this. Those vms all had 500GB and one 5TB. Neither of those were even close to that limit. Same with the “tempstorage” server. I got bout 1.8TB of data added before I woke up and the server was half running. The last drive was orange which means there is an issue with the drive. However shortly after rebooting the server my HPE server corrected the drive and it turned green. So I guess it was a result of running out of space.
Now I cannot get the lvm2 to start. The only message I get is “Manual repair required!” message. I have tried the:
lvconvert --repair pve/data
After running the above I once again get the “Manual repair required!” message and then when I check dmesg the only entry I see for thin_repair is:
[ 638.183185] thin_repair[944]: segfault at 8 ip 00007fea094fb5be sp 00007ffedb8e5d18 error 4 in libstdc++.so.6.0.30[7fea094d5000+116000]
I still yield the same results with the message “Manual repair required!”.
fdisk -l
Disk /dev/sda: 1.36 TiB, 1499832039424 bytes, 2929359452 sectors
Disk model: LOGICAL VOLUME
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 1310720 bytes
Disklabel type: gpt
Disk identifier: A3A2F44C-5D01-4566-9F1D-A0474AA28060
Device Start End Sectors Size Type
/dev/sda1 2560 5119 2560 1.3M BIOS boot
/dev/sda2 5120 1054719 1049600 512.5M EFI System
/dev/sda3 1054720 2929357454 2928302735 1.4T Linux filesystem
Disk /dev/sdb: 16.37 TiB, 18003355459584 bytes, 35162803632 sectors
Disk model: LOGICAL VOLUME
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 1572864 bytes
Disklabel type: gpt
Disk identifier: 6B0CC303-8D66-4D65-90DE-196D9B7C90AA
Device Start End Sectors Size Type
/dev/sdb1 34 2047 2014 1007K BIOS boot
/dev/sdb2 2048 1050623 1048576 512M EFI System
/dev/sdb3 1050624 35162803598 35161752975 16.4T Linux LVM
Disk /dev/loop0: 673.75 MiB, 706473984 bytes, 1379832 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
pvs -a
PV VG Fmt Attr PSize PFree
/dev/sda2 --- 0 0
/dev/sda3 --- 0 0
/dev/sdb2 --- 0 0
/dev/sdb3 pve lvm2 a-- 16.37t <16.38g
vgs -a
VG #PV #LV #SN Attr VSize VFree
pve 1 9 0 wz--n- 16.37t <16.38g
lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi---tz-- <16.23t
[data_tdata] pve Twi------- <16.23t
[data_tmeta] pve ewi------- 15.81g
[lvol0_pmspare] pve ewi------- 15.81g
root pve -wi-a----- 96.00g
swap pve -wi-a----- 8.00g
vm-100-disk-0 pve Vwi---tz-- 500.00g data
vm-101-disk-0 pve Vwi---tz-- 5.00t data
vm-102-disk-0 pve Vwi---tz-- 500.00g data
vm-103-disk-0 pve Vwi---tz-- 5.00t data
vm-104-disk-0 pve Vwi---tz-- 100.00g data
vm-105-disk-0 pve Vwi---tz-- 16.00t data
pvscan
PV /dev/sdb3 VG pve lvm2 [16.37 TiB / <16.38 GiB free]
Total: 1 [16.37 TiB] / in use: 1 [16.37 TiB] / in no VG: 0 [0 ]
lvscan
inactive '/dev/pve/data' [<16.23 TiB] inherit
ACTIVE '/dev/pve/swap' [8.00 GiB] inherit
ACTIVE '/dev/pve/root' [96.00 GiB] inherit
inactive '/dev/pve/vm-100-disk-0' [500.00 GiB] inherit
inactive '/dev/pve/vm-101-disk-0' [5.00 TiB] inherit
inactive '/dev/pve/vm-102-disk-0' [500.00 GiB] inherit
inactive '/dev/pve/vm-103-disk-0' [5.00 TiB] inherit
inactive '/dev/pve/vm-104-disk-0' [100.00 GiB] inherit
inactive '/dev/pve/vm-105-disk-0' [16.00 TiB] inherit
vgdisplay
--- Volume group ---
VG Name pve
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 31
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 9
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 16.37 TiB
PE Size 4.00 MiB
Total PE 4292205
Alloc PE / Size 4288013 / <16.36 TiB
Free PE / Size 4192 / <16.38 GiB
VG UUID FwXD4j-Y1q4-DyV0-i34b-qGSw-zwAO-2aHh2f