ProxMox Server LVM2 Thin LVs Cannot Be Activated - Manual repair required!

Linux
, ,
1

Hi All,

I need some help.

Our ProxMox server’s LVM LV cannot be activated. lvchange -an vg/lv doesn’t work.

# vgchange -a y pve
Check of pool pve/data failed (status:1). Manual repair required!
2 logical volume(s) in volume group "pve" now active

# lvchange -a y pve/data
Check of pool pve/data failed (status:1). Manual repair required!

#lvconvert --repair pve/data
bad checksum in superblock
Repair of thin metadata volume of thin pool pve/data failed (status:1). Manual repair required!

=============== Details ===============

#pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 446.88g 15.83g

# pvscan
PV /dev/sda3 VG pve lvm2 [446.88 GiB / 15.83 GiB free]
Total: 1 [446.88 GiB] / in use: 1 [446.88 GiB] / in no VG: 0 [0 ]

# pvscan
PV /dev/sda3 VG pve lvm2 [446.88 GiB / 15.83 GiB free]
Total: 1 [446.88 GiB] / in use: 1 [446.88 GiB] / in no VG: 0 [0 ]

# pvdisplay
--- Physical volume ---
PV Name /dev/sda3
VG Name pve
PV Size 446.88 GiB / not usable 4.82 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 114400
Free PE 4052
Allocated PE 110348
PV UUID llHl00-31Am-wEPo-NLO9-eve3-3ehb-xj2yvw

# lvscan
ACTIVE '/dev/pve/swap' [8.00 GiB] inherit
ACTIVE '/dev/pve/root' [96.00 GiB] inherit
inactive '/dev/pve/data' [326.88 GiB] inherit
inactive '/dev/pve/vm-100-disk-1' [10.00 GiB] inherit
inactive '/dev/pve/vm-101-disk-1' [10.00 GiB] inherit
inactive '/dev/pve/vm-102-disk-1' [32.00 GiB] inherit
inactive '/dev/pve/vm-103-disk-1' [30.00 GiB] inherit
inactive '/dev/pve/vm-105-disk-1' [32.00 GiB] inherit
inactive '/dev/pve/vm-106-disk-1' [32.00 GiB] inherit
inactive '/dev/pve/vm-107-disk-1' [32.00 GiB] inherit
inactive '/dev/pve/vm-104-disk-1' [32.00 GiB] inherit
inactive '/dev/pve/vm-108-disk-1' [32.00 GiB] inherit
inactive '/dev/pve/vm-109-disk-1' [32.00 GiB] inherit
inactive '/dev/pve/vm-110-disk-1' [160.00 GiB] inherit
inactive '/dev/pve/vm-111-disk-1' [60.00 GiB] inherit
inactive '/dev/pve/vm-112-disk-1' [60.00 GiB] inherit
inactive '/dev/pve/vm-113-disk-1' [60.00 GiB] inherit

# lvdisplay
--- Logical volume ---
LV Path /dev/pve/swap
LV Name swap
VG Name pve
LV UUID mwfufw-UNeX-fIgL-lekx-I6nX-cNxr-msUy1J
LV Write Access read/write
LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
LV Status available
# open 2
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 251:1

--- Logical volume ---
LV Path /dev/pve/root
LV Name root
VG Name pve
LV UUID V0cSbF-oWO0-juSr-wxcO-HqPw-Ubu3-pgILFc
LV Write Access read/write
LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
LV Status available
# open 1
LV Size 96.00 GiB
Current LE 24576
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 251:0

--- Logical volume ---
LV Name data
VG Name pve
LV UUID UL3eR6-ylB0-Rr5I-mwjE-DyQQ-VKC3-IaOq1p
LV Write Access read/write
LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
LV Pool metadata data_tmeta
LV Pool data data_tdata
LV Status NOT available
LV Size 326.88 GiB
Current LE 83682
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-100-disk-1
LV Name vm-100-disk-1
VG Name pve
LV UUID 3p938q-sXPY-DOgf-a8d1-xraO-137b-UP0uQp
LV Write Access read/write
LV Creation host, time main, 2017-03-21 16:15:38 -0400
LV Pool name data
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-101-disk-1
LV Name vm-101-disk-1
VG Name pve
LV UUID zFEnq3-fg0A-RVZ1-Ayuu-IgtY-xmDB-ninqCP
LV Write Access read/write
LV Creation host, time main, 2017-03-22 12:11:09 -0400
LV Pool name data
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-102-disk-1
LV Name vm-102-disk-1
VG Name pve
LV UUID RvPdhS-lPrd-WoYc-Bend-coaC-mIXt-a7ezoI
LV Write Access read/write
LV Creation host, time main, 2017-03-23 15:48:47 -0400
LV Pool name data
LV Status NOT available
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-103-disk-1
LV Name vm-103-disk-1
VG Name pve
LV UUID LAvhrN-pLGb-7281-AEdS-M9EC-49nQ-b92bl1
LV Write Access read/write
LV Creation host, time main, 2017-03-27 16:33:42 -0400
LV Pool name data
LV Status NOT available
LV Size 30.00 GiB
Current LE 7680
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-105-disk-1
LV Name vm-105-disk-1
VG Name pve
LV UUID RdXIsP-Dc72-sRp0-5PES-Ow3t-zJEZ-YpyI2W
LV Write Access read/write
LV Creation host, time main, 2017-04-12 13:55:19 -0400
LV Pool name data
LV Status NOT available
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-106-disk-1
LV Name vm-106-disk-1
VG Name pve
LV UUID hbUnO0-tJTm-LECO-W0fo-n1kU-ewbw-gCJctq
LV Write Access read/write
LV Creation host, time main, 2017-05-15 14:01:49 -0400
LV Pool name data
LV Status NOT available
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-107-disk-1
LV Name vm-107-disk-1
VG Name pve
LV UUID P3mMXw-GaLd-knsl-bVEv-wuqm-Ibmh-3GL3vz
LV Write Access read/write
LV Creation host, time main, 2017-05-31 09:49:16 -0400
LV Pool name data
LV Status NOT available
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-104-disk-1
LV Name vm-104-disk-1
VG Name pve
LV UUID VmvhlG-uuan-N0Hp-RLSs-7JqT-jnMW-9pGqA7
LV Write Access read/write
LV Creation host, time main, 2017-05-31 19:27:13 -0400
LV Pool name data
LV Status NOT available
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-108-disk-1
LV Name vm-108-disk-1
VG Name pve
LV UUID HUKhL8-xpm2-kR6V-cK1c-V0sR-afyD-DkoczT
LV Write Access read/write
LV Creation host, time main, 2017-07-13 11:44:30 -0400
LV Pool name data
LV Status NOT available
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-109-disk-1
LV Name vm-109-disk-1
VG Name pve
LV UUID 4QYmDz-DjwS-RYue-OIjL-Ckki-QBN2-CLWIYK
LV Write Access read/write
LV Creation host, time main, 2017-07-13 18:30:49 -0400
LV Pool name data
LV Status NOT available
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-110-disk-1
LV Name vm-110-disk-1
VG Name pve
LV UUID Pj2Yn3-SCvV-LKLk-jOpS-pPHk-jGvz-0kYZcE
LV Write Access read/write
LV Creation host, time main, 2017-10-16 14:21:51 -0400
LV Pool name data
LV Status NOT available
LV Size 160.00 GiB
Current LE 40960
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-111-disk-1
LV Name vm-111-disk-1
VG Name pve
LV UUID 4NJeJu-hhkc-JMpZ-Fo41-WSyG-4G0b-mumoWe
LV Write Access read/write
LV Creation host, time main, 2017-10-19 11:07:17 -0400
LV Pool name data
LV Status NOT available
LV Size 60.00 GiB
Current LE 15360
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-112-disk-1
LV Name vm-112-disk-1
VG Name pve
LV UUID ZYG1Hx-UFoO-9prZ-J6jm-2lnT-AhLZ-g3rcbC
LV Write Access read/write
LV Creation host, time main, 2017-10-20 10:28:05 -0400
LV Pool name data
LV Status NOT available
LV Size 60.00 GiB
Current LE 15360
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Path /dev/pve/vm-113-disk-1
LV Name vm-113-disk-1
VG Name pve
LV UUID 3ncpyl-b2oM-82TJ-O0m1-ziPX-L2IB-en4qTv
LV Write Access read/write
LV Creation host, time main, 2017-10-20 10:49:41 -0400
LV Pool name data
LV Status NOT available
LV Size 60.00 GiB
Current LE 15360
Segments 1
Allocation inherit
Read ahead sectors auto


# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 894.3G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 256M 0 part
└─sda3 8:3 0 446.9G 0 part
├─pve-root 251:0 0 96G 0 lvm /
└─pve-swap 251:1 0 8G 0 lvm [SWAP]

# df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 6.3G 17M 6.3G 1% /run
/dev/dm-0 95G 9.3G 81G 11% /
tmpfs 16G 25M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/fuse 30M 20K 30M 1% /etc/pve
2

Bad super block! more of file system issue try repair the file system on these LVs

3

How?

Running this from system rescue boot

[liveuser@localhost —]$ sudo e2fsck -y /dev/sda3
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
e2fsck: Superblock invalid, trying backup blocks…
e2fsck: Bad magic number in super-block while trying to open /dev/sda3

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193

4

You are making a couple of mistakes. You needs to deactivate each of the volume under the pool first, and run the repair on the thin pool. You are doing the opposite. I have a KB written for this:

https://mellowhost.com/billing/index.php?rp=/knowledgebase/65/How-to-Repair-a-lvm-thin-pool.html

This error is common if your metadata volume was overflown, people usually forget that allocating more storage than available, would actually overflown metadata volume overtime. Considering the importance of small metadata volume, I always recommend to back it up twice a day using thin_dump. You can also change the size of metadata volume:

https://mellowhost.com/billing/index.php?rp=/knowledgebase/68/How-to-Extend-meta-data-of-a-thin-pool.html

1 Like
5

@mellowhost

I have some questions.

The first 3 commands I showed on the first part of my post is to show the errors.

  1. vgchange -a y pve
  2. lvchange -a y pve/data
  3. lvconvert --repair pve/data

All LVs under thin pool “data” are already inactive when I run the repair (#lvconvert --repair pve/data)

These are under the thin pool “data”

inactive ‘/dev/pve/vm-100-disk-1’ [10.00 GiB] inherit
inactive ‘/dev/pve/vm-101-disk-1’ [10.00 GiB] inherit
inactive ‘/dev/pve/vm-102-disk-1’ [32.00 GiB] inherit
inactive ‘/dev/pve/vm-103-disk-1’ [30.00 GiB] inherit
inactive ‘/dev/pve/vm-105-disk-1’ [32.00 GiB] inherit
inactive ‘/dev/pve/vm-106-disk-1’ [32.00 GiB] inherit
inactive ‘/dev/pve/vm-107-disk-1’ [32.00 GiB] inherit
inactive ‘/dev/pve/vm-104-disk-1’ [32.00 GiB] inherit
inactive ‘/dev/pve/vm-108-disk-1’ [32.00 GiB] inherit
inactive ‘/dev/pve/vm-109-disk-1’ [32.00 GiB] inherit
inactive ‘/dev/pve/vm-110-disk-1’ [160.00 GiB] inherit
inactive ‘/dev/pve/vm-111-disk-1’ [60.00 GiB] inherit
inactive ‘/dev/pve/vm-112-disk-1’ [60.00 GiB] inherit
inactive ‘/dev/pve/vm-113-disk-1’ [60.00 GiB] inherit

Would you suggest that I need to extend my metadata before repairing?
Do you know what does “bad checksum in superblock” means when I run the #lvconvert --repair pve/data?, does that mean that I need to check and repair a filesystem? If yes which one? The sda3?

lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 894.3G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 256M 0 part
└─sda3 8:3 0 446.9G 0 part
├─pve-root 251:0 0 96G 0 lvm /
└─pve-swap 251:1 0 8G 0 lvm

Doing a FSCK to /dev/sda3 from LiveCD doesn’t work either

[liveuser@localhost —]$ sudo e2fsck -y /dev/sda3 
e2fsck 1.42.9 (28-Dec-2013) 
ext2fs_open2: Bad magic number in super-block 
e2fsck: Superblock invalid, trying backup blocks... 
e2fsck: Bad magic number in super-block while trying to open /dev/sda3 

The superblock could not be read or does not describe a correct ext2 
filesystem. If the device is valid and it really contains an ext2 
filesystem (and not swap or ufs or something else), then the superblock 
is corrupt, and you might try running e2fsck with an alternate superblock: 
      e2fsck -b 8193 <device> 


[liveuser@localhost —]$ sudo e2fsck -b 8193 /dev/sda3 
e2fsck 1.42.9 (28-Dec-2013) 
e2fsck: Bad magic number in super-block while trying to open /dev/sda3 

The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: 
      e2fsck -b 8193 <device>
6

Hi,

Can you please post the output of the following command:

lvs -a
7

@mellowhost

This is what I got.

lvs -a

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy %Sync Convert
data pve twi—tz-- 326.88g
[data_tdata] pve Twi------- 326.88g
[data_tmeta] pve ewi------- 84.00m
[lvol0_pmspare] pve ewi------- 84.00m
root pve -wi-ao---- 96.00g
swap pve -wi-ao---- 8.00g
vm-100-disk-1 pve Vwi—tz-- 10.00g data
vm-101-disk-1 pve Vwi—tz-- 10.00g data
vm-102-disk-1 pve Vwi—tz-- 32.00g data
vm-103-disk-1 pve Vwi—tz-- 30.00g data
vm-104-disk-1 pve Vwi—tz-- 32.00g data
vm-105-disk-1 pve Vwi—tz-- 32.00g data
vm-106-disk-1 pve Vwi—tz-- 32.00g data
vm-107-disk-1 pve Vwi—tz-- 32.00g data
vm-108-disk-1 pve Vwi—tz-- 32.00g data
vm-109-disk-1 pve Vwi—tz-- 32.00g data
vm-110-disk-1 pve Vwi—tz-- 160.00g data
vm-111-disk-1 pve Vwi—tz-- 60.00g data
vm-112-disk-1 pve Vwi—tz-- 60.00g data
vm-113-disk-1 pve Vwi—tz-- 60.00g data

8

Did you miss something? Percentages are not added. Please try to use a screenshot instead.

9

@mellowhost
Nothing on the percentages column, screen shot is attached.166 - PuTTY

I think that column will only be available if the LVs are active.

10

This clearly suggest something wrong with the thin pool. Without properly looking into it on hand, I doubt, predicting anything further would be wise.

11

I’m in the same boat BIOS reset & thin pool stopped working | Proxmox Support Forum