Structure needs cleaning error fix

I have recently experienced an error in my k8s setup “Relabel failed… structure needs cleaning”. As a result one of the k8s container/pod failed to start.

Here is the fix.

Personal notes: My personal notes on how to setup stuff to make it easier to repeat next time.

The problem was probably caused by earlier fsck check errors and force fixing these errors on Longhorn volume.

After that one of the folders had entries like this:

[root@oke-ckmqzdi4heq-ntiy6tib3ta-skse535hrna-2 1]# ls -l 
# ls: cannot access 'iamglass': Structure needs cleaning
total 10280
-rw-------. 1 mail mail    8192 Jul 14 12:07 docdata.glass
-rw-------. 1 mail mail       0 Aug 11 02:06 flintlock
-?????????? ? ?    ?          ?            ? iamglass
-rw-------. 1 mail mail    8192 Jul 14 12:07 position.glass
-rw-------. 1 mail mail 9936896 Aug 11 02:07 postlist.glass
-rw-------. 1 mail mail    8192 Jul 14 12:07 spelling.glass
-rw-------. 1 mail mail    8192 Jul 14 12:07 synonym.glass
-rw-------. 1 mail mail  860160 Aug 11 02:07 termlist.glass

And you cannot simply rm -rf iamglass.

As it appears, a solution is debugfs.

First, check the mountpoint for the folder:

[root@oke-ckmqzdi4heq-ntiy6tib3ta-skse535hrna-2]# lsblk
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                  8:0    0   300G  0 disk 
├─sda1               8:1    0   100M  0 part /boot/efi
├─sda2               8:2    0     1G  0 part /boot
└─sda3               8:3    0 298.9G  0 part 
  ├─ocivolume-root 252:0    0 288.9G  0 lvm  /
  └─ocivolume-oled 252:1    0    10G  0 lvm  /var/oled
sdb                  8:16   0     5G  0 disk /var/lib/kubelet/pods/ac47/volume-subpaths/pvc-4847/redis/0
sdc                  8:32   0     5G  0 disk /var/lib/kubelet/pods/c429/volumes/kubernetes.io~csi/pvc-1956/mount
sdd                  8:48   0     5G  0 disk /var/lib/kubelet/pods/7023/volumes/kubernetes.io~csi/pvc-592c/mount
sde                  8:64   0    10G  0 disk /var/lib/kubelet/pods/43a2/volumes/kubernetes.io~csi/pvc-b001/mount
sdf                  8:80   0   300G  0 disk /var/lib/kubelet/pods/5585/volume-subpaths/pvc-50e5/admin/0
[root@oke-ckmqzdi4heq-ntiy6tib3ta-skse535hrna-2 xapian-indexes]# 

Once you know which volume needs repair, run the following:

debugfs -w /dev/sdf
debugfs:

In my case this was /dev/sdf device. Now, at the debuggs: prompt use commands ls/cd until you get to the affected folder or file and then clean it up:

debugfs:  clri iamglass
debugfs:  rm iamglass

This should solve the problem.