I have two dedicated NAS devices (a DNS-323 and a newer DNS-325) that hold pretty much all our data. I do have backups for our photos, however I’ve been lazy and hadn’t got around to backing up our home videos yet. Well last week when I tried accessing the DNS-323 I noticed it was inaccessible, after logging into the web portal I noticed the 323 was wanting to format the drive…. not good, now I start getting worried. I pulled my Western Digital 2TB Green drive from the DNS-323 and plugged in into my USB 3.0 SATA dock and plug into a Ubuntu machine… the partition would not automount. I couldn’t remember which partition should be mounted so I ran fdisk -l to list the partitions:
fdisk -l /dev/sdc Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xdd20b579 Device Boot Start End Blocks Id System /dev/sdc1 1 66 530112 82 Linux swap / Solaris Partition 1 does not end on cylinder boundary. /dev/sdc2 130 243201 1952469848+ 83 Linux /dev/sdc4 66 130 512040 83 Linux Partition 4 does not end on cylinder boundary. Partition table entries are not in disk order
Partition 2 is what I’m looking for, so I try mounting manually:
mount /dev/sdc2 /mnt/test2
mount: wrong fs type, bad option, bad superblock on /dev/sdc2,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
Next I run dumpe2fs:
dumpe2fs /dev/sdc2 dumpe2fs 1.41.12 (17-May-2010) dumpe2fs: The ext2 superblock is corrupt while trying to open /dev/sdc2 Couldn't find valid filesystem superblock.
Now I’m really getting worried and before anything else I decide to wait and buy a new bigger drive so I can clone this one. I ended up getting a 3TB Seagate and cloned using dd:
dd if=/dev/sdc of=/dev/sdd bs=512 conv=noerror,sync
10 hours later I have a copy of my WD 2TB drive and ready for more attempts at recovery. I previously had good results recovering SD cards and the like with EaseUS recovery software which runs in Windows but supposedly can recover ext2 partitions. After searching 14 hours it had found 100 or so NTFS partitions to recover and all with crazy directory structures that never would have existed on that drive. I did recover bogus image and movie files that definitely could not be opened by anything. After wasting a couple of days with Windows recovery tools it was back to Linux tools where I found pointers to Testdisk. Using testdisk I was able to find backup superblocks and the blocksize.
Start testdisk –> create (log file) –> select disk –> Intel –> Advanced —> select partition
(Also check out the CG Security page on this here)
And up came the following:
TestDisk 6.11, Data Recovery Utility, April 2009
Christophe GRENIER
http://www.cgsecurity.org
Disk /dev/sdc - 2000 GB / 1863 GiB - CHS 243201 255 63
Partition Start End Size in sectors
Linux 129 190 14 243200 254 62 3904939696
superblock 0, blocksize=4096 []
superblock 98304, blocksize=4096 []
superblock 163840, blocksize=4096 []
superblock 229376, blocksize=4096 []
superblock 819200, blocksize=4096 []
superblock 884736, blocksize=4096 []
superblock 1605632, blocksize=4096 []
superblock 2654208, blocksize=4096 []
superblock 4096000, blocksize=4096 []
superblock 7962624, blocksize=4096 []
So it found a superblock at 98304 with blocksize 4096, now I run the following:
fcsk -b 98304 -B 4096 -n /dev/sdb2
The output was thousands of lines of errors (here’s a very small sample):
Inode 16082, i_blocks is 4016691468, should be 0. Fix? no Inode 16083 is in use, but has dtime set. Fix? no Inode 16083 has imagic flag set. Clear? no Inode 16083 has a extra size (36805) which is invalid Fix? no Inode 13085, i_size is 14734504452836535872, should be 0. Fix? no Inode 13085, i_blocks is 2731559226, should be 0. Fix? no Inode 8500 has a bad extended attribute block 440923962. Clear? no Extended attribute block 440923962 has h_blocks > 1. Clear? no Inode 8500 has illegal block(s). Clear? no Illegal block #0 (1681097748) in inode 8500. IGNORED. Illegal block #1 (1139813161) in inode 8500. IGNORED. Inode 8500 is too big. Truncate? no Block #2 (43330056) causes symlink to be too big. IGNORED. Illegal block #3 (1979478033) in inode 8500. IGNORED. Illegal block #4 (3764348932) in inode 8500. IGNORED. Illegal block #5 (3662943551) in inode 8500. IGNORED. Illegal block #6 (3755323347) in inode 8500. IGNORED. Block #7 (466222846) causes symlink to be too big. IGNORED. Block #8 (214100409) causes symlink to be too big. IGNORED. Block #9 (15168821) causes symlink to be too big. IGNORED. Illegal block #10 (3738791589) in inode 8500. IGNORED. Too many illegal blocks in inode 8500. Clear inode? no
After this comes the real test… to see if I can recover my filesystem. I run this on the new 3TB drive which was cloned to match the 2TB drive and include the -y option so as not to be prompted for all fixes.
fsck -b 98304 -B 4096 -y /dev/sdb2
I let this run over an hour and then decided I should just head to bed and let it run overnight. Next morning when I got up fsck had finished it’s thing and I was pleasantly surprised to find out I could mount my partition ok. My directory structure was intact and as far as I could tell all my home videos are safe and sound, so I immediately copied that directory elsewhere.
My plan going forward is to keep the new 3TB drive in my Proxmox server. An Ubuntu VM on the Proxmox server will mount that drive and act as my file server/NAS via samba, nfs and minidlna. My Dlink DNS-325 will remain on location as the backup for the most important files. The DNS-323 will be brought to a remote location and serve as secondary backup via IPSec VPN and rsync.