DNS-323: Recovering your corrupt ext2 filesystem

I have two dedicated NAS devices (a DNS-323 and a newer DNS-325) that hold pretty much all our data. I do have backups for our photos, however I’ve been lazy and hadn’t got around to backing up our home videos yet. Well last week when I tried accessing the DNS-323 I noticed it was inaccessible, after logging into the web portal I noticed the 323 was wanting to format the drive…. not good, now I start getting worried. I pulled my Western Digital 2TB Green drive from the DNS-323 and plugged in into my USB 3.0 SATA dock and plug into a Ubuntu machine… the partition would not automount. I couldn’t remember which partition should be mounted so I ran fdisk -l to list the partitions:

fdisk -l /dev/sdc

Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdd20b579

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1          66      530112   82  Linux swap / Solaris
Partition 1 does not end on cylinder boundary.
/dev/sdc2             130      243201  1952469848+  83  Linux
/dev/sdc4              66         130      512040   83  Linux
Partition 4 does not end on cylinder boundary.

Partition table entries are not in disk order

Partition 2 is what I’m looking for, so I try mounting manually:

mount /dev/sdc2 /mnt/test2
mount: wrong fs type, bad option, bad superblock on /dev/sdc2,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

Next I run dumpe2fs:

dumpe2fs /dev/sdc2
dumpe2fs 1.41.12 (17-May-2010)
dumpe2fs: The ext2 superblock is corrupt while trying to open /dev/sdc2
Couldn't find valid filesystem superblock.

Now I’m really getting worried and before anything else I decide to wait and buy a new bigger drive so I can clone this one. I ended up getting a 3TB Seagate and cloned using dd:

dd if=/dev/sdc of=/dev/sdd bs=512 conv=noerror,sync

10 hours later I have a copy of my WD 2TB drive and ready for more attempts at recovery. I previously had good results recovering SD cards and the like with EaseUS recovery software which runs in Windows but supposedly can recover ext2 partitions. After searching 14 hours it had found 100 or so NTFS partitions to recover and all with crazy directory structures that never would have existed on that drive. I did recover bogus image and movie files that definitely could not be opened by anything. After wasting a couple of days with Windows recovery tools it was back to Linux tools where I found pointers to Testdisk. Using testdisk I was able to find backup superblocks and the blocksize.

Start testdisk –> create (log file) –> select disk –> Intel –> Advanced —> select partition
(Also check out the CG Security page on this here)

And up came the following:

TestDisk 6.11, Data Recovery Utility, April 2009
Christophe GRENIER 


Disk /dev/sdc - 2000 GB / 1863 GiB - CHS 243201 255 63

     Partition                  Start        End    Size in sectors

  Linux                  129 190 14 243200 254 62 3904939696
superblock 0, blocksize=4096 []
superblock 98304, blocksize=4096 []
superblock 163840, blocksize=4096 []
superblock 229376, blocksize=4096 []
superblock 819200, blocksize=4096 []
superblock 884736, blocksize=4096 []
superblock 1605632, blocksize=4096 []
superblock 2654208, blocksize=4096 []
superblock 4096000, blocksize=4096 []
superblock 7962624, blocksize=4096 []

So it found a superblock at 98304 with blocksize 4096, now I run the following:

fcsk -b 98304 -B 4096 -n /dev/sdb2

The output was thousands of lines of errors (here’s a very small sample):

Inode 16082, i_blocks is 4016691468, should be 0.  Fix? no

Inode 16083 is in use, but has dtime set.  Fix? no

Inode 16083 has imagic flag set.  Clear? no

Inode 16083 has a extra size (36805) which is invalid
Fix? no

Inode 13085, i_size is 14734504452836535872, should be 0.  Fix? no

Inode 13085, i_blocks is 2731559226, should be 0.  Fix? no

Inode 8500 has a bad extended attribute block 440923962.  Clear? no

Extended attribute block 440923962 has h_blocks > 1.  Clear? no

Inode 8500 has illegal block(s).  Clear? no

Illegal block #0 (1681097748) in inode 8500.  IGNORED.
Illegal block #1 (1139813161) in inode 8500.  IGNORED.
Inode 8500 is too big.  Truncate? no

Block #2 (43330056) causes symlink to be too big.  IGNORED.
Illegal block #3 (1979478033) in inode 8500.  IGNORED.
Illegal block #4 (3764348932) in inode 8500.  IGNORED.
Illegal block #5 (3662943551) in inode 8500.  IGNORED.
Illegal block #6 (3755323347) in inode 8500.  IGNORED.
Block #7 (466222846) causes symlink to be too big.  IGNORED.
Block #8 (214100409) causes symlink to be too big.  IGNORED.
Block #9 (15168821) causes symlink to be too big.  IGNORED.
Illegal block #10 (3738791589) in inode 8500.  IGNORED.
Too many illegal blocks in inode 8500.
Clear inode? no

After this comes the real test… to see if I can recover my filesystem. I run this on the new 3TB drive which was cloned to match the 2TB drive and include the -y option so as not to be prompted for all fixes.

fsck -b 98304 -B 4096 -y /dev/sdb2

I let this run over an hour and then decided I should just head to bed and let it run overnight. Next morning when I got up fsck had finished it’s thing and I was pleasantly surprised to find out I could mount my partition ok. My directory structure was intact and as far as I could tell all my home videos are safe and sound, so I immediately copied that directory elsewhere.

My plan going forward is to keep the new 3TB drive in my Proxmox server. An Ubuntu VM on the Proxmox server will mount that drive and act as my file server/NAS via samba, nfs and minidlna. My Dlink DNS-325 will remain on location as the backup for the most important files. The DNS-323 will be brought to a remote location and serve as secondary backup via IPSec VPN and rsync.

Leave a Reply