Read only EXT4 filesystem (Ubuntu) & pending hard drive failure

I've been experiencing woes with a Samba server recently with all sorts of gremlins around access, and write access specifically. After gutting the Samba config several times to the bare minimum I happened to notice that mysteriously the whole EXT4 file system partition on which the Samba shares resided had become mounted as read only. After a little investigating I think the culprit is a hard drive on its way out. Ironically the drive in question is a Seagate drive and the image I think is therefore, rather suited. Read on...

I first detected the issue resided outside of Samba after logging on to the box and trying to manually copy, paste and create files on the filesystem where I was greeted with a read only error. After checking directory permission with ls -l and chmod'ing to 777 for good measure I ruled out file permissions as the issue.


My first port of call was to check how the disk had been mounted through using the built-in Ubuntu Disk Utility manager, which is accessible from:


System > Administration > Disk Utility


Immediately the SMART read out showed that the disk had a few bad sectors. On closer inspection of the SMART read out it was apparent that the Reallocated Sector Account was reading 13 sectors and the disk had tripped the SMART warning.


Some further hunting through the log files showed where the read only remount had occurred and the source of my problems. Needless to say the disk was backed up with Rsync and is now pending RMA, the drive a Seagate ST31500341AS is under a year old and it appears that this specific type of drive with the CC1H firmware are not problem free :(


I checked the system messages log:


cat /var/log/messages | grep read

[sdb] Add. Sense: Unrecovered read error - auto reallocate failed

Then onto the kernel log:


cat /var/log/kern.log | grep remount

Jul  6 02:17:54 MediaServer kernel: [89408.597785] sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]

Jul  6 02:17:54 MediaServer kernel: [89408.597798] Descriptor sense data with sense descriptors (in hex):

Jul  6 02:17:54 MediaServer kernel: [89408.597803]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 

Jul  6 02:17:54 MediaServer kernel: [89408.597825]         84 40 00 85 

Jul  6 02:17:54 MediaServer kernel: [89408.597834] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed

Jul  6 02:17:54 MediaServer kernel: [89408.597845] sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 84 40 00 7f 00 00 08 00

Jul  6 02:17:54 MediaServer kernel: [89408.597864] end_request: I/O error, dev sdb, sector 2218786949

Jul  6 02:17:54 MediaServer kernel: [89408.597902] EXT4-fs error (device sdb1): ext4_read_block_bitmap: Cannot read block bitmap - block_group = 8472, block_bitmap = 277348360

Jul  6 02:17:54 MediaServer kernel: [89408.597916] ata2: EH complete

Jul  6 02:17:54 MediaServer kernel: [89408.597920] 

Jul  6 02:17:54 MediaServer kernel: [89408.597927] Aborting journal on device sdb1-8.

Jul  6 02:17:54 MediaServer kernel: [89408.598460] EXT4-fs (sdb1): Remounting filesystem read-only

Jul  6 02:17:54 MediaServer kernel: [89408.598859] EXT4-fs error (device sdb1) in ext4_mb_free_blocks: IO failure

Jul  6 02:17:54 MediaServer kernel: [89408.598876] EXT4-fs error (device sdb1) in ext4_ext_remove_space: Journal has aborted

Jul  6 02:17:54 MediaServer kernel: [89408.598890] EXT4-fs error (device sdb1) in ext4_reserve_inode_write: Journal has aborted

Jul  6 02:17:54 MediaServer kernel: [89408.598897] EXT4-fs error (device sdb1) in ext4_ext_truncate: Journal has aborted

Jul  6 02:17:54 MediaServer kernel: [89408.598906] EXT4-fs error (device sdb1) in ext4_reserve_inode_write: Journal has aborted

Jul  6 02:17:54 MediaServer kernel: [89408.598912] EXT4-fs error (device sdb1) in ext4_orphan_del: Journal has aborted

Jul  6 02:17:54 MediaServer kernel: [89408.598919] EXT4-fs error (device sdb1) in ext4_reserve_inode_write: Journal has aborted

Jul  6 02:17:54 MediaServer kernel: [89408.599433] EXT4-fs (sdb1): ext4_da_writepages: jbd2_start: 1930 pages, ino 18877325; err -30