Linux - Storage : hard drives, USB keys, ...

Can't mount NTFS-formatted USB drive : Error mounting /dev/sde1 at /mount/point: Unknown error when mounting /dev/sde1

Situation :

/dev/sde1 : When I plug it on the Debian machine, I get this message on an pop-up window of the graphical interface :
Error mounting /dev/sde1 at /mount/point: Unknown error when mounting /dev/sde1

Details :

fdisk -l /dev/sde1
This command and further interactive checks with fdisk list data normally :
  • device name and manufacturer
  • partition label, size
  • ability to read partition table, toggle flags, write changes, undo them, ...
showing there seem to be no hardware damage to this device .
mount /dev/sde1 /mount/point
ntfs_mst_post_read_fixup_warn: magic: 0x5acebe02  size: 4096   usa_ofs: 30771  usa_count: 3182: Invalid argument
Actual VCN (0x57f9577b3dca807b) of index buffer is different from expected VCN (0x0).
Failed to open $Secure: No such file or directory
Failed to mount '/dev/sde1': No such file or directory
ntfsfix -n /dev/sde
(source)
Mounting volume... NTFS signature is missing.
FAILED
Attempting to correct errors... NTFS signature is missing.
FAILED
Failed to startup volume: Invalid argument
NTFS signature is missing.
Trying the alternate boot sector
The alternate bootsector is usable
Set sector count to 60437491 instead of 60435443
The startup data can be fixed, but no change was requested
Volume is corrupt. You should run chkdsk.
No change made			ok since this was a dry-run
ntfsfix -d /dev/sde
Mounting volume... NTFS signature is missing.
FAILED
Attempting to correct errors... NTFS signature is missing.
FAILED
Failed to startup volume: Invalid argument
NTFS signature is missing.
Trying the alternate boot sector
The alternate bootsector is usable
Set sector count to 60437491 instead of 60435443
Rewriting the bootsector
The boot sector has been rewritten
ntfs_mst_post_read_fixup_warn: magic: 0x00000000  size: 1024   usa_ofs: 0  usa_count: 0: Invalid argument
Record 0 has no FILE magic (0x0)
Failed to load $MFT: Input/output error
Volume is corrupt. You should run chkdsk.			will do, thx 
chkdsk (on the W10 box) was helpless)
chkdsk on the Windows machine :
Sadly, chkdsk provided no useful help since it was only able to complain:
Windows cannot access the F: drive.
  • At least, it didn't suggest to contact my administrator .
  • The good news is that the flash drive was "alive" enough to get a drive letter (which it didn't get before using ntfsfix.)
/sbin/badblocks -vs /dev/sde
(source)
Checking blocks 0 to 30218745
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found. (0/0/0 errors)
This took about 20 minutes for a 32GB drive.

Solution :

Looks like this is "only" a software error, even though I have no idea about the root cause. I've not been able to fix it (and save the data ), and the solution was to :
  1. delete + re-create the partition, with fdisk in interactive mode
  2. format it :
    mkfs.ntfs -f -L myUsbFlashDrive /dev/sde1
    • -f : fast format : skip both zeroing of the volume and bad sector checking
    • -L volumeLabel : set the volume label to volumeLabel
  3. And voilà !

Filesystem in USErspace aka FUSE

fuse: bad mount point `/mount/point': Transport endpoint is not connected
fusermount -uz "/mount/point" && mount "/mount/point"
(source)

Comparison of file systems

FS + -
Btrfs for large server workloads
  • not for desktops
  • said to be slow, and fragments a lot (?)
ext2
ext3
obsolete, use ext4
ext4
  • good for desktop and server use
  • unclean shutdown-resistant
JFS
  • best of both worlds between ext4 and XFS (?)
  • fast, reliable, can stand load without eating CPU
  • less prone to fragmentation than ext3
  • lower performance than ext4 (?)
  • _may_ lose data because of postponing journal writes (?)
ReiserFS Performs great with numerous files + small files
  • Reiser3 is now obsolete
  • Reiser4 is still being maintained, but unlikely to be merged into mainline Linux without corporate backing
XFS
  • best for extremely large file systems, large files, and lots of files
  • great on parallel I/O
  • highly configurable
  • maybe overkill for desktop use
  • does not handle unclean shutdowns very well (?)
ZFS designed for storage servers being deprecated (?)
OpenZFS
  • No idea how this one compares to ZFS (and others), but don't confuse them
  • OpenZFS vs XFS

This is just an as-accurate-as-possible summary of my readings. There is a lot to say about filesystems (features, limitations, ...), and when looking for the "best" of them, the only shared answer is "It depends" (on what you value most : reliability, performance, scalability, ...).
Anyway, this research highlighted that ext4 is a pretty decent general-purpose solution, both for desktops and servers. Other filesystems will prove useful in special situations where specific features (RAID, SAN, snapshots, ...) are expected and some time is dedicated to tuning / administration.

How to use a NTFS drive ?

  1. apt-get install ntfs-3g
  2. plug the drive, identify + mount it
  3. append to /etc/fstab :
    UUID=1A0B4F9B2D10391B	/mount/point	ntfs-3g	noauto,users,uid=1000,gid=1000,dmask=022,fmask=133	0	0

So far, only root can mount a NTFS filesystem. However, thanks to the uid=1000,gid=1000 parameters, the corresponding non-root user has write access to the volume.

What if root gets write-access denied to the filesystem ?

Situation :

While logged as root, I get Permission denied errors when trying to write on the filesystem.

Details :

After investigation, I figured that (in my specific case) : Something's definitely read-only ! What ?

Solution :

Some points worth checking :
  1. have a look at directory permissions. Investigate parents just in case...
  2. make sure the filesystem is not full : df -h
  3. make sure the filesystem has some inodes left : df -i
  4. make sure the filesystem is mounted rw : mount
  5. have a look at logs, just in case something went wrong at boot time :
    • dmesg
    • less /var/log/messages
  6. if you go down the directories tree since /, touching a test file at each level to find out where the Permission denied error starts, you may find that the directory /my/directory/tree is read-only. Read its attributes by cding to its parent (/my/directory), then lsattr. If it outputs ----i-------- directory/tree, you're done : the directory is immutable. Reset this with chattr -i tree
  7. ...
  8. ...
  9. you may also consider fscking the filesystem...

How to mount an ext4 disk on a system which kernel only knows ext4 as a development filesystem ?

To do so, you'll have to ask the kernel to consider this filesystem as a test filesystem.

tune2fs -E test_fs /dev/sdb1 && mount -t ext4dev /dev/sdb1 /mnt/tmp

Details :

  1. specify advanced disk parameters : tune2fs -E test_fs /dev/sdb1.
    • -E is for tune2fs extended options.
    • test_fs is a flag instructing the kernel to use experimental code (such as ext4dev) to mount the disk.
  2. "debug" the filesystem to view/edit its state ( ??? not always necessary) and open it in r/w mode : debugfs -w /dev/sdb1
  3. check filesystem flags : tune2fs -l /dev/sdb1 | grep flags should output Filesystem flags: signed_directory_hash test_filesystem
  4. then mount : mount -t ext4dev /dev/sdb1 /mnt/tmp

Don't forget that on recent distributions, the ext4 filesystem is not available in "dev" anymore as it is now fully supported by the kernel.

How to rename a USB mass-storage device (=change its label ) ?

Situation :

When I plug a USB mass-storage device (USB key, external HDD, CF card, ...), it appears with a pre-defined name in Nautilus. How can I change this ?

Details :

Actually, the name used in Nautilus to refer to a USB mass-storage device is matched to this device mounting point : if the device is mounted in /media/device_xyz, the device will be displayed as device_xyz in Nautilus.

Fine then, but where does this device_xyz come from ? As most of the time USB storage devices are formatted with FAT filesystems (FAT 16 / FAT 32), their name is just the DOS partition label. To change this, 2 solutions :

Solution :

Smoothly change the partition label with mtools (as root on FAT volumes) :
  1. ignore warnings :
    echo mtools_skip_check=1 >> ~/.mtoolsrc
  2. read the label :
    mlabel -i /dev/sdg1 -s ::
  3. change the label :
    mlabel -i /dev/sdg1 ::newLabel

Alternate solution :

The quick-n-dirty method is to format the partition while giving it a new name :
mkfs.vfat -n newName /dev/sdf1

Which filesystem from a USB device should I mount ?

Situation :

I've plugged my USB key / external HDD, automount is OFF and I don't know which filesystem to mount

Details :

Which /dev/sdxx matches my USB device ?

Solution :

How to change a volume label ?

For EXT volumes (source 1, 2) :

e2label /dev/sdc1 newLabel

For FAT volumes :

How to rename a USB mass-storage device (=change its label ) ?

What are inodes ?

File names and directory facts about inodes (source) :

  • inodes store file metadata (ownership, access mode (read, write, execute permissions), and file type) but they don't hold the file name itself
  • Unix directories are lists of association structures, each of which contains one filename and one inode number. The file system driver must search a directory looking for a particular filename and then convert the filename to the correct corresponding inode number.
  • The operating system kernel's in-memory representation of this data is called struct inode in Linux.

Get a file's inode number :

ls -i fileName

Monitor inode usage :

df -hi

Every filesystem can handle a limited number of inodes (the inodes table uses space, so allowing a huge number of files on a filesystem means a big inode table, hence less space for files.). When running out of free inodes, there are 3 solutions :

  1. remove old / unused files
  2. move files to another filesystem
  3. dump the filesystem and re-create it with a larger inodes table

Delete a file from its inode number :

find path/to/be/searched -inum inodeNumberOfFileToDelete -delete

How to always mount a specific USB storage device at a specific mount point ?

For any personal reason, I'd like my USB key to always be mounted into /media/usbKey/ and my USB HDD into /media/usbHdd/. Indeed, depending on the DOS-label of the media (if any), and on the plugging order, I have no control on the mount point, which can be critical if I want to build scripts accessing content on removable media.

The UUID method :

  1. as root :
    blkid
    
    /dev/sdd1: LABEL="ConPow" UUID="05b3250f-3478-4831-8a85-c8da270839c3" TYPE="ext3"
    
    
  2. For the ConPow volume, add into /etc/fstab :
    UUID=05b3250f-3478-4831-8a85-c8da270839c3	/media/ConPow	ext3	noauto,users	0	0
  3. Don't forget to create the mount point and to manage permissions accordingly

Let non-root users mount + have write access to filesystems :

As root :
  1. specify either the user or users option in /etc/fstab
  2. manually mount the filesystem
  3. "give" the mount point to the user :
    chown bob:users /mount/point/
  4. unmount the filesystem
You should be able to mount this filesystem and write there as bob, now

The udev method :

This method is more complex and takes longer to apply than the UUID method above. It _may_ also be obsolete. It's still there only for "historical" reasons.
  1. Identify the device :
    • fdisk -l
    • or less /proc/partitions
    • or mount | grep "sd"
    • or cat /etc/mtab | grep "sd"
    which may output : /dev/sdf1 on /media/usbdisk type ext3 (rw,nosuid,nodev)
  2. Investigate the device :
    udevadm info --name=/dev/sdb1 --attribute-walk
    looking at device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0/block/sdb/sdb1':
    	KERNEL=="sdb1"
    	SUBSYSTEM=="block"
    	DRIVER==""
    	ATTR{partition}=="1"
    	ATTR{start}=="63"
    	ATTR{size}=="1250258562"
    	[...]
    
    looking at parent device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0/block/sdb':
    	KERNELS=="sdb"
    	SUBSYSTEMS=="block"
    	DRIVERS==""
    	ATTRS{range}=="16"
    	ATTRS{ext_range}=="256"
    	ATTRS{removable}=="0"
    	ATTRS{ro}=="0"
    	ATTRS{size}=="1250263728"
    	ATTRS{alignment_offset}=="0"
    	ATTRS{discard_alignment}=="0"
    	ATTRS{capability}=="50"
    	[...]
    
    looking at parent device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0':
    	KERNELS=="12:0:0:0"
    	SUBSYSTEMS=="scsi"
    	DRIVERS=="sd"
    	[...]
  3. Write the udev rule into /etc/udev/rules.d/10-local.rules :

    KERNEL=="sd?1", ATTRS{size}=="1250263728", ATTRS{capability}=="50", KERNEL=="sd?1", NAME="ConPow"

    More about udevinfo, udev rules syntax and basic udev rules
  4. Restart udev : service udev restart
  5. Create the mount point /media/ConPow
  6. Finally, to give it a fixed mount point, append to /etc/fstab :

    /dev/ConPow /media/ConPow ext3 auto,user 0 0

  7. Don't forget to unplug/replug the USB drive (or to /etc/init.d/udev restart)so that it gets re-discovered and handled by the udev rule.
  8. At this step, "normal" users still have not enough privileges to write on the device : set permissions

Additional stuff:

Instead of the UUID, it is possible to identify filesystems by their volume label. This volume label can be specified while creating the file system :

On existing filesystems, this can be done with tune2fs. Then, it is possible to mount filesystems, or refer to them in /etc/fstab with the label keyword.

A FAT32-formatted USB key suddenly complains bus error and is remounted read-only.

Situation :

  1. I plug my USB key and mount it normally
  2. I use it as usual for a while
  3. Suddenly, when trying to read from it, it says bus error.

What's going on ?

Details :

  1. mount outputs /dev/sdd1 on /media/myUsbKey type vfat (ro, ... ,errors=remount-ro)
    So it seems some error occurred on the device and -as instructed- it's been remounted read-only.
  2. dmesg is more verbose :
    [ 5962.398581] FAT-fs (sdd1): error, fat_get_cluster: invalid cluster chain (i_pos 14085575)
    [ 5985.072996] FAT-fs (sdd1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
    [ 5995.422661] FAT-fs (sdd1): error, fat_get_cluster: invalid cluster chain (i_pos 14085575)
    [ 5995.422671] FAT-fs (sdd1): Filesystem has been set read-only

For a detailed view on how things work, read this ServerFault discussion (or a local copy)

Solution :

  1. Try to fix the filesystem automatically : fsck.msdos -aw /dev/sdd1
  2. If this doesn't work, try a manual approach : fsck.vfat /dev/sdd1
    If it highlights differences on the "boot sector" :
    There are differences between boot sector and its backup.
    Differences: (offset:original/backup)
    	65:01/00
    1) Copy original to backup
    2) Copy backup to original
    3) No action
    ?
    think twice before overwriting usable data with 0's (have another look at the article I linked above !)
  3. As for me, this didn't work either, so I had to backup everything then format the USB drive and it works fine anew

How to clone a CD ?

  1. Insert the source CD into the reader, but do not mount it.
  2. Copy the source CD to the hard disk as a bitstream: dd if=/dev/hdx of=/path/to/image.iso
  3. Identify the SCSI fake number ID of the CD writer :
    cdrecord -scanbus
    It should output an ID such as : x,x,x
  4. To blank a CD-RW :
    cdrecord blank=fast dev=x,x,x
  5. Then burn :
    cdrecord -v speed=[2/4/6/8/10/12/...] dev=x,x,x /path/to/image.iso
  6. The CD writer unit is /dev/scdx

How to connect a USB Digital Camera (and play with SCSI emulation) ?

With modern distributions, all steps below are now handled automatically / differently and this whole procedure is now obsolete. I keep it for "historical" reasons.

  1. Ensure all the required kernel modules are loaded (usb-storage is mandatory, others are usually already loaded) : lsmod | grep 'usb'
  2. If usb-storage is not currently loaded : modprobe usb-storage
  3. Check that USB drivers (especially usb-storage) are ready : less /proc/bus/usb/drivers
  4. Connect the USB device, turn it on, then list all currently connected USB devices : less /proc/bus/usb/devices | grep "S:"
  5. Since USB devices are emulated through SCSI, list all found SCSI devices : less /proc/scsi/scsi
  6. Read complete information about the SCSI-emulated USB device : less /proc/scsi/usb-storage-0/x (x is the id of the SCSI device, starting at 0)
  7. Create a mount point : mkdir /mnt/usb
  8. SCSI devices are named sxxi, with :
    • xx : 2 letters identifying the type of device:
      • cd : CD, DVD, CD-Recorder unit
      • d[a-g] : storage device. The 1st one is da, then db, etc. The letter corresponds to the device order within all the devices of the same family (for example, the 1st HDD drive is sda even if there are other SCSI devices)
    • i : device or partition number. Keep in mind that FAT32 partitions usually start numbering at 5.
  9. mount the device : mount /dev/sda1 /mnt/usb