Linux - Storage : hard drives, USB keys, ...

Filesystem in USErspace aka FUSE

fuse: bad mount point `/mount/point': Transport endpoint is not connected
fusermount -uz "/mount/point" && mount "/mount/point"
(source)

Comparison of file systems

FS + -
Btrfs for large server workloads
  • not for desktops
  • said to be slow, and fragments a lot (?)
ext2
ext3
obsolete, use ext4
ext4
  • good for desktop and server use
  • unclean shutdown-resistant
JFS
  • best of both worlds between ext4 and XFS (?)
  • fast, reliable, can stand load without eating CPU
  • less prone to fragmentation than ext3
  • lower performance than ext4 (?)
  • _may_ lose data because of postponing journal writes (?)
ReiserFS Performs great with numerous files + small files
  • Reiser3 is now obsolete
  • Reiser4 is still being maintained, but unlikely to be merged into mainline Linux without corporate backing
XFS
  • best for extremely large file systems, large files, and lots of files
  • great on parallel I/O
  • highly configurable
  • maybe overkill for desktop use
  • does not handle unclean shutdowns very well (?)
ZFS designed for storage servers being deprecated (?)
OpenZFS
  • No idea how this one compares to ZFS (and others), but don't confuse them
  • OpenZFS vs XFS

This is just an as-accurate-as-possible summary of my readings. There is a lot to say about filesystems (features, limitations, ...), and when looking for the "best" of them, the only shared answer is "It depends" (on what you value most : reliability, performance, scalability, ...).
Anyway, this research highlighted that ext4 is a pretty decent general-purpose solution, both for desktops and servers. Other filesystems will prove useful in special situations where specific features (RAID, SAN, snapshots, ...) are expected and some time is dedicated to tuning / administration.

How to use a NTFS drive ?

  1. apt-get install ntfs-3g
  2. plug the drive, identify + mount it
  3. append to /etc/fstab :
    UUID=1A0B4F9B2D10391B	/mount/point	ntfs-3g	noauto,users,uid=1000,gid=1000,dmask=022,fmask=133	0	0

So far, only root can mount a NTFS filesystem. However, thanks to the uid=1000,gid=1000 parameters, the corresponding non-root user has write access to the volume.

What if root gets write-access denied to the filesystem ?

Situation :

While logged as root, I get Permission denied errors when trying to write on the filesystem.

Details :

After investigation, I figured that (in my specific case) : Something's definitely read-only ! What ?

Solution :

Some points worth checking :
  1. have a look at directory permissions. Investigate parents just in case...
  2. make sure the filesystem is not full : df -h
  3. make sure the filesystem has some inodes left : df -i
  4. make sure the filesystem is mounted rw : mount
  5. have a look at logs, just in case something went wrong at boot time :
    • dmesg
    • less /var/log/messages
  6. if you go down the directories tree since /, touching a test file at each level to find out where the Permission denied error starts, you may find that the directory /my/directory/tree is read-only. Read its attributes by cding to its parent (/my/directory), then lsattr. If it outputs ----i-------- directory/tree, you're done : the directory is immutable. Reset this with chattr -i tree
  7. ...
  8. ...
  9. you may also consider fscking the filesystem...

How to mount an ext4 disk on a system which kernel only knows ext4 as a development filesystem ?

To do so, you'll have to ask the kernel to consider this filesystem as a test filesystem.

tune2fs -E test_fs /dev/sdb1 && mount -t ext4dev /dev/sdb1 /mnt/tmp

Details :

  1. specify advanced disk parameters : tune2fs -E test_fs /dev/sdb1.
    • -E is for tune2fs extended options.
    • test_fs is a flag instructing the kernel to use experimental code (such as ext4dev) to mount the disk.
  2. "debug" the filesystem to view/edit its state ( ??? not always necessary) and open it in r/w mode : debugfs -w /dev/sdb1
  3. check filesystem flags : tune2fs -l /dev/sdb1 | grep flags should output Filesystem flags: signed_directory_hash test_filesystem
  4. then mount : mount -t ext4dev /dev/sdb1 /mnt/tmp

Don't forget that on recent distributions, the ext4 filesystem is not available in "dev" anymore as it is now fully supported by the kernel.

How to rename a USB mass-storage device (=change its label ) ?

Situation :

When I plug a USB mass-storage device (USB key, external HDD, CF card, ...), it appears with a pre-defined name in Nautilus. How can I change this ?

Details :

Actually, the name used in Nautilus to refer to a USB mass-storage device is matched to this device mounting point : if the device is mounted in /media/device_xyz, the device will be displayed as device_xyz in Nautilus.

Fine then, but where does this device_xyz come from ? As most of the time USB storage devices are formatted with FAT filesystems (FAT 16 / FAT 32), their name is just the DOS partition label. To change this, 2 solutions :

Solution :

Smoothly change the partition label with mtools (as root on FAT volumes) :
  1. ignore warnings :
    echo mtools_skip_check=1 >> ~/.mtoolsrc
  2. read the label :
    mlabel -i /dev/sdg1 -s ::
  3. change the label :
    mlabel -i /dev/sdg1 ::newLabel

Alternate solution :

The quick-n-dirty method is to format the partition while giving it a new name :
mkfs.vfat -n newName /dev/sdf1

Which filesystem from a USB device should I mount ?

Situation :

I've plugged my USB key / external HDD, automount is OFF and I don't know which filesystem to mount

Details :

Which /dev/sdxx matches my USB device ?

Solution :

How to change a volume label ?

For EXT volumes (source 1, 2) :

e2label /dev/sdc1 newLabel

For FAT volumes :

How to rename a USB mass-storage device (=change its label ) ?

What are inodes ?

The i-node (or inode) of a file is its unique id in the FAT.

File names and directory facts about inodes (source) :

  • inodes store file metadata (ownership, access mode (read, write, execute permissions), and file type) but they don't hold the file name itself
  • Unix directories are lists of association structures, each of which contains one filename and one inode number. The file system driver must search a directory looking for a particular filename and then convert the filename to the correct corresponding inode number.
  • The operating system kernel's in-memory representation of this data is called struct inode in Linux.

Get a file's inode number :

ls -i fileName

Monitor inode usage :

df -hi

Every filesystem can handle a limited number of inodes (the inodes table uses space, so allowing a huge number of files on a filesystem means a big inode table, hence less space for files.). When running out of free inodes, there are 3 solutions :

  1. remove old / unused files
  2. move files to another filesystem
  3. dump the filesystem and re-create it with a larger inodes table

Delete a file from its inode number :

find path/to/be/searched -inum inodeNumberOfFileToDelete -delete

How to always mount a specific USB storage device at a specific mount point ?

For any personal reason, I'd like my USB key to always be mounted into /media/usbKey/ and my USB HDD into /media/usbHdd/. Indeed, depending on the DOS-label of the media (if any), and on the plugging order, I have no control on the mount point, which can be critical if I want to build scripts accessing content on removable media.

The UUID method :

  1. as root :
    blkid
    ...
    /dev/sdd1: LABEL="ConPow" UUID="05b3250f-3478-4831-8a85-c8da270839c3" TYPE="ext3"
    ...
    
  2. For the ConPow volume, add into /etc/fstab :
    UUID=05b3250f-3478-4831-8a85-c8da270839c3	/media/ConPow	ext3	noauto,users	0	0
  3. Don't forget to create the mount point and to manage permissions accordingly

Let non-root users mount + have write access to filesystems :

As root :
  1. specify either the user or users option in /etc/fstab
  2. manually mount the filesystem
  3. "give" the mount point to the user :
    chown bob:users /mount/point/
  4. unmount the filesystem
You should be able to mount this filesystem and write there as bob, now

The udev method :

This method is more complex and takes longer to apply than the UUID method above. It _may_ also be obsolete. It's still there only for "historical" reasons.
  1. Identify the device :
    • fdisk -l
    • or less /proc/partitions
    • or mount | grep "sd"
    • or cat /etc/mtab | grep "sd"
    which may output : /dev/sdf1 on /media/usbdisk type ext3 (rw,nosuid,nodev)
  2. Investigate the device :
    udevadm info --name=/dev/sdb1 --attribute-walk
    looking at device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0/block/sdb/sdb1':
    	KERNEL=="sdb1"
    	SUBSYSTEM=="block"
    	DRIVER==""
    	ATTR{partition}=="1"
    	ATTR{start}=="63"
    	ATTR{size}=="1250258562"
    	[...]
    
    looking at parent device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0/block/sdb':
    	KERNELS=="sdb"
    	SUBSYSTEMS=="block"
    	DRIVERS==""
    	ATTRS{range}=="16"
    	ATTRS{ext_range}=="256"
    	ATTRS{removable}=="0"
    	ATTRS{ro}=="0"
    	ATTRS{size}=="1250263728"
    	ATTRS{alignment_offset}=="0"
    	ATTRS{discard_alignment}=="0"
    	ATTRS{capability}=="50"
    	[...]
    
    looking at parent device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0':
    	KERNELS=="12:0:0:0"
    	SUBSYSTEMS=="scsi"
    	DRIVERS=="sd"
    	[...]
  3. Write the udev rule into /etc/udev/rules.d/10-local.rules :

    KERNEL=="sd?1", ATTRS{size}=="1250263728", ATTRS{capability}=="50", KERNEL=="sd?1", NAME="ConPow"

    More about udevinfo, udev rules syntax and basic udev rules
  4. Restart udev : service udev restart
  5. Create the mount point /media/ConPow
  6. Finally, to give it a fixed mount point, append to /etc/fstab :

    /dev/ConPow /media/ConPow ext3 auto,user 0 0

  7. Don't forget to unplug/replug the USB drive (or to /etc/init.d/udev restart)so that it gets re-discovered and handled by the udev rule.
  8. At this step, "normal" users still have not enough privileges to write on the device : set permissions

Additional stuff:

Instead of the UUID, it is possible to identify filesystems by their volume label. This volume label can be specified while creating the file system :

On existing filesystems, this can be done with tune2fs. Then, it is possible to mount filesystems, or refer to them in /etc/fstab with the label keyword.

A FAT32-formatted USB key suddenly complains bus error and is remounted read-only.

Situation :

  1. I plug my USB key and mount it normally
  2. I use it as usual for a while
  3. Suddenly, when trying to read from it, it says bus error.

What's going on ?

Details :

  1. mount outputs /dev/sdd1 on /media/myUsbKey type vfat (ro, ... ,errors=remount-ro)
    So it seems some error occurred on the device and -as instructed- it's been remounted read-only.
  2. dmesg is more verbose :
    [ 5962.398581] FAT-fs (sdd1): error, fat_get_cluster: invalid cluster chain (i_pos 14085575)
    [ 5985.072996] FAT-fs (sdd1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
    [ 5995.422661] FAT-fs (sdd1): error, fat_get_cluster: invalid cluster chain (i_pos 14085575)
    [ 5995.422671] FAT-fs (sdd1): Filesystem has been set read-only

For a detailed view on how things work, read this ServerFault discussion (or a local copy)

Solution :

  1. Try to fix the filesystem automatically : fsck.msdos -aw /dev/sdd1
  2. If this doesn't work, try a manual approach : fsck.vfat /dev/sdd1
    If it highlights differences on the "boot sector" :
    There are differences between boot sector and its backup.
    Differences: (offset:original/backup)
    	65:01/00
    1) Copy original to backup
    2) Copy backup to original
    3) No action
    ?
    think twice before overwriting usable data with 0's (have another look at the article I linked above !)
  3. As for me, this didn't work either, so I had to backup everything then format the USB drive and it works fine anew

How to clone a CD ?

  1. Insert the source CD into the reader, but do not mount it.
  2. Copy the source CD to the hard disk as a bitstream: dd if=/dev/hdx of=/path/to/image.iso
  3. Identify the SCSI fake number ID of the CD writer :
    cdrecord -scanbus
    It should output an ID such as : x,x,x
  4. To blank a CD-RW :
    cdrecord blank=fast dev=x,x,x
  5. Then burn :
    cdrecord -v speed=[2/4/6/8/10/12/...] dev=x,x,x /path/to/image.iso
  6. The CD writer unit is /dev/scdx

How to connect a USB Digital Camera (and play with SCSI emulation) ?

With modern distributions, all steps below are now handled automatically / differently and this whole procedure is now obsolete. I keep it for "historical" reasons.

  1. Ensure all the required kernel modules are loaded (usb-storage is mandatory, others are usually already loaded) : lsmod | grep 'usb'
  2. If usb-storage is not currently loaded : modprobe usb-storage
  3. Check that USB drivers (especially usb-storage) are ready : less /proc/bus/usb/drivers
  4. Connect the USB device, turn it on, then list all currently connected USB devices : less /proc/bus/usb/devices | grep "S:"
  5. Since USB devices are emulated through SCSI, list all found SCSI devices : less /proc/scsi/scsi
  6. Read complete information about the SCSI-emulated USB device : less /proc/scsi/usb-storage-0/x (x is the id of the SCSI device, starting at 0)
  7. Create a mount point : mkdir /mnt/usb
  8. SCSI devices are named sxxi, with :
    • xx : 2 letters identifying the type of device:
      • cd : CD, DVD, CD-Recorder unit
      • d[a-g] : storage device. The 1st one is da, then db, etc. The letter corresponds to the device order within all the devices of the same family (for example, the 1st HDD drive is sda even if there are other SCSI devices)
    • i : device or partition number. Keep in mind that FAT32 partitions usually start numbering at 5.
  9. mount the device : mount /dev/sda1 /mnt/usb