Linux - Storage : hard drives, USB keys, ...

mail

How to find which filesystem any specific file belongs to ?

df path/to/file

Example :

Filesystem		1K-blocks	Used		Available	Use%	Mounted on
/dev/mapper/vg-home	25066604	4513268		19256952	19%	/home

mail

How to mount filesystems automatically at boot ?

There are several options to automatically mount a filesystem at boot time :
mail

How to optimize SSD usage ?

  1. The only SSD drives I'm using so far are in Debian machines, so I will only list commands and tools related to this operating system. Nevertheless, overall concepts may apply to other environments.
  2. "Optimizing" means finding the best mix of :
    • read / write performance
    • drive lifetime
    • configuration / tweaking time
  3. The Internetz suggests lots of things / hacks. Some are true, some are not, and most of them may just be obsolete.
Statement : "You MUST ..." Status Comment
... buy a SLC SSD drive it depends
  • SLC, MLC and TLC SSD drives all have their pro's and con's. From the fastest / highest number of writes / most expensive to the slowest / lowest writes / cheapest (sources : 1, 2) :
    1. SLC
    2. MLC
    3. TLC
  • buy one that fits both your needs and your budget
... leave space for over-provisioning true
  • this is considered a safe practice, the only dispute being the amount of unallocated space : 7% of the drive capacity, with a maximum of 10GB. Newer drives have some built-in unallocated space for this, which isn't accessible to the user, so no need to leave unused space. (source)
  • this is not completely clear to me whether "over-provisioning" is :
    • having some free space on the drive (within partitioned space)
    • OR leaving some partitioned space
... align partitions to 32 bits / 4K false
  • the requirement to align partitions was obsolete even before SSDs existed. This only comes from a limitation in the BIOS itself, and has nothing to do with SSD drives or GNU/Linux (source)
  • since Wheezy, tools handle this automatically
... have SWAP and tmpfs filesystems NOT on the SSD true
  • this will save writes on the SSD and make it last longer
  • Debian already puts tmpfs filesystems in memory (details)
... use ext4 true
  • ext4 works just fine with SSDs (sources : 1, 2)
... use the noatime mount option true
... use the discard mount option false
... use the data=ordered mount option false
... setup a regular TRIM with cron false
  • TRIM is not necessary : in some situations, TRIM can improve speed —in other cases, it can make the system significantly slower. And it is only ever a help until the disk is getting fairly full.
  • What do you lose if you DON'T enable TRIM ?
    When a filesystem deletes a file, it knows the logical blocks are free, but the SSD keeps them around. When the filesystem re-uses them for new data, the SSD then knows that the old physical blocks can be garbage-collected and re-used. So all you are really doing by not using TRIM is delaying the collection of unneeded blocks. As long as the SSD has plenty of spare blocks (remember over-provisioning ?), TRIM gains you nothing at all here.
(source)
... worry about wear leveling TODO
... use SSDs in RAID for extreme performance TODO I've read somewhere this is a bad idea, offering no performance improvement
mail

Can't mount NTFS-formatted USB drive : Error mounting /dev/sde1 at /mount/point: Unknown error when mounting /dev/sde1

Situation

/dev/sde1 : When I plug it on the Debian machine, I get this message on an pop-up window of the graphical interface :
Error mounting /dev/sde1 at /mount/point: Unknown error when mounting /dev/sde1

Details

fdisk -l /dev/sde1
This command and further interactive checks with fdisk list data normally :
  • device name and manufacturer
  • partition label, size
  • ability to read partition table, toggle flags, write changes, undo them, ...
showing there seem to be no hardware damage to this device .
mount /dev/sde1 /mount/point
ntfs_mst_post_read_fixup_warn: magic: 0x5acebe02  size: 4096   usa_ofs: 30771  usa_count: 3182: Invalid argument
Actual VCN (0x57f9577b3dca807b) of index buffer is different from expected VCN (0x0).
Failed to open $Secure: No such file or directory
Failed to mount '/dev/sde1': No such file or directory
ntfsfix -n /dev/sde
(source)
Mounting volume... NTFS signature is missing.
FAILED
Attempting to correct errors... NTFS signature is missing.
FAILED
Failed to startup volume: Invalid argument
NTFS signature is missing.
Trying the alternate boot sector
The alternate bootsector is usable
Set sector count to 60437491 instead of 60435443
The startup data can be fixed, but no change was requested
Volume is corrupt. You should run chkdsk.
No change made			ok since this was a dry-run
ntfsfix -d /dev/sde
Mounting volume... NTFS signature is missing.
FAILED
Attempting to correct errors... NTFS signature is missing.
FAILED
Failed to startup volume: Invalid argument
NTFS signature is missing.
Trying the alternate boot sector
The alternate bootsector is usable
Set sector count to 60437491 instead of 60435443
Rewriting the bootsector
The boot sector has been rewritten
ntfs_mst_post_read_fixup_warn: magic: 0x00000000  size: 1024   usa_ofs: 0  usa_count: 0: Invalid argument
Record 0 has no FILE magic (0x0)
Failed to load $MFT: Input/output error
Volume is corrupt. You should run chkdsk.			will do, thx 
chkdsk (on the W10 box) was helpless)
chkdsk on the Windows machine :
Sadly, chkdsk provided no useful help since it was only able to complain:
Windows cannot access the F: drive.
  • At least, it didn't suggest to contact my administrator .
  • The good news is that the flash drive was "alive" enough to get a drive letter (which it didn't get before using ntfsfix.)
/sbin/badblocks -vs /dev/sde
(source)
Checking blocks 0 to 30218745
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found. (0/0/0 errors)
This took about 20 minutes for a 32GB drive.

Solution

Looks like this is "only" a software error, even though I have no idea about the root cause. I've not been able to fix it (and save the data ), and the solution was to :
  1. delete + re-create the partition, with fdisk in interactive mode
  2. format it :
    mkfs.ntfs -f -L myUsbFlashDrive /dev/sde1
    • -f : fast format : skip both zeroing of the volume and bad sector checking
    • -L volumeLabel : set the volume label to volumeLabel
  3. And voilà !
mail

Filesystem in USErspace aka FUSE

fuse: bad mount point `/mount/point': Transport endpoint is not connected
fusermount -uz "/mount/point" && mount "/mount/point"
(source)
mail

Comparison of file systems

FS + -
Btrfs (1, 2, 3, 4) for large server workloads
  • not for desktops
  • said to be slow, and fragments a lot
ext2
ext3
obsolete, use ext4
ext4
  • good for desktop and server use
  • unclean shutdown-resistant
JFS
  • best of both worlds between ext4 and XFS
  • fast, reliable, can stand load without eating CPU
  • less prone to fragmentation than ext3
  • lower performance than ext4
  • _may_ lose data because of postponing journal writes
OpenZFS
ReiserFS Performs great with numerous files + small files
  • Reiser3 is now obsolete
  • Reiser4 is still being maintained, but unlikely to be merged into mainline Linux without corporate backing
XFS
  • best for extremely large file systems, large files, and lots of files
  • great on parallel I/O
  • highly configurable
  • maybe overkill for desktop use
  • does not handle unclean shutdowns very well
ZFS designed for storage servers being deprecated

The table above is a best-effort summary of my readings. There is a lot to say about filesystems (features, limitations, ...), and when looking for the "best" of them, the only shared answer is It depends on what you value most :

Anyway, this research also highlighted that ext4 is a pretty decent general-purpose solution, both for desktops and servers. Other filesystems will prove useful in special situations where specific features (RAID, SAN, snapshots, ...) are expected and some time is dedicated to tuning and administration.

mail

How to use a NTFS drive ?

  1. apt-get install ntfs-3g
  2. plug the drive, identify + mount it
  3. append to /etc/fstab :
    UUID=1A0B4F9B2D10391B	/mount/point	ntfs-3g	noauto,users,uid=1000,gid=1000,dmask=022,fmask=133	0	0

So far, only root can mount a NTFS filesystem. However, thanks to the uid=1000,gid=1000 parameters, the corresponding non-root user has write access to the volume.

mail

What if root gets write-access denied to the filesystem ?

Situation

While logged as root, I get Permission denied errors when trying to write on the filesystem.

Details

After investigation, I figured that (in my specific case) : Something's definitely read-only ! What ?

Solution

Some points worth checking :
  1. have a look at directory permissions. Investigate parents just in case...
  2. make sure the filesystem is not full : df -h
  3. make sure the filesystem has some inodes left : df -i
  4. make sure the filesystem is mounted rw : mount
  5. have a look at logs, just in case something went wrong at boot time :
    • dmesg
    • less /var/log/messages
  6. if you go down the directories tree since /, touching a test file at each level to find out where the Permission denied error starts, you may find that the directory /my/directory/tree is read-only. Read its attributes by cding to its parent (/my/directory), then lsattr. If it outputs ----i-------- directory/tree, you're done : the directory is immutable. Reset this with chattr -i tree
  7. ...
  8. ...
  9. you may also consider fscking the filesystem...
mail

How to mount an ext4 disk on a system which kernel only knows ext4 as a development filesystem ?

To do so, you'll have to ask the kernel to consider this filesystem as a test filesystem.

tune2fs -E test_fs /dev/sdb1 && mount -t ext4dev /dev/sdb1 /mnt/tmp

Details :

  1. specify advanced disk parameters : tune2fs -E test_fs /dev/sdb1.
    • -E is for tune2fs extended options.
    • test_fs is a flag instructing the kernel to use experimental code (such as ext4dev) to mount the disk.
  2. "debug" the filesystem to view/edit its state ( ??? not always necessary) and open it in r/w mode : debugfs -w /dev/sdb1
  3. check filesystem flags : tune2fs -l /dev/sdb1 | grep flags should output Filesystem flags: signed_directory_hash test_filesystem
  4. then mount : mount -t ext4dev /dev/sdb1 /mnt/tmp

Don't forget that on recent distributions, the ext4 filesystem is not available in "dev" anymore as it is now fully supported by the kernel.

mail

How to rename a USB mass-storage device (=change its label ) ?

Situation

When I plug a USB mass-storage device (USB key, external HDD, CF card, ...), it appears with a pre-defined name in Nautilus. How can I change this ?

Details

Actually, the name used in Nautilus to refer to a USB mass-storage device is matched to this device mounting point : if the device is mounted in /media/device_xyz, the device will be displayed as device_xyz in Nautilus.

Fine then, but where does this device_xyz come from ? As most of the time USB storage devices are formatted with FAT filesystems (FAT 16 / FAT 32), their name is just the DOS partition label. To change this, 2 solutions :

Solution

Smoothly change the partition label with mtools (as root on FAT volumes) :
  1. ignore warnings :
    echo mtools_skip_check=1 >> ~/.mtoolsrc
  2. read the label :
    mlabel -i /dev/sdg1 -s ::
  3. change the label :
    mlabel -i /dev/sdg1 ::newLabel

Alternate solution

The quick-n-dirty method is to format the partition while giving it a new name :
mkfs.vfat -n newName /dev/sdf1
mail

Which filesystem from a USB device should I mount ?

Situation

I've plugged my USB key / external HDD, automount is OFF and I don't know which filesystem to mount

Details

Which /dev/sdxx matches my USB device ?

Solution

mail

How to change a volume label ?

For EXT volumes (source 1, 2) :

e2label /dev/sdc1 newLabel

For FAT volumes :

How to rename a USB mass-storage device (=change its label ) ?
mail

What are inodes ?

File names and directory facts about inodes (source) :

  • inodes store file metadata (ownership, access mode (read, write, execute permissions), and file type) but they don't hold the file name itself
  • Unix directories are lists of association structures, each of which contains one filename and one inode number. The file system driver must search a directory looking for a particular filename and then convert the filename to the correct corresponding inode number.
  • The operating system kernel's in-memory representation of this data is called struct inode in Linux.

Get a file's inode number :

ls -i fileName

Monitor inode usage :

df -hi

Every filesystem can handle a limited number of inodes (the inodes table uses space, so allowing a huge number of files on a filesystem means a big inode table, hence less space for files.). When running out of free inodes, there are 3 solutions :

  1. remove old / unused files
  2. move files to another filesystem
  3. dump the filesystem and re-create it with a larger inodes table

Delete a file from its inode number :

find path/to/be/searched -inum inodeNumberOfFileToDelete -delete

mail

How to always mount a specific USB storage device at a specific mount point ?

For any personal reason, I'd like my USB key to always be mounted into /media/usbKey/ and my USB HDD into /media/usbHdd/. Indeed, depending on the DOS-label of the media (if any), and on the plugging order, I have no control on the mount point, which can be critical if I want to build scripts accessing content on removable media.

The UUID method :

  1. as root :
    blkid
    
    /dev/sdd1: LABEL="ConPow" UUID="05b3250f-3478-4831-8a85-c8da270839c3" TYPE="ext3"
    
    
  2. For the ConPow volume, add into /etc/fstab :
    UUID=05b3250f-3478-4831-8a85-c8da270839c3	/media/ConPow	ext3	noauto,users	0	0
  3. Don't forget to create the mount point and to manage permissions accordingly

Let non-root users mount + have write access to filesystems :

As root :
  1. specify either the user or users option in /etc/fstab
  2. manually mount the filesystem
  3. "give" the mount point to the user :
    chown bob:users /mount/point/
  4. unmount the filesystem
You should be able to mount this filesystem and write there as bob, now

The udev method :

This method is more complex and takes longer to apply than the UUID method above. It _may_ also be obsolete. It's still there only for "historical" reasons.
  1. Identify the device :
    • fdisk -l
    • or less /proc/partitions
    • or mount | grep "sd"
    • or cat /etc/mtab | grep "sd"
    which may output : /dev/sdf1 on /media/usbdisk type ext3 (rw,nosuid,nodev)
  2. Investigate the device :
    udevadm info --name=/dev/sdb1 --attribute-walk
    looking at device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0/block/sdb/sdb1':
    	KERNEL=="sdb1"
    	SUBSYSTEM=="block"
    	DRIVER==""
    	ATTR{partition}=="1"
    	ATTR{start}=="63"
    	ATTR{size}=="1250258562"
    	[...]
    
    looking at parent device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0/block/sdb':
    	KERNELS=="sdb"
    	SUBSYSTEMS=="block"
    	DRIVERS==""
    	ATTRS{range}=="16"
    	ATTRS{ext_range}=="256"
    	ATTRS{removable}=="0"
    	ATTRS{ro}=="0"
    	ATTRS{size}=="1250263728"
    	ATTRS{alignment_offset}=="0"
    	ATTRS{discard_alignment}=="0"
    	ATTRS{capability}=="50"
    	[...]
    
    looking at parent device '/devices/pci0000:00/0000:00:13.2/usb2/2-2/2-2:1.0/host12/target12:0:0/12:0:0:0':
    	KERNELS=="12:0:0:0"
    	SUBSYSTEMS=="scsi"
    	DRIVERS=="sd"
    	[...]
  3. Write the udev rule into /etc/udev/rules.d/10-local.rules :

    KERNEL=="sd?1", ATTRS{size}=="1250263728", ATTRS{capability}=="50", KERNEL=="sd?1", NAME="ConPow"

    More about udevinfo, udev rules syntax and basic udev rules
  4. Restart udev : service udev restart
  5. Create the mount point /media/ConPow
  6. Finally, to give it a fixed mount point, append to /etc/fstab :

    /dev/ConPow /media/ConPow ext3 auto,user 0 0

  7. Don't forget to unplug/replug the USB drive (or to /etc/init.d/udev restart)so that it gets re-discovered and handled by the udev rule.
  8. At this step, "normal" users still have not enough privileges to write on the device : set permissions

Additional stuff:

Instead of the UUID, it is possible to identify filesystems by their volume label. This volume label can be specified while creating the file system :

On existing filesystems, this can be done with tune2fs. Then, it is possible to mount filesystems, or refer to them in /etc/fstab with the label keyword.

mail

A FAT32-formatted USB key suddenly complains bus error and is remounted read-only.

Situation

  1. I plug my USB key and mount it normally
  2. I use it as usual for a while
  3. Suddenly, when trying to read from it, it says bus error.

What's going on ?

Details

  1. mount outputs /dev/sdd1 on /media/myUsbKey type vfat (ro, ... ,errors=remount-ro)
    So it seems some error occurred on the device and -as instructed- it's been remounted read-only.
  2. dmesg is more verbose :
    [ 5962.398581] FAT-fs (sdd1): error, fat_get_cluster: invalid cluster chain (i_pos 14085575)
    [ 5985.072996] FAT-fs (sdd1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
    [ 5995.422661] FAT-fs (sdd1): error, fat_get_cluster: invalid cluster chain (i_pos 14085575)
    [ 5995.422671] FAT-fs (sdd1): Filesystem has been set read-only

For a detailed view on how things work, read this ServerFault discussion (or a local copy)

Solution

  1. Try to fix the filesystem automatically : fsck.msdos -aw /dev/sdd1
  2. If this doesn't work, try a manual approach : fsck.vfat /dev/sdd1
    If it highlights differences on the "boot sector" :
    There are differences between boot sector and its backup.
    Differences: (offset:original/backup)
    	65:01/00
    1) Copy original to backup
    2) Copy backup to original
    3) No action
    ?
    think twice before overwriting usable data with 0's (have another look at the article I linked above !)
  3. As for me, this didn't work either, so I had to backup everything then format the USB drive and it works fine anew
mail

How to clone a CD ?

  1. Insert the source CD into the reader, but do not mount it.
  2. Copy the source CD to the hard disk as a bitstream: dd if=/dev/hdx of=/path/to/image.iso
  3. Identify the SCSI fake number ID of the CD writer :
    cdrecord -scanbus
    It should output an ID such as : x,x,x
  4. To blank a CD-RW :
    cdrecord blank=fast dev=x,x,x
  5. Then burn :
    cdrecord -v speed=[2/4/6/8/10/12/...] dev=x,x,x /path/to/image.iso
  6. The CD writer unit is /dev/scdx
mail

How to connect a USB Digital Camera (and play with SCSI emulation) ?

With modern distributions, all steps below are now handled automatically / differently and this whole procedure is now obsolete. I keep it for "historical" reasons.

  1. Ensure all the required kernel modules are loaded (usb-storage is mandatory, others are usually already loaded) : lsmod | grep 'usb'
  2. If usb-storage is not currently loaded : modprobe usb-storage
  3. Check that USB drivers (especially usb-storage) are ready : less /proc/bus/usb/drivers
  4. Connect the USB device, turn it on, then list all currently connected USB devices : less /proc/bus/usb/devices | grep "S:"
  5. Since USB devices are emulated through SCSI, list all found SCSI devices : less /proc/scsi/scsi
  6. Read complete information about the SCSI-emulated USB device : less /proc/scsi/usb-storage-0/x (x is the id of the SCSI device, starting at 0)
  7. Create a mount point : mkdir /mnt/usb
  8. SCSI devices are named sxxi, with :
    • xx : 2 letters identifying the type of device:
      • cd : CD, DVD, CD-Recorder unit
      • d[a-g] : storage device. The 1st one is da, then db, etc. The letter corresponds to the device order within all the devices of the same family (for example, the 1st HDD drive is sda even if there are other SCSI devices)
    • i : device or partition number. Keep in mind that FAT32 partitions usually start numbering at 5.
  9. mount the device : mount /dev/sda1 /mnt/usb