Linux - System files

/bin, /sbin, /usr/bin, /usr/local/bin & al.

/ system root
bin for executable programs which are needed in single user mode and to bring the system up or repair it
opt for monolithic non-distribution packages (mostly for large, poorly behaved third-party packages such as Oracle)
sbin Like /bin, holds commands needed to boot the system, but which are usually not executed by normal users
bin primary directory for executable programs (i.e. distribution-managed normal user programs)
local where programs which are local to the site typically go
bin for normal user programs not managed by the distribution package manager (locally compiled packages or programs provided as a single binary)
sbin locally installed programs for system administration
sbin program binaries for system administration which are not essential for the boot process, for mounting /usr, or for system repair

/etc/resolv.conf

By default, /etc/resolv.conf is managed dynamically by various network service daemons. This is intended for laptops and other highly mobile systems which may connect to different networks. This is why trying to change it manually doesn't work.

Several solutions exist to workaround this behavior : Configuring /etc/resolv.conf.

Test the DNS servers defined in /etc/resolv.conf :

awk '/^nameserver/ {print $2}' /etc/resolv.conf | xargs -I ipAddress nc -vzu ipAddress 53

/etc/sudoers

  • No change should be made to /etc/sudoers. Instead, sudo privileges can be configured in /etc/sudoers.d/kevin (which must be "chmod 440")
  • sudoers must belong to the sudo group, don't forget to : usermod -aG sudo kevin

Basic syntax (source : 1, 2, 3) :

This is basically :
sudoUser	SUDOHOST=(sudoRunAs)	sudoCommand	sudoOption
SUDOHOST is a string of uppercase letters, numbers and underscores starting with an uppercase letter (source).
Example 1 :

stuart ALL=(kevin) /usr/bin/vi

means that Stuart can run /usr/bin/vi on any machine as Kevin. This is equivalent to :

stuart@localhost:~$ sudo -u kevin vi anyFile

Example 2 :
To let Bob run nmap as root on myServer, add to the sudoers file :
bob		MYSERVER=(root)		/usr/bin/nmap
This way, Bob will have to enter his own password to run nmap.
We can also configure sudo so that Bob doesn't need to enter any password :
bob		MYSERVER=(root)		NOPASSWD:/usr/bin/nmap

Detailed syntax (source) :

The sudoers file has 2 types of entries :
  • aliases : key/value pairs used to set mnemonics to (possibly long / complex) values.
    There are 4 types of aliases :
    • User_Alias :
      • refer to a single user or to a list of users via user name, UID, group, GID, ...
      • these are the users who are granted extra privileges
      • there are advanced rules to use these fields (prefixes, separators, ...), quote special characters, negate one entry (with !), ... :
    • Runas_Alias : refer to users (similar grammar than User_Alias) to "run commands as"
    • Host_Alias :
      • refer to one or more machines via hostname, IP address, network number (subnet + mask), ...
      • there are advanced rules to use this, define lists, ... :
    • Cmnd_Alias :
      • one or more commands (wildcards accepted) with or without arguments
      • plenty of options here too :
  • user specifications : who's allowed to do what

Example :

sudoers file snippet :
Host_Alias	myServer = 10.2.0.188


User_Alias	APP_DEVELOPPER = stuart
Cmnd_Alias	IMPORT_DATA = /path/to/script/importData *
APP_DEVELOPPER	myServer = (root)NOPASSWD: IMPORT_DATA


User_Alias	OTHER_USER = kevin
Cmnd_Alias	DO_SOME_STUFF = /path/to/scriptThatGetsThingsDone *
Cmnd_Alias	RUN_SQL = /path/to/script.sql *
OTHER_USER	myServer = (root)NOPASSWD: RUN_SQL, (root)NOPASSWD: DO_SOME_STUFF


Runas_Alias	DATA_EXTRACTOR = stuart					see note below
Cmnd_Alias	EXTRACT_DATA = /path/to/scripts/extract.sh *
OTHER_USER	myServer = (DATA_EXTRACTOR)NOPASSWD: EXTRACT_DATA
APP_DEVELOPPER	myServer = (DATA_EXTRACTOR)NOPASSWD: EXTRACT_DATA
This example was inspired by a configuration found on a server at my company. I don't see the point of the 4th line of this block, where stuart is allowed to run the EXTRACT_DATA command... as himself (looks like an error to me).
This also explains the last line of the output below.
Result of sudo -l, run as stuart :
User stuart may run the following commands on this host:
	(root) NOPASSWD: /path/to/script/importData *
	(root) NOPASSWD: /path/to/script.sql *
	(stuart) NOPASSWD: /path/to/scripts/extract.sh *

About NOPASSWD :

https://askubuntu.com/questions/334318/sudoers-file-enable-nopasswd-for-user-all-commands
http://www.ducea.com/2006/06/18/linux-tips-password-usage-in-sudo-passwd-nopasswd/
			

A funny sudo option, insults :

Provided everything is configured adequately, after typing any sudo command you'll be prompted for your password. If you fail that step, sudo will return Sorry, try again.. But you can configure it to return something funnier, like insults : Your mind just hasn't been the same since the electro-shock, has it?.

To do so, append to /etc/sudoers (with visudo) :

Defaults	insults

As root : echo -e "Defaults\tinsults" >> /etc/sudoers

How to specify custom mount options for the root filesystem "/" ?

This is a work in progress on a topic not (yet) clear to me.

WHERE MOUNT OPTIONS COME FROM FOR THE ROOT FILESYSTEM '/' :
 - mount options are not read from /etc/fstab since this file not available yet at boot time. '/' mount options are passed via the kernel boot options (http://man7.org/linux/man-pages/man7/bootparam.7.html)

 - some sources suggest that SYSTEMD reads '/' mount options from /etc/fstab to prepare boot options parameters (??? not clear ???)

==========================================8<=========================================================
https://www.suse.com/support/kb/doc/?id=7016840

the mount option(s) have to be specified on the kernel command line, via the grub2 bootloader.

While SUSE is in the process of developing a final fix for this, a workaround would be to:

Edit the grub2 boot loader config at /etc/default/grub and add the:

	rootflags=

option with the file system flag added to the GRUB_CMDLINE_LINUX_DEFAULT="" variable, for example:

	GRUB_CMDLINE_LINUX_DEFAULT="rootflags=discard"


	==> then "update-grub" ?

The systemd fstab generator, used to mount the root filesystem, only takes into account the mount options specified in the kernel command line through the "rootflags=" variable.
If this variable is not specified, the root filesystem is mounted firstly with the default mount options and then remounted with the options specified in /etc/fstab.

==========================================8<=========================================================
BOOT ARGUMENTS

http://www.tldp.org/HOWTO/BootPrompt-HOWTO-3.html#ss3.1

==========================================8<=========================================================
How does a kernel mount the root partition?

https://unix.stackexchange.com/questions/9944/how-does-a-kernel-mount-the-root-partition#answer-18055

https://unix.stackexchange.com/questions/9944/how-does-a-kernel-mount-the-root-partition#answer-9962
	The kernel can accept command line arguments like any other program. GRUB, or most other bootloaders can accept command line arguments as user input, or store them and make various combinations of command line arguments available via a menu. The bootloader passes the command line arguments to the kernel when it loads it (I don't know the name or mechanics of this convention but it's probably similar to how an application receives command line arguments from a calling process in a running kernel).
	One of those command line options is root, where you can specify the root filesystem, i.e. root=/dev/sda1.
==========================================8<=========================================================
SYSTEMD

systemctl list-units
	-.mount				loaded active mounted	Root Mount
	boot.mount			loaded active mounted	/boot


systemctl status -- -.mount
	● -.mount - Root Mount
	   Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
	   Active: active (mounted) since Mon 2017-11-27 08:26:54 CET; 4 days ago
	    Where: /
	     What: /dev/mapper/caramba--vg-root
	     Docs: man:fstab(5)
	           man:systemd-fstab-generator(8)
	    Tasks: 0 (limit: 4915)
	   CGroup: /system.slice/-.mount

systemctl show -- -.mount
	Where=/
	What=/dev/mapper/caramba--vg-root
	Options=rw,relatime,errors=remount-ro,data=ordered
	Type=ext4
	TimeoutUSec=1min 30s
	ControlPID=0
	DirectoryMode=0755
	...


systemctl list-unit-files -- -.mount

https://www.freedesktop.org/software/systemd/man/systemd.mount.html
https://en.wikipedia.org/wiki/Linux_startup_process#systemd

==========================================8<=========================================================
FLAGS :

https://man.cx/ext4
	rootflags=data=journal.


http://man7.org/linux/man-pages/man7/bootparam.7.html
	'rootflags=...'
		This parameter sets the mount option string for the root filesystem (see also fstab(5)).
==========================================8<=========================================================
https://unix.stackexchange.com/questions/92720/does-initramfs-use-etc-fstab

THE BOOT PROCESS :
bios
MBR / whatever
boot loader (grub2)
load the kernel (initrd / initramfs)
is instructed who's '/' with the 'root=...' boot parameter
may have additional parameters (such as 'rootflags')
kernel mounts / read-only
start /sbin/init (or systemd ?)
one of the 'init' jobs reads /etc/fstab and "mount -o remount,rw /"

/etc/group

/etc/group lists the groups of users and the group members. There is one entry per line, with fields separated with a colon : :

dba:x:1000:oracle

Each line has the following format :

  1. group name
  2. password. Seldom used, and generally set as x.
  3. group ID
  4. group members : list of user names separated with commas ,.

/proc/26207/fd/3 -> socket:[145565953]

While wandering in the /proc/ tree, I found some processes having a fd symlinking a socket :
ls -l /proc/26207/fd
total 0
lrwx------ 1 root root 64 nov.	2 09:19 0 -> /dev/null
lrwx------ 1 root root 64 nov.	2 09:19 1 -> /dev/null
lrwx------ 1 root root 64 nov.	2 09:19 2 -> /dev/null
lr-x------ 1 root root 64 nov.	2 09:19 3 -> socket:[145565953]
l-wx------ 1 root root 64 nov.	2 09:19 4 -> /dev/ptmx
lrwx------ 1 root root 64 nov.	2 09:19 5 -> socket:[145565284]
What does socket:[145565953] refer to ?

The netstat approach :

netstat -laputen | grep 145565953

tcp	0	64	192.168.1.101:22	a.b.c.d:7840	ESTABLISHED	0	145565953	26207/sshd:	bob

Bob is using an SSH connection from a.b.c.d:7840 to this host.

The "let's get dirty" approach :

/proc/net/tcp (as well as /proc/net/udp and /proc/net/unix) can tell us more about sockets :
head -n1 < /proc/net/tcp; grep -a 145565953 /proc/net/tcp
sl	local_address	rem_address	st	tx_queue rx_queue	tr tm->when	retrnsmt	uid	timeout	inode
8:	6501A8C0:0016	xxxxxxxx:1EA0	01	00000110:00000000	01:0000001B	00000000	0	0	145565953 4 ffff8801172c9240 29 4 29 10 -1
The ports (source) :
The local_address and rem_address columns respectively display : 6501A8C0:0016 and xxxxxxxx:1EA0. Let's compute the ports numbers in decimal :
echo 'ibase=16; 0016; 1EA0' | bc
22
7840
The IP addresses :
6501A8C0 is the local IP address in network notation (4 hex bytes, i.e. 8 chars). To convert it to the usual dotted quad notation (123.45.67.89) :
echo [0xc0, 0xa8, 0x01, 0x65] | irb (source)
[192, 168, 1, 101]
printf "%d.%d.%d.%d\n" 0xc0 0xa8 0x01 0x65 (source)
192.168.1.101

/etc/fstab

/etc/fstab contains mount information about all the system filesystems and devices. Each line is made of 6 parameters :
Number Parameter Usage
1 device This can be specified as a device name (/dev/sdb1), or using the device ID returned by blkid (UUID=5FF1-07D3)
  • Linux : /dev/hda1
  • FreeBSD : /dev/ad0s1a
2 mount point
3 filesystem type
  • Linux : ext2, ext3, reiserfs, swap, msdos, vfat, ntfs, auto, ...
  • FreeBSD : ufs, swap, cd9660, msdosfs, ...
4 options (details)
  • ro / rw : read only / read write
  • auto / noauto : automount (mount at boot or with mount -a) / NO automount (explicit mount only)
  • user / users / nouser :
    • user : mountable by an ordinary user (i.e. non-root). The name of this user is written in /etc/mtab so that only this user can unmount the filesystem.
    • users : mountable by an ordinary user (i.e. non-root). Any ordinary user can unmount the filesystem (source).
    • nouser : mountable by root only.
  • exec / noexec : allow / forbid execution of binaries stored on this filesystem
  • sync / async : physically write data synchronously (when a write command is launched) / asynchronously (cache data and write later)
  • defaults : use default settings : rw,suid,dev,exec,auto,nouser,async
  • uid=value, gid=value : set the owner and group of the files in the filesystem (default: uid=gid=0)
  • suid / nosuid : enable / disable effect of setuid and setgid bits
  • file times (source) :
    • atime : write atime every time a file is read. This implies nodiratime : DRY
    • noatime : never write atime
    • relatime : update atime only if the previous atime was earlier than the current mtime or ctime (sources : 1, 2)
      In addition :
      • since Linux 2.6.30
      • AND when the defaults, atime or no options at all are specified
      • AND when the previous atime was more than 24 hours old
      atime is always updated.
      The purpose of relatime is to provide a mix between atime (need access times) and noatime (limit disk I/O).
5 dump Boolean allowing the filesystem to be backuped by dump. The root partition "/" is usually set to 1, and others to 0. 0 or nothing means it is not backuped.
6 pass A number telling the order in which the filesystems should be checked at reboot time by fsck :
  • 0 : don't fsck this filesystem
  • 1 : fsck this filesystem 1st
  • 2 : fsck this filesystem after
The root partition "/" should be 1.
  1. Don't forget the noauto option on removable devices, or this will block the boot process if the filesystem is missing. The Internetz suggest nobootwait to prevent this, but this option is unknown to fstab documentation (not found here, nor here, ).
  2. Only root can mount devices that are not listed in /etc/fstab, which implies non-root users can mount devices only if :
  3. The uid / gid options don't apply to Ext filesystems.

/etc/passwd

Fields are separated with colons :, and are :

The safe way to edit this file is with vipw.

/etc/shadow

Format (source) :

/etc/shadow has 9 fields separated by colons : :
  1. Username, up to 8 characters. Case-sensitive, usually all lowercase. A direct match to the username found in /etc/passwd.
  2. Encrypted password or status exception value. The encrypted password is formatted like : $id$salt$hashed, with :
    • $id : the algorithm used. On GNU/Linux, $6 is for SHA-512.
    • $salt
    • $hashed
  3. number of days since January 1, 1970 since the password was last changed
  4. number of days before password may be changed (0 indicates it may be changed at any time)
  5. number of days after which password must be changed (99999 indicates user can keep his or her password unchanged for many, many years)
  6. number of days to warn user of an expiring password (7 for a full week)
  7. number of days after password expires that account is disabled
  8. number of days since January 1, 1970 that an account expires
  9. reserved for future use

The safe way to edit this file is with vipw.

Exclamation marks and asterisks in the password field

The documentation says :
  • If the password field contains some string that is not a valid result of crypt (for instance ! or *), the user will not be able to use a unix password to log in (but the user may log in the system by other means).
  • This field may be empty (::), in which case no passwords are required to authenticate as the specified login name. However, some applications which read /etc/shadow may decide not to permit any access at all if the password field is empty.
  • A password field which starts with a exclamation mark (!) means that the password is locked. The remaining characters on the line represent the password field before the password was locked.
In order to clarify the contents of the password field described above, and what can / can not be done, let's experiment and see what happens :

Commands below must be run as root .

Let's create a user account with default settings :
useradd -m bob; grep bob /etc/{passwd,shadow}
/etc/passwd:bob:x:1001:1001::/home/bob:/bin/sh
/etc/shadow:bob:!:16679:0:99999:7:::
The x in /etc/passwd indicates the password is stored in /etc/shadow.
I can not log in as Bob. su bob prompts for a password and always fails.
Let's give Bob an empty password :
passwd bob fails if we just press . Let's do something BAD and edit /etc/shadow : vipw -s and remove the !.
Then grep bob /etc/{passwd,shadow}
/etc/passwd:bob:x:1001:1001::/home/bob:/bin/sh
/etc/shadow:bob::16679:0:99999:7:::
Now the ! is gone, I still can't su bob, but I can log in as Bob without being prompted for a password.
Return of the ! :
Editing again /etc/shadow to have :
bob:!:16679:0:99999:7:::
bob:!!:16679:0:99999:7:::
bob:*:16679:0:99999:7:::
bob:foo:16679:0:99999:7:::
Gives the same result : can not log in, can not su bob.
Let's give a password to Bob, then lock its account :
passwd bob bob bob, then grep bob /etc/{passwd,shadow}
/etc/passwd:bob:x:1001:1001::/home/bob:/bin/sh
/etc/shadow:bob:$6$b3AMx8c1$6DF/Ux9qhlOQSPdC0WNycWK3DYQ7tfsBgWLG74qtpW/ZO39fInB6jnTUF0alL8oH0Z7bF62BiOJ8wkNi1UEqQ/:16679:0:99999:7:::
Finally : a password !
Let's lock it : usermod -L bob; grep bob /etc/{passwd,shadow}
/etc/passwd:bob:x:1001:1001::/home/bob:/bin/sh
/etc/shadow:bob:!$6$b3AMx8c1$6DF/Ux9qhlOQSPdC0WNycWK3DYQ7tfsBgWLG74qtpW/ZO39fInB6jnTUF0alL8oH0Z7bF62BiOJ8wkNi1UEqQ/:16679:0:99999:7:::
The password now has a leading ! showing it is locked.
As a summary :
  • :: (empty password field) : no password required to authenticate
  • ! or !! or * or any string : can not log in with password, but other means may work
  • !any string : the password is locked. Can not log in with password, but other means may work

Don't forget to delete the insecure account Bob we've created to experiment :

userdel -r bob; grep bob /etc/{passwd,shadow}
(should return nothing)

GNU/Linux file tree

/ system root
dev devices and pseudo-devices
random PRNG based on the system's hardware-derived entropy pool. od -Ax -t x1 /dev/random shows the entropy pool drying up. Read from it with od -l -N 4 < /dev/random | head -1
shm RAMdisk
urandom PRNG that is not cryptographically secure
zero special file providing as many null characters (ASCII NUL, 0x00) as are read from it
etc host-specific system configuration
apt (Debianoids only!) config for APT
sources.list lists the sources from which packages can be obtained
apt.conf.d
60recommends configuration values for recommends and suggests parameters (and others...)
bashrc or /etc/bash.bashrc : system-wide Bash settings (global ~/.bashrc)
default
environment system-wide definition of environment variables (details)
fstab mount information about filesystems and devices
group information about local system groups and group members
localtime refers to the current timezone file (symlink on Red Hatoids / copy on Debianoids. details)
mtab Lists all currently mounted filesystems along with their initialization options (details).
modules list of kernel modules to load at boot. To add moduleName : echo moduleName >> /etc/modules
modules-load.d
modules.conf Symlink to /etc/modules on Debian Strech/Sid
modprobe.d
whatever.conf List of modules and settings to be loaded by udev + modprobe during system boot (source). blacklist may be specified to forbid loading a specific module (source)
network
interfaces network interfaces configuration
passwd users accounts information (login, home directory, shell, UID, ...)
profile system-wide shell settings (not only Bash). Sets some environment variables.
profile.d directory for overrides to /etc/profile.
resolv.conf domain settings + name servers to query for DNS resolution
security
limits.conf Can be used to set some quotas on resources usage.
services service names and ports numbers assigned by the IANA (see also)
shadow password and account expiration information for users
ssh
ssh_config system-wide SSH client settings
ssh_known_hosts system-wide list of known SSH host keys
sysconfig
clock Current timezone settings (Red Hatoids)
timezone Current timezone settings (Debianoids)
udev
rules.d
70-persistent-net.rules match MAC addresses to ethx interfaces
home users' personal directories
bob Bob's personal directory (details on privacy of this directory)
.bash_aliases Bash command aliases (see alias)
.bash_history history of all commands typed in Bash shell as Bob
.bashrc Bob's Bash settings
.screenrc Bob's Screen settings
.ssh directory for Bob's SSH settings
config Bob's personal settings for the SSH client
known_hosts list of hosts Bob already connected to via SSH
.xinitrc sourced when running startx and other programs which call xinit (source : 1, 2). To launch a script upon X session start, consider ~/.xsession instead (details).
lib essential shared libraries and kernel modules
firmware firmwares used by some hardware components such as wireless controllers
modules
$(uname -r) kernel modules
lost+found corrupted files recovered by e2fsck. Details about these files : for i in $(ls /mnt/mountPoint/lost+found/); do file $i; done
proc pseudo filesystem for stuff created when the machine is up. Information found here is read by ps, top, free, .... (see also)
loadavg Stats on system loads and processes. Also returned by uptime
meminfo information on memory usage
mounts Records the same information than /etc/mtab except that it is more up-to-date. This is especially valuable when / turns read-only and /etc/mtab can not be updated. (details)
PID directory containing full details about the process PID
cmdline command line of the process PID, including arguments
cwd Link to the current working directory
environ Values of environment variables
exe symlink to the process binary (if any)
maps Memory maps
mem Memory held by the process PID
root Link to the root directory of the process PID
stat Process status
statm Process memory status information
status Process status in human readable form
scsi
usb-storage
n metadata about USB storage number n
sys
kernel
pid_max The maximum number a PID can have. Starts over (from 2?) after reaching this limit.
run (read details in Debian Wheezy documentation)
shm RAMdisk, symlink to /dev/shm (found on Debianoids, does not exist on Red Hatoids)
sys pseudo-filesystem used for exporting kernel objects (details)
class
scsi_host
hostn symlink to /sys/devices/pci0000:00/0000:00:07.1/ata2/hostn/scsi_host/hostn
fs
cgroup conventionally used as a mount point for a tmpfs filesystem containing mount points for cgroups filesystems
tmp
usr usually mounted from a separate partition. It should hold only shareable, read-only data, so that it can be mounted by various machines running Linux
share
zoneinfo Timezone definition files (more)
var
cache
apt
archives Can be cleaned with apt autoclean (even better than apt-get clean)
lib
dhcp
dhclient.wlan0.leases DHCP leases for interface wlan0
logrotate
status date of latest log rotation by logrotate
log
auth.log authentication log
kern.log kernel log
syslog system log
spool
mail users mailboxes
cron
crontabs all users' crontabs. found on Debianoids but not on Red Hatoids.
www-data www-data's crontab. Not to be edited as-is.

For even more details : man hier

/proc/meminfo

/proc/meminfo lists MANY fields of information (which are read by free) :
Dirty
The total amount of memory waiting to be written back to the disk. Watch it with :
watch -n 1 -d 'grep Dirty /proc/meminfo'

Format of /proc/loadavg

cat /proc/loadavg outputs : 0.16 0.20 0.19 1/441 20304, which means :
  1. 0.16 : load1
  2. 0.20 : load5
  3. 0.19 : load15
  4. 1 : number of runnable kernel entities (processes or threads). This is ≤ number of CPU
  5. 441 : total number of kernel scheduling entities
  6. 20304 : newest PID

Monitor server loads :

watch -n 1 -d 'for serverId in {8..12}; do serverName=myServer$serverId; echo $serverName; ssh -i /home/bob/.ssh/id_rsa bob@$serverName cat /proc/loadavg; done'