Linux - Looking under the hood

• share a single inode with their target.
• can not refer to :
  • a missing file
  • a directory
  The target must be an existing regular file.
• can only refer to a file found on the same filesystem (because inodes are not unique across filesystems).
• How to spot a hard link ?

• can refer to directories, missing files, another symbolic link, and even cross file system boundaries.
• have their own inode.

• On Linux, the permissions of a symbolic link link are not used in any operations. The permissions are always 0777 and can't be changed.
• You can change the owner of a symbolic link with chown -h.
• When most operations (opening, reading, writing, ...) are passed to the symbolic link file, the kernel automatically "dereferences" the link and operates on the target of the link. But some operations (e.g., removing) work on the link file itself, rather than on its target.
• Most commands dereference (i.e. follow) symbolic links, with the notable exception of :
  • mv
  • rm
  • file
  • ls
  • and MANY others when using their recursive (-R) flag
• It is not possible to create a hard link for a directory :

  cd /tmp; touch myFile; ln myFile linkToMyFile; ls -l *yFile

  -rw------- 2 bob developers 0 Jan 22 16:34 linkToMyFile
  -rw------- 2 bob developers 0 Jan 22 16:34 myFile

  mkdir myDir; ln myDir linkToMyDir

  ln: myDir: hard link not allowed for directory

  ln -s myDir linkToMyDir; ls -ld *yDir

  lrwxrwxrwx 1 bob developers    5 Jan 22 16:36 linkToMyDir -> myDir
  drwx------ 2 bob developers 1024 Jan 22 16:35 myDir

  rm linkToMyDir linkToMyFile myFile; rmdir myDir

• outside a sticky world-writable directory. Sticky world-writable directories, like /tmp, have drwxrwxrwt permissions
• OR when the UID of the symlink and follower match
• OR when the directory owner matches the symlink's owner

echo 1 > /proc/sys/fs/protected_symlinks

• the software to be available in your distro's list of packages
• and your workstation to have a deb-src entry in /etc/apt/sources.list

apt-get source sshpass
cd sshpass-1.06/
./configure
make
ldd sshpass
linux-vdso.so.1 (0x00007ffe45de8000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3830e37000)
/lib64/ld-linux-x86-64.so.2 (0x00007f38313da000)

make clean
make SHARED=0 CC='gcc -static'		not sure about the SHARED=0 part 
ldd sshpass
not a dynamic executable		what we were trying to do 

What happens after moving our new binary to 32bitMachine ?

1. (source)
   ./configure --host=i686-linux-gnu --build=i686-linux-gnu CC="gcc -m32" CXX="g++ -m32" CFLAGS="-O2 -march=i686" CXXFLAGS="-O2 -march=i686"

   checking for a BSD-compatible install... /usr/bin/install -c
   checking whether build environment is sane... yes
   checking for a thread-safe mkdir -p... /bin/mkdir -p
   checking for gawk... gawk
   checking whether make sets $(MAKE)... yes
   checking whether make supports nested variables... yes
   checking for style of include used by make... GNU
   checking for i686-linux-gnu-gcc... gcc -m32
   checking whether the C compiler works... no
   configure: error: in `.../path/to/sshpass-1.06':
   configure: error: C compiler cannot create executables
   See `config.log' for more details

2. (source)
   make clean && ./configure && make CC='gcc -static -m32'

   make all-am
   make[1]: Entering directory '.../path/to/sshpass-1.06'
   gcc -static -m32 -DHAVE_CONFIG_H -I.     -g -O2 -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c
   main.c:25:23: fatal error: sys/types.h: No such file or directory
    #include <sys/types.h>
                          ^
   compilation terminated.
   Makefile:415: recipe for target 'main.o' failed
   make[1]: *** [main.o] Error 1
   make[1]: Leaving directory '.../path/to/sshpass-1.06'
   Makefile:307: recipe for target 'all' failed
   make: *** [all] Error 2

3. (source)
   apt-cache search libc | grep -E 'dev.*86'

   libc6-dev-i386 - GNU C Library: 32-bit development libraries for AMD64

   apt install libc6-dev-i386
   make clean && ./configure && make CC='gcc -static -m32'

   make all-am
   make[1]: Entering directory '.../path/to/sshpass-1.06'
   gcc -static -m32 -DHAVE_CONFIG_H -I.     -g -O2 -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c
   In file included from /usr/include/sys/ioctl.h:26:0,
                    from main.c:28:
   /usr/include/bits/ioctls.h:23:24: fatal error: asm/ioctls.h: No such file or directory
    #include <asm/ioctls.h>
                           ^
   compilation terminated.
   Makefile:415: recipe for target 'main.o' failed
   make[1]: *** [main.o] Error 1
   make[1]: Leaving directory '.../path/to/sshpass-1.06'
   Makefile:307: recipe for target 'all' failed
   make: *** [all] Error 2

4. (source)
   apt install gcc-multilib
   make clean && ./configure && make CC='gcc -static -m32'
   no error
   file sshpass

   sshpass: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=886a41e4c4e7631776b20a44f2262b85ad3fa240, not stripped

kevin@32bitMachine ~$ file sshpass

sshpass: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), for GNU/Linux 2.6.32, statically linked, not stripped

kevin@32bitMachine ~$ (a command including sshpass)

FATAL: kernel too old
Segmentation fault

Indeed :

kevin@32bitMachine ~$ uname -a

Linux 32bitMachine 2.6.9-89.0.3.ELsmp #1 SMP Sat Jun 13 07:05:54 EDT 2009 i686 i686 i386 GNU/Linux

I compiled for a 2.6.32 kernel, while I have a 2.6.9.

Back to my 64bit workstation, I tried :

make clean && ./configure --enable-kernel=2.6.9 && make CC='gcc -static -m32'

... but it's not that simple

I've not been able to build a binary as I wanted to. However, I've learnt some extra things in the process, which is GOOD .

Let me conclude with an inspiring quote by Bruce Lee :

Don't fear failure. Not failure, but low aim, is the crime. In great attempts it is glorious even to fail.

• Priority is the actual priority value of a process at any given moment inside of the kernel.
• Niceness is just a hint for the kernel what the priority the process should have.

• Priority is relevant for the kernel task scheduler.
• In most cases, priority can be computed by : PR = 20 + NI
• The kernel may change the priority value by itself (but not the niceness value). For instance :
  • reduce the priority of a process if it consumes too much CPU
  • increase the priority of a process if that process had no chance to run for a long time because of other higher priority processes
• Priority is actually a dynamic value :
  • all of this is handled by scheduling policies
  • when using the default scheduling policy (SCHED_OTHER), process priority ranges from 0 to 39 (PR = 20 + NI with -20 ≤ NI ≤ 19)
  • some processes may use real time scheduling policies (SCHED_RR, SCHED_FIFO) and have a priority value less than 0
  • you may alter the priority value with :
    • sudo chrt -r -p 1 $(pidof vlc)
    • sudo chrt -r -p 99 $(pidof vlc)
    • sudo chrt -r -p 99 $(pidof pulseaudio)
    • • while playing with the commands above, top displays a priority value which is -1 * (-p value +1)
      • when using 99, top displays rt because its column is only 2 characters wide, and -100 wouldn't fit

Set a new value to the EDITOR environment variable :

By changing the EDITOR value for that single command only :

1. Append to ~/.profile :

   export EDITOR=vi|emacs|nano|

2. Reload changes : source ~/.profile
3. Check it with : crontab -e

1. As root, run : update-alternatives --config editor
2. Follow the on-screen instructions
3. Check it with : crontab -e

• "deleted" files are not actually deleted : instead, the storage space they use is marked as available. So information is still there until it's overwritten; the sooner you try to recover data, the higher your chances of success
• grep can not only search in regular files, but also in a whole disk

Last-minute tips :

1. install : apt install scalpel
2. edit /etc/scalpel/scalpel.conf (append rules such as these for OpenOffice files)
3. edit the recovered file size (10MB on this example) : upon finding the start tag, Scalpel will recover 10MB as if they were part of the recovered file.
4. umount the volume to analyze
5. mkdir -p /directory/for/recovered/files
6. scalpel /dev/sdXY -b -o /directory/for/recovered/files
7. wait...
8. once done, hope you'll find anything valuable in /directory/for/recovered/files

Flags (need more details):
-b : recover a file even if it's missing a footer
-o : output directory (?)

You can detect file encoding with : file -i fileName.

• If this returns the file MIME type instead of its encoding, you may try creating a new file next to fileName : it's very likely to have the same encoding.
• If fileName contains no special character (i.e. ASCII characters only), its charset will be a basic *-ascii. To have it use an "advanced" charset, it must contain some non-ASCII characters : éêèàâî....

From the shell :

• iconv --from-code=ISO-8859-1 --to-code=UTF-8 iso_file.txt > utf8_file.txt

  The source and destination must be distinct files.

  fileToTranscode='itr_install_db_050000.ksh'; cp "$fileToTranscode"{,_BITI}; iconv --from-code=ISO-8859-1 --to-code=UTF-8 "${fileToTranscode}_BITI" > "$fileToTranscode"

• recode Latin1..UTF8 file.txt (recode -l to list available encodings)

With Emacs (source : 1, 2) :

1. Open the file to transcode
2. Notice some letters in the lower left corner identifying the charset :

   Letters	charset	Comment
   UU-	us-ascii
   UU0	iso-8859-15	file -i reports iso-8859-15 as iso-8859-1
   UU1	iso-8859-1
   UUU	utf-8

3. • CTRL-x f
   • OR : CTRL-x CTRL-m f
4. At the prompt, type the new coding system : iso-8859-1 / utf-8 / ...
5. Save the transcoded file

With Vim (source) :

1. Open the file to transcode
2. :write ++enc=utf-8

With Notepad++ :

Subtleties about traditional Unix permissions (sources : 1, 2)

the Setuid bit

used only for binaries or scripts, this permission indicates that the file is to be executed with the permissions of the owner of the file, and not with the permissions of the user executing the file (which is the case without setuid). This is indicated by the character s in the place of the x in the owner category. If the owner of the file does not have execute permissions, a capital S reflects this fact.

the Setgid bit

used primarily for binaries, this permission indicates that the file is executed with the permissions of the group owning the file and not with the permissions of the group of the user executing the file (which is the case without setgid).
If applied to a directory, all files created within the directory are owned by the group owning the directory, and not by the group of the user creating the file. The setgid permission is indicated by the character s in place of the x in the group category. If the group owning the file or directory does not have execute permissions, a capital S reflects this fact.

the Sticky bit

used primarily on directories, this bit dictates that a file created in the directory can be removed only by the user who created the file. It is indicated by the character t in place of the x in the everyone category. If the everyone category does not have execute permissions, the T is capitalized to reflect this fact.
Under Red Hat Enterprise Linux, the sticky bit is set by default on the /tmp/ directory for exactly this reason.

An environment variable is basically just a shell variable that is available system-wide (i.e.: to sub-shells). A shell variable can be changed into an environment variable with export.

• $ + TAB : names only. By convention, environment variables names are all UPPERCASE.
• export -p
• printenv
• env or set : names and values (need details of the difference, if any)

• env | grep VARNAME
• printenv VARNAME

for several / all commands

for a single command

ignore temporarily source :

Depending on the shell type, i.e. (non-)?(login|interactive) :

export http_proxy=http://bob:password@host:port/
export https_proxy=$http_proxy

in /etc/environment (source) :

http_proxy=http://bob:password@host:port/
https_proxy=https://bob:password@host:port/

• Bash : export VARNAME=value This is NOT permanent !
• C-Shell : setenv VARNAME value

1. Create a file having the size of your future filesystem : dd bs=1M count=100 if=/dev/zero of=fsInAFile
2. Format the file (using any filesystem you like) : mkfs.ext4 fsInAFile
   mkfs will ask : fsInAFile is not a block special device. Proceed anyway? (y,n). Answer y.
3. Mount the file with a loop device : mount -o loop fsInAFile /path/to/mount/point
   When using the option -o loop the first available loop device is taken. It is possible to select the loop device to use by giving the option -o loop=/dev/loop0.
4. You can check that the device is perfectly mounted with mount. It should display a line such as :

   fsInAFile on /mount/point type ext4 (rw,loop=/dev/loop0)

5. It is now possible to create files and directories within /mount/point, then unmount the device and remount it back as any removable device.

1. uncompress it:
   • for gzip'ed archives : gunzip archiveName.tar.gz
   • for bzip2'ed archives : bunzip2 archiveName.tar.bz2
   This will result in a archiveName.tar file.
2. unpack it: tar -xf archiveName.tar
   It _may_ create a directory called archiveName depending on how the archive was created.
3. you can also uncompress + unpack in a single operation : tar -zxf archiveName.tar.gz or tar -xjf archiveName.tar.bz2
4. inside the corresponding directory, have a look at the README file.
5. ./configure
6. make
7. as root : make install