Software - Notes on my favourite programs

tmux

http://blog.hawkhost.com/2010/06/28/tmux-the-terminal-multiplexer/
Configuration file : ~/.tmux.conf


Default key binding :

	Ctrl-b c	Create new window
	Ctrl-b d	Detach current client
	Ctrl-b l	Move to previously selected window
	Ctrl-b n	Move to the next window
	Ctrl-b p	Move to the previous window
	Ctrl-b &	Kill the current window
	Ctrl-b ,	Rename the current window
	Ctrl-b %	Split the current window into two panes
	Ctrl-b q	Show pane numbers (used to switch between panes)
	Ctrl-b o	Switch to the next pane
	Ctrl-b ?	List all keybindings

How to remove line breaks inside cells in LibreOffice Calc ?

  1. Select the cells containing the line breaks you want to remove (or don't select anything if you want to alter the whole datasheet)
  2. CTRL-h (or Edit | Find & Replace)
  3. In the Find field, enter \n
  4. In the Replace field, enter whatever you would like to replace line breaks with
  5. Then click Other options
  6. Tick Current selection only if you wish to alter selected cells only
  7. Tick Regular expressions
  8. Then Replace (or Replace All if you're brave enough)

X2go


Server install on Debian (source, procedure for Stretch) :

Preliminary notes :
  • All commands below must be run as root, use sudo accordingly.
  • The installation process itself can be run via SSH
for Jessie
  1. Add the X2go Debian repository :
    apt-key adv --recv-keys --keyserver keys.gnupg.net E1F958385BFE2B6E
  2. Create a dedicated sources.list-like :
    cat << EOF > /etc/apt/sources.list.d/x2go.list
    # X2Go Repository (release builds)
    deb http://packages.x2go.org/debian jessie main
    # X2Go Repository (sources of release builds)
    deb-src http://packages.x2go.org/debian jessie main
    EOF
  3. apt-get update
  4. apt-get install x2go-keyring && apt-get update
  5. Install the server package : apt-get install x2goserver
    This also installs x2goserver-xsession.
for Stretch
  1. add the repository key
    Should you get a gpg: keyserver receive failed: No dirmngr error, it can be fixed with (source) :
    apt install dirmngr
  2. declare the repository :
    cat << EOF > /etc/apt/sources.list.d/x2go.list
    # X2Go Repository (release builds)
    deb http://packages.x2go.org/debian stretch extras main
    # X2Go Repository (sources of release builds)
    deb-src http://packages.x2go.org/debian stretch extras main
    EOF
  3. apt update && apt install x2goserver x2goserver-xsession

Client install

On Windows 7 :
Just follow these instructions.
On my side, the "install without admin rights and without a start menu entry" hack didn't work
On Stretch (source) :
apt install x2goclient
Bug : the connection closes instantly : Status: finished
Debug server-side :
  1. ls -lrt /var/log/*log
    ...
    -rw-r----- 1 root	adm	97K	Dec 22 11:43 /var/log/user.log
    -rw-r----- 1 root	adm	1.3M	Dec 22 11:43 /var/log/syslog
    -rw-r----- 1 root	adm	67K	Dec 22 11:44 /var/log/auth.log
  2. tail -f /var/log/auth.log + retry opening a client session
    ...
    Dec 22 11:51:38 myX2goServer sshd[27225]: Accepted publickey for kevin from my.public.IP.address port 16531 ssh2: RSA SHA256:0afdoG/gObils6+n8zzo5mycmiqAaOwT0qSHJgOWTZH
    Dec 22 11:51:42 myX2goServer sshd[27286]: refused local port forward: originator localhost port 56810, target localhost port 44406
    Dec 22 11:51:42 myX2goServer sshd[27286]: channel_by_id: 0: bad id: channel free
    Dec 22 11:51:42 myX2goServer sshd[27286]: Disconnecting: Received ieof for nonexistent channel 0.
  3. Let's try :
    iptables -P FORWARD ACCEPT
    No change
  4. Let's try something else : in /etc/ssh/sshd_config, change :
    AllowTcpForwarding no
    into
    AllowTcpForwarding yes
    then
    systemctl reload sshd
    and try again opening a client session :

Usage :

The idea is to connect from anywhere I'm forced to use Windows 7 to any remote server (via TCP 443). I've not been able to make the X2go client go through the corporate web proxy, so to workaround this, I've created a local SSH tunnel between my workstation and the distant TCP 443.

You can use SSH keys to authenticate to the remote host (guess you've already been there via PuTTY ), but don't forget to use private keys at the OpenSSH format, not .ppk files (PuTTY private key format).

LFTP

LFTP is a sophisticated ftp/http client, and a file transfer program supporting a number of network protocols (including SSL).
It is free software, distributed under the GNU GPL license (details).

The mirror functionality (source) :

lftp "ftpLogin:ftpPassword@ftpServer" -e "mirror --verbose=2 --continue --delete --log=/path/to/logFile --parallel=10 --allow-chown --only-newer --exclude=excludedContent /source/directory/on/ftp/server /destination/directory/on/local/server; quit"

More about --exclude.

Keep in mind that FTP admins frequently chroot jail users when specifying the source directory as well as the excluded contents : all paths are relative to the user's home directory !

LFTP and symlinks (source) :

Very basically speaking (this is QnD, needs polish), symlinks are supported, BUT absolute paths found in symlinks may be broken once a chroot has been applied. Keep in mind that a user's home directory becomes / afterwards and symlinks must handle this.

How to recover the Weblogic web console admin password ?

  1. View the boot.properties file :
    cat /appli/projects/myApplicationName/weblogic_10.0MP1/wl-domains/DOMAINE-WLS-MYAPP/boot.properties
    #Mon Mar 21 17:00:17 MET 2011
    password={3DES}sRO9OiEXAWNw2AE7UfkAxw\=\=
    username={3DES}kpclVYTJKNeNXN1ZPNzfNQ\=\=
    And save the encrypted password for later.
  2. Move to the security child directory of the directory containing the boot.properties file :
    cd /appli/projects/myApplicationName/weblogic_10.0MP1/wl-domains/DOMAINE-WLS-MYAPP/security/
  3. Make sure there is a SerializedSystemIni.dat file in this directory (this file will be read by the script below) :
    ls -l SerializedSystemIni.dat
  4. In this directory, create a Python script :
    cat << EOF > script.py
    from weblogic.security.internal import *
    from weblogic.security.internal.encryption import *
    
    # Remind user about how to use
    raw_input("Please ensure SerializedSystemIni.dat is in the current directory now, and press ENTER to continue.")
    
    # Get encryption service
    encryptionService = SerializedSystemIni.getEncryptionService(".")
    clearOrEncryptService = ClearOrEncryptedService(encryptionService)
    
    # Get user to enter password
    pwd = raw_input("Enter encrypted password (Eg. {3DES}Y1fA34S...): ")
    
    # Remove unnecessary escape characters
    preppwd = pwd.replace("\\\\", "")
    
    # Decrypt the password
    print "Recovered password is: " + clearOrEncryptService.decrypt(preppwd)
    EOF

    There are extra \ in the line : preppwd = pwd.replace("\\\\", "") to escape the regular \ when writing the file with cat and heredocs.
    The real line will be : preppwd = pwd.replace("\\", "")

  5. Then run :
    • as webadm, the user owning the Weblogic processes : ${WL_HOME}/common/bin/wlst.sh script.py
    • as root : /logiciels/bea/weblogic_10.0MP1/wlserver_10.0/common/bin/wlst.sh script.py
      (or /export/logiciels/bea/weblogic_10.0MP1/wlserver_10.0/common/bin/wlst.sh ???)
    ... and enter the encrypted password when prompted.
  6. Enjoy !

New Relic

What is New Relic ?

New Relic is an APM software dedicated to web apps (Ruby, PHP, JAVA, .Net, Python, Node.js). I has quite a wide product offer, among which :

PHP APM : Enable the agent :

  1. Install the agent (1, 2)
  2. Enable the New Relic PHP module so that you have : /etc/php5/cgi/conf.d/20-newrelic.ini -> ../mods-available/newrelic.ini :
    cd /etc/php5/cgi/conf.d/; ln -s ../mods-available/newrelic.ini 20-newrelic.ini
  3. Make sure newrelic.appname is set accordingly in /etc/php5/mods-available/newrelic.ini
  4. Restart the webserver. Should be fine !

PHP APM : Configuring the application name (source) :

The application name can be configured system-wide with the newrelic.appname directive in :
  • php.ini
  • /etc/php5/conf.d/newrelic.ini
What if more than one application is installed on the server ?
  • It is possible to specify the appname in the code (source) :
    if (extension_loaded('newrelic')) {
    	newrelic_set_appname (name);
    	}
  • you can define PHP values in Apache vhost (source, Specify a custom php.ini for a website) :
    <VirtualHost 192.168.42.43>
    	ServerName www.myvhost1.com
    	DocumentRoot "/path/to/vhost1/"
    	...
    	<IfModule PHP_MODULE>
    		php_value newrelic.appname "Application 1"
    	</IfModule>
    </VirtualHost>
    
    <VirtualHost 192.168.123.45>
    	ServerName www.myvhost2.com
    	DocumentRoot "/path/to/vhost2/"
    	...
    	<IfModule PHP_MODULE>
    		php_value newrelic.appname "Application 2"
    	</IfModule>
    </VirtualHost>
  • With Lighttpd, it is possible to set per-directory PHP config values in .htaccess files (source) with the htscanner extension.

logrotate

Flags :

Flag Usage
-d debug mode : (implies -v : verbose mode) no changes will be made to the logs or to the state file
-f
force log rotation :
/usr/sbin/logrotate -f /etc/logrotate.d/serviceName; echo $?
If things don't work as expected (and since logrotate generates no logs), try this again with -d to view error messages :
/usr/sbin/logrotate -df /etc/logrotate.d/serviceName 2>&1 | grep error
with -d, log rotation is simulated.

Configuration directives :

Directive Usage
compresscmd /bin/bzip2 Compress logs with /bin/bzip2 instead of the default gzip.
Should be used with compressext, otherwise compressed files get a .gz extension.
compressext .bz2 Give compressed files a .bz2 extension
dateext When rotating logs, instead of appending numbers to successive rotations (from 1 (newest) to n (oldest)), append the log rotation date (default format : YYYYMMDD)
While rotated logs are re-numbered at each rotation, logs with dates are not renamed.

Usage :

To rotate logs (or any kind of file, such as .tar backups), you just have to :
  1. Create a new /etc/logrotate.d/serviceName configuration file
  2. Then edit it like :
    /path/to/serviceName/*log {
    	# Rotate logs on a ... basis
    	daily
    
    	# Keep n old log files (i.e. n days back with daily rotation)
    	rotate 10
    
    	# compress rotated logs
    	compress
    
    	# Immediately after rotation, create a new empty log file with :
    	# - same name than rotated file
    	# - [mode] [owner] [group] as specified below
    	create 644 userName groupName
    
    	# Don't issue an error if the log file is missing
    	missingok
    
    	# Store rotated logs into :
    	olddir /path/to/serviceName/rotatedLogs
    	}
    • you can list several patterns sharing the same log rotation specification :
      /var/log/tomcat8/catalina.out /var/log/tomcat8/*log {
      	...
      	}
    • patterns like /var/log/tomcat8/*/*log are also supported
  3. Save and exit. Nothing to restart / reload

logrotate itself is fired by cron :

find /etc/cron* -name logrotate
/etc/cron.d/logrotate
/etc/cron.daily/logrotate

Rotation intervals / conditions (source) :

daily
Each rotated file is listed in the state file (/var/lib/logrotate.status) with the date of its latest rotation. If current date > date in state file, the file is rotated (source).
weekly
logs are rotated if the current week day is lower than the week day of the last rotation (i.e. Monday is less than Friday) or if the last rotation occurred more than a week before the present
monthly
logs are rotated every month on the first day of the month that logrotate runs, which is often the first day of the month
yearly
logs are rotated when the current year differs from the date of the last rotation
size
logs are rotated based on their size rather than on a periodic schedule : size n[kMG]

Latest round (source) :

The date of the latest rotation can be found with :
grep serviceName /var/lib/logrotate/status
Looks like the 1st rotation must wait at least 24 hours before being processed.

How to alter PDF files ?

Edit the file contents : change text, hide a zone of the page, insert an element, ...

Even though not everything can be done (depends on how the PDF was made), Inkscape can help.

Extract pages of a document

Very quick solution
Open the PDF file with any viewer and print the selected page(s) into a file.
With pdftk
pdftk can do MUCH MORE than this to PDF documents (join, rotate pages, ...). For more : man pdftk
A basic example
pdftk A=input.pdf cat A2 output output.pdf
  • A serves as a handle on an input file
  • A2 means "page 2 of file with handle A"
A not-so-basic example
Let's imagine you received a scanned document of more than 100 pages having randomly-placed blank pages. You'd like to generate a PDF without those blank pages, which means you want to keep pages (pages listed below are included) :
  • odd pages between page 1 and until page 51
  • then all pages from 53 to 77
  • then all pages from 79 to 101
  • then the page 103
The command would then be :
pdftk A='documentWithBlankPages.pdf' cat 1-51odd 53-77 79-101 103 output cleanDocument.pdf
Don't forget the output keyword

Concatenate several files into a single PDF document

The Ghostscript method (source)

gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=concatenated.pdf part1.pdf part2.pdf ... partn.pdf

Or as an alias :

pdfcat () {
	gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=concatenated.pdf "$@"
	}

With PDFtk (source)
pdftk part1.pdf part2.pdf ... partn.pdf cat output concatenated.pdf
Other

There is also pdfjoin, but I've not tried it yet.

Rotate PDF pages (source)

pdftk myDocument.pdf cat 1-endE output out.pdf

SVN

For all SVN commands requiring login and password :
svn --username userName --password 'password' svnCommand
Download sources from the SVN repository :
svn checkout http://SVN_repository_url/project/tagName[@revision] destinationDirectory
Register new files into the SVN repository :
svn add newFile1 newFile2 ...
Unregister files from the repository (details) :
  • schedule files for deletion from repository upon next commit + remove files from working copy : svn del /path/to/oldFile1 /path/to/oldFile2 ...
  • delete files from the repository (immediate commit) : svn del protocol://url/to/oldFile1 ...
Apply local changes to the repository :
  • save ALL changes : svn commit -m "description of changes"
  • OR save listed files only : svn commit file1 file2 ... -m "description of changes"
Update working copy after a commit by one of the project contributors :
svn update
Get information on working copy :
svn stat (or svn status)
Parameters :
Parameter Usage Example
--depth empty|files|immediates|infinity Investigate not only the current directory svn stat --depth infinity
Output :
Output Meaning
? file file is unknown to SVN
D file file has been marked for deletion with svn del
M file file has local modifications that are not yet saved into the repository
Information on local file :
svn info localFile
Discard local changes : roll back to the previous repository version :
svn revert path/to/file
Roll back to any previous repository version (source : 1, 2, SVN book) :
  1. Testing : svn merge --dry-run -r currentRevision:revisionToRevertTo path/to/file
  2. Same as above without --dry-run
  3. Verify that the change is removed : svn diff
  4. svn commit -m "Reverted to revision xxx."
Show differences between the working copy and the latest revision :
svn diff file
Differences between 2 revisions :
svn diff -r rev_n:rev_m file
Show modification author :
svn blame file
Show commit messages :
svn log file

How to concatenate video files (ffmpeg, mencoder) ?

The cat method :

cat 01.mpg 02.mpg 03.mpg >> final.mpg

  • This only works with real MPEG files.
  • The displayed video duration may be the duration of 01.mpg instead of the sum of the concatenated videos duration. The progress bar will reach 100% at the end of 01.mpg.

The ffmpeg method (source) :

This was used to convert Quicktime videos (in .mpg files ?) into MPEG videos :

ffmpeg -i 02.mpg -deinterlace -sameq 02.mpeg

ffmpeg + cat (tested on Quicktime to MPEG):

oldExtension='.mpg';newExtension='.mpeg';for vdo in *$oldExtension;do ffmpeg -i $vdo -sameq $(basename $vdo $oldExtension)$newExtension;done;cat *$newExtension > final$newExtension

The files to convert + concatenate must be separated from others in a work directory.

  1. type 1 in an empty cell, then Rclick cell | copy
  2. hold CTRL and select all cells where hyperlinks have to be removed
  3. Rclick any of the selected cells (containing hyperlink to remove), then "collage spécial"
  4. In the Opération area, select multiplication | OK

Encode wav files into MP3 with Lame

Flags :

Flag Usage
-b set bitrate
-v enable variable bitrate

Example :

Encode *wav into *mp3 :

for i in *wav; do lame -v -b 256 "$i" "$i.mp3"; done

Finding duplicates in an Excel column

  1. let's assume the column containing duplicates is the F column
  2. insert an empty column (the G column) next to the column to investigate for duplicates
  3. in G1 (or G2 if column have headers), insert the formula =NB.SI($F$2:$F$120;F2)>1
    • $F$2:$F$120 is the range of cells to search into
    • F2 is the value to look for
    • =NB.SI(range;value) counts the occurrences of value in range. Unique values occur just once, hence the >1 to highlight duplicates
  4. drag'n'copy this formula until the bottom of the G column
  5. lines containing duplicates in the F colum show a true in the G column

Advanced formula : =SI(NB.SI($F$2:$F$120;F2)>1;"DOUBLON !!";"")

When Outlook complains : "the set of directories could not be opened..."

Situation :

Not possible anymore to open Outlook, which outputs the error message : "Cannot start Microsoft Office Outlook. Unable to open Outlook window. The set of directories could not be opened."

Details :

This is due to a corrupted .ost file. These files contain the "local copy" of the Outlook inbox.

Solution :

Scan and repair the damaged .ost file with the scanost.exe utility. This was found in C:\Program Files\Common Files\System\MSMAPI\1033 on my computer.

RDP shortcut keys

Keys Action
CTRL-ALT-End Equivalent to CTRL-ALT-DEL into a remote session
CTRL-ALT-Break Toggle fullscreen

Comments in programming, scripts, ...

Context Comment starts with ... Ends with ...
ASP ' end of line
Bash, MySQL query # end of line
C, PHP, Javascript // (single line) end of line
/* (multiple lines) */
CSS /* */
HTML <!-- -->
LATEX % end of line
MS-Dos Batch (.bat, .cmd) REM end of line
Oracle query -- end of line
PERL # (single line) end of line
=cut

(multiple lines)
=cut

VBA, VBS ' end of line

MPlayer

How to record a streamed content ?

mplayer -dumpfile mediaFile.avi -dumpstream mms://media.website.com/path/to/mediaFile.avi

How to make screenshots of videos ?

Making a screenshot of the MPlayer window while playing a video just shows a blank window, so how to proceed ? Most of the time, the system tool making screenshots captures a blank window because the video is rendered through hardware components, whereas the screenshot tool has no access to this rendered information. Solution :
  1. Start MPlayer with software video rendering : mplayer a_great_movie.avi -vf screenshot
  2. While the movie is playing, use :
    • [SPACE] to pause / un-pause playing
    • . (1st time) to pause the movie
    • . (next times) to play the movie frame by frame
    • s to make a screenshot. It will be saved in the current directory as shot0001.png. The frame that will be captured is the one right after the frame currently displayed.
    • S to take continuous screenshots. Don't use this for too long as it 'eats' HDD free space !!
    • s to stop continuous screenshots

How to change the font size in xterm ?

  1. Pick a font from xfontsel. Get its specification string into the clipboard with .
  2. Append your chosen font specification string into ~/.Xresources :
    xterm*font:	*-fixed-medium-r-*-*-18-*
  3. Restart X or reload settings immediately : xrdb -merge ~/.Xresources