Python - Tips & snippets

mail

Share files with Python and a HTTP server

How do you do this ? (source)

  1. on the host having the files to share (aka theServer), cd to the directory containing those files
  2. share files :
    • with Python 2 : python -m SimpleHTTPServer port
    • with Python 3 : python -m http.server port
  3. On the client side, files will be available at http://theServer:port. You can retrieve files easily with wget -c url

Is it efficient, at least ? Let's check this !

Let's start by generating some test data :

dd if=/dev/urandom of=/tmp/testFile bs=1M count=500 & watch -n 1 -d 'ls -lh /tmp/testFile'

Don't try this with /dev/random or it will take forever.

The Python method :

Get the server's IP address :
hostname -i
Serve the file :
cd /tmp; python -m SimpleHTTPServer 8888
Get the file from another host :
cd /tmp; time wget http://12.34.56.78:8888/testFile -O ./testFile
-O ./testFile is there so that we can repeat the download without creating ./testFile.1, ./testFile.2, ...
Length: 524288000 (500M) [application/octet-stream]
Saving to: `./testFile'

100%[============================================================================>] 524,288,000 90.8M/s in 5.2s

2015-06-12 11:55:00 (96.7 MB/s) - `./testFile' saved [524288000/524288000]


real	0m5.289s
user	0m0.137s
sys	0m1.950s

The scp method :

Get the file from another host :
time scp @12.34.56.78:/tmp/testFile /tmp/testFile 
testFile								100%	500MB	33.3MB/s	00:15

real	0m14.233s
user	0m11.118s
sys	0m4.453s

The Rsync method :

Get the file from another host :
[ -f "/tmp/testFile" ] && rm -f /tmp/testFile; time rsync -avz -e ssh @12.34.56.78:/tmp/testFile /tmp/testFile 
receiving incremental file list
testFile

sent 30 bytes	received 524543976 bytes	13987840.16 bytes/sec
total size is 524288000	speedup is 1.00

real	0m37.000s
user	0m13.829s
sys	0m3.601s

Results :

  • From fastest to slowest : Python, scp, Rsync.
  • Files sent via the Python method will transit in clear format (no encryption), which should be avoided outside of a controlled environment such as a LAN.
  • If the shared file contains "incremental" data (such as a database dump, a logfile, ..., meaning most of the data has already been transmitted), Rsync is able to transfer only the updated chunks and may be the optimal solution.
mail

Python's Virtual environments, virtualenv, venv

What are Virtual environments ? Why do we need them ? (source)

  • if you work on a single project : you don't need Virtual environments
  • if you work on projectA and projectB, which respectively require lib1 + lib2 and lib3 + lib4 : you don't need Virtual environments
  • if you work on projectA and projectB, which respectively require someGreatLib (version 12.9) and someGreatLib (version 42.0) : Virtual environments will help you have both versions on your computer.

Usage (source) :

virtualenv is the Python 2 tool to interact with virtual environments and won't be discussed here. I'll focus on Python 3 tools and methods only.

The Python 3 tool to manage virtual environments is venv, which —depending on your distro / version :

  • is built in the standard library
  • can be installed with : apt install python3-venv

Create a virtual environment :

The virtual environment itself :
  • is a directory (full of subdirs, libs and stuff )
  • is named after the "virtual environment name" you will supply
  • is created in the current directory
  • is a developer tool and need NOT be created by
  1. cd my/development/directory
  2. create the virtual environment :
    virtualEnvName='myVirtualEnvironmentName'; python3 -m venv "$virtualEnvName"
  3. activate it :
    source "$virtualEnvName/bin/activate"
    The shell prompt should now be preceded by : 
    (myVirtualEnvironmentName)
    showing the virtual environment is enabled.

Exit a virtual environment :

From within the virtual environment directory or not, as long as the virtual environment name is visible in the shell prompt :
deactivate
(myVirtualEnvironmentName) disappears.

Resume / reload a virtual environment :

Simply :
source bin/activate
based on the virtual environment you want to resume. To do so :
  • cd into the corresponding virtual environment directory first
  • or prepend bin/activate with the corresponding path
mail

How to use dynamic variables ?

A code snippet is worth 1000 words :
#!/usr/bin/env python3

class Person(object):

    def __init__(self, firstName, lastName, age):
        self.firstName = firstName
        self.lastName = lastName
        self.age = age


    def getItem(self, itemName):
        return getattr(self, itemName)


user = Person('John', 'Smith', 42)

for item in [ 'firstName', 'lastName', 'age' ]:
    print(user.getItem(item))
will display :
John
Smith
42
mail

The Python Challenge

The Python Challenge is a series of puzzles aiming at learning / practicing / improving Python development skills.
In notes below :
  • some python code is code fired from the Python shell : /usr/bin/python3
  • some python code is code stored in a file

Level 0 :

import math
print(math.pow(2,38))
274877906944.0

Next level : http://www.pythonchallenge.com/pc/def/274877906944.html

Level 1 :

inTable = 'KOE'
outTable = 'MQG'
message = 'KOE'
print(message.translate({ord(x): y for (x, y) in zip(inTable, outTable)}))
MQG

List Comprehensions (source : PEP 202, Wikipedia) :

List comprehensions provide a more concise way to create lists in situations where nested loops would currently be used.

print({x: y for (x, y) in zip(inTable, outTable)})
{'K': 'M', 'E': 'G', 'O': 'Q'}

About zip :

print({ord(x): y for (x, y) in zip(inTable, outTable)})
{75: 'M', 69: 'G', 79: 'Q'}
inTable = 'abcdefghijklmnopqrstuvwxyz'
outTable = 'cdefghijklmnopqrstuvwxyzab'
message = "g fmnc wms bgblr rpylqjyrc gr zw fylb. rfyrq ufyr amknsrcpq ypc dmp. bmgle gr gl zw fylb gq glcddgagclr ylb rfyr'q ufw rfgq rcvr gq qm jmle. sqgle qrpgle.kyicrpylq() gq pcamkkclbcb. lmu ynnjw ml rfc spj."
print(message.translate({ord(x): y for (x, y) in zip(inTable, outTable)}))
i hope you didnt translate it by hand. thats what computers are for. doing it in by hand is inefficient and that's why this text is so long. using string.maketrans() is recommended. now apply on the url.
A smarter solution to build the inTable and outTable strings : 
import string
inTable = string.ascii_lowercase
outTable = string.ascii_lowercase[2:] + string.ascii_lowercase[:2]

Next level : http://www.pythonchallenge.com/pc/def/ocr.html

Level 2 :

The question is in the HTML source :

find rare characters in the mess below:

%%$@_$^__#)^)&!_+]!*@&^}@[@%]()%+$&[(_@%+%$*^@$^!+]!&_#)_*}{}}!}_]$[%}@[{_@#_^{*
@##&{#&{&)*%(]{{([*}@[@&]+!!*{)!}{%+{))])[!^})+)$]#{*+^((@^@}$[**$&^{$!@#$%)!@(&
+^!{%_$&@^!}$_${)$_#)!({@!)(^}!*^&!$%_&&}&_#&@{)]{+)%*{&*%*&@%$+]!*__(#!*){%&@++
!_)^$&&%#+)}!@!)&^}**#!_$([$!$}#*^}$+&#[{*{}{((#$]{[$[$$()_#}!@}^@_&
...

The Bash answer :

localFile='./level02.txt'; >"$localFile"; wget -O "$localFile" http://www.pythonchallenge.com/pc/def/ocr.html; sed -i '38,1257 !d' "$localFile"; sed -ri 'sa[][)({}@%&$_+*^#!]aag' "$localFile"; sed -i '/^$/ d' "$localFile"; tr '\n' '\0' < "$localFile"

Next level :

mail

How to transform an XML file using XSLT in Python ?

  1. Install requirements, as :
    apt install libxml2-dev libxslt-dev
    Otherwise, the next step will fail complaining make sure the development packages of libxml2 and libxslt are installed.
  2. Install lxml, the library that will do the job (still as ) :
    pip-3.2 install lxml
    /usr/bin/pip-3.2 is Python's pip for Python 3.2, since I have both 2.x and 3.x versions installed.
  3. Then write a transcoding script such as : 
    #!/usr/bin/env python3

import lxml.etree as ET

dom = ET.parse('data.xml')
xslt = ET.parse('stylesheet.xsl')
transform = ET.XSLT(xslt)
newdom = transform(dom)
print(ET.tostring(newdom, pretty_print=True))
    Due to Python's byte objects, print's internal logic and things that are not yet clear to me (and Python3.x limitations on lxml side ?), this script outputs a "byte object string", that I'll clean (the ugliest possible way) with sed :
    • changing \n into [newline]
    • changing \t into [tab]
    • changing \' into '
  4. Don't forget to move all the related .xsl files to the work directory
  5. Then run the script : python3 ./script.py | sed -r 's_^..(.*).$_\1_g' | sed -r 's_\\n_\n_g' | sed -r 's_\\t_\t_g' | sed -r "s_\\\'_'_g" > output.html
  6. Enjoy !
mail

How to compute date and time ? 

import datetime
from datetime import datetime, timedelta
timeFormat = '%Y-%m-%d %H:%M:%S'
now = datetime.now()
print(now.strftime(timeFormat))
future = now + timedelta(seconds=500)
print(future.strftime(timeFormat))