Python - Tips & snippets

Error after upgrading pip : ImportError: cannot import name 'main'

Situation :

I don't have the exact sequence of events and commands that led me here, but what I read while fixing this emphasize that PEBKAC :
  1. while actually trying to use youtube-dl, it displayed my current version was out-of-date and I should upgrade (which I did)
  2. the upgrade process suggested to upgrade pip too, and even offered the command line to do so. I copy-pasted it and executed it ()
  3. after that, every youtube-dl command failed on :
    Traceback (most recent call last):
      File "/usr/local/bin/youtube-dl", line 6, in <module>
        from youtube_dl import main
    ImportError: cannot import name 'main'

Details :

Solution :

This GitHub issue details the reasons of this error (the PEBKAC part, notably) and hints to debug / fix.

Unless explicitly specified, all commands below were run as a non-root user.
  1. First, identify the full absolute path of the executable script that you are running :
    which youtube-dl
    /usr/local/bin/youtube-dl
  2. Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script :
    /usr/local/bin/youtube-dl
    Traceback (most recent call last):
      File "/usr/local/bin/youtube-dl", line 6, in <module>
        from youtube_dl import main
    ImportError: cannot import name 'main'
  3. work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script :
    head /usr/local/bin/youtube-dl
    #!/usr/bin/python3
  4. Once you know which Python is running pip, you can run python -m pip to invoke pip :
    python -m pip
    (returns the 'usage' page, but no error)
    python3 -m pip
      ...
      File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 639, in __init__
      File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 695, in add_entry
      File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2012, in find_on_path
    PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/dist-packages/youtube_dl-2019.3.18.dist-info'
    The last line is quite surprising and also recalls something seen minutes ago. The permissions were actually kinda weird :
  5. as root :
    ll /usr/local/lib/python3.5/dist-packages
    ...
    drwx--S--- 6 root staff 4.0K Mar 19 08:52 youtube_dl
    drwx--S--- 2 root staff 4.0K Mar 19 08:52 youtube_dl-2019.3.18.dist-info
  6. as root again :
    chmod -R a+rX /usr/local/lib/python3.5/dist-packages/
  7. then, as non-root :
    youtube-dl
    (the 'usage' page )

This is highly context-specific and not exactly a "clean" solution, but maybe this will give you hints to find the root cause on your side... If anything, this issue,at least, led me to some best practices :

Best practices (source) :

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that "it's your fault and we won't help", but rather to help users to identify what went wrong, so that they can work out what to do next more easily.

  1. Only ever use your system package manager to upgrade the system pip. The system installed pip is owned by the distribution, and if you don't use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says "you should upgrade with pip install -U pip" - that's true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don't. We're working with them on this, but it's not going to happen soon (remember, we're looking at cases where people are upgrading old versions of pip here, so patches to new versions won't help).
  2. Never use sudo with pip. This follows on from the first point. If you think you need to use sudo, you're probably trying to modify a distribution-owned file. See point 1.
  3. Prefer to use --user. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are PATH issues you need to be aware of here. We'll cover these later. Put simply, it's possible to follow this advice, and still hit problems, because you're not actually running the wrapper you installed as --user.

pip ends on error TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

Situation :

While trying to install mps-youtube (sudo pip3 install mps-youtube), I get the error :
...
MANY lines describing the exceptions
...
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

Details :

This is because the installed version of pip (probably from the Debian packages) is too old.

Solution :

  1. apt remove python-pip python3-pip
  2. proxyString="Kevin:p4ssw0rd@proxy:3128"; export http_proxy="http://$proxyString"; export https_proxy="https://$proxyString"
  3. cd /tmp && wget https://bootstrap.pypa.io/get-pip.py
  4. python get-pip.py
    Collecting pip
      Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
        100% |████████████████████████████████| 1.3MB 781kB/s
    Installing collected packages: pip
    Successfully installed pip-9.0.1
  5. python3 get-pip.py
    Collecting pip
      Using cached pip-9.0.1-py2.py3-none-any.whl
    Collecting wheel
      Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB)
        100% |████████████████████████████████| 51kB 718kB/s
    Installing collected packages: pip, wheel
    Successfully installed pip-9.0.1 wheel-0.30.0
  6. You can now retry installing :
    /usr/local/bin/pip3 install mps-youtube
    Collecting mps-youtube
      Downloading mps_youtube-0.2.7.1-py3-none-any.whl (74kB)
        100% |████████████████████████████████| 81kB 974kB/s
    Requirement already satisfied: pafy!=0.4.0,!=0.4.1,!=0.4.2,>=0.3.82 in /usr/lib/python3/dist-packages (from mps-youtube)
    Installing collected packages: mps-youtube
    Successfully installed mps-youtube-0.2.7.1

How to use dynamic variables ?

A code snippet is worth 1000 words :
#!/usr/bin/env python3

class Person(object):

    def __init__(self, firstName, lastName, age):
        self.firstName = firstName
        self.lastName = lastName
        self.age = age


    def getItem(self, itemName):
        return getattr(self, itemName)


user = Person('John', 'Smith', 42)

for item in [ 'firstName', 'lastName', 'age' ]:
    print(user.getItem(item))
will display :
John
Smith
42

The Python Challenge

The Python Challenge is a series of puzzles aiming at learning / practicing / improving Python development skills.
In notes below :

Level 0 :

import math
print(math.pow(2,38))
274877906944.0

Next level : http://www.pythonchallenge.com/pc/def/274877906944.html

Level 1 :

inTable = 'KOE'
outTable = 'MQG'
message = 'KOE'
print(message.translate({ord(x): y for (x, y) in zip(inTable, outTable)}))
MQG
List Comprehensions (source : PEP 202, Wikipedia) :

List comprehensions provide a more concise way to create lists in situations where nested loops would currently be used.

print({x: y for (x, y) in zip(inTable, outTable)})
{'K': 'M', 'E': 'G', 'O': 'Q'}
About zip :
print({ord(x): y for (x, y) in zip(inTable, outTable)})
{75: 'M', 69: 'G', 79: 'Q'}
inTable = 'abcdefghijklmnopqrstuvwxyz'
outTable = 'cdefghijklmnopqrstuvwxyzab'
message = "g fmnc wms bgblr rpylqjyrc gr zw fylb. rfyrq ufyr amknsrcpq ypc dmp. bmgle gr gl zw fylb gq glcddgagclr ylb rfyr'q ufw rfgq rcvr gq qm jmle. sqgle qrpgle.kyicrpylq() gq pcamkkclbcb. lmu ynnjw ml rfc spj."
print(message.translate({ord(x): y for (x, y) in zip(inTable, outTable)}))
i hope you didnt translate it by hand. thats what computers are for. doing it in by hand is inefficient and that's why this text is so long. using string.maketrans() is recommended. now apply on the url.
A smarter solution to build the inTable and outTable strings :
import string
inTable = string.ascii_lowercase
outTable = string.ascii_lowercase[2:] + string.ascii_lowercase[:2]

Next level : http://www.pythonchallenge.com/pc/def/ocr.html

Level 2 :

The question is in the HTML source :

find rare characters in the mess below:

%%$@_$^__#)^)&!_+]!*@&^}@[@%]()%+$&[(_@%+%$*^@$^!+]!&_#)_*}{}}!}_]$[%}@[{_@#_^{*
@##&{#&{&)*%(]{{([*}@[@&]+!!*{)!}{%+{))])[!^})+)$]#{*+^((@^@}$[**$&^{$!@#$%)!@(&
+^!{%_$&@^!}$_${)$_#)!({@!)(^}!*^&!$%_&&}&_#&@{)]{+)%*{&*%*&@%$+]!*__(#!*){%&@++
!_)^$&&%#+)}!@!)&^}**#!_$([$!$}#*^}$+&#[{*{}{((#$]{[$[$$()_#}!@}^@_&
...

The Bash answer :
localFile='./level02.txt'; >"$localFile"; wget -O "$localFile" http://www.pythonchallenge.com/pc/def/ocr.html; sed -i '38,1257 !d' "$localFile"; sed -ri 'sa[][)({}@%&$_+*^#!]aag' "$localFile"; sed -i '/^$/ d' "$localFile"; tr '\n' '\0' < "$localFile"

Next level :

How to transform an XML file using XSLT in Python ?

  1. Install requirements, as root :
    apt install libxml2-dev libxslt-dev
    Otherwise, the next step will fail complaining make sure the development packages of libxml2 and libxslt are installed.
  2. Install lxml, the library that will do the job (still as root) :
    pip-3.2 install lxml
    /usr/bin/pip-3.2 is Python's pip for Python 3.2, since I have both 2.x and 3.x versions installed.
  3. Then write a transcoding script such as :
    #!/usr/bin/env python3
    
    import lxml.etree as ET
    
    dom = ET.parse('data.xml')
    xslt = ET.parse('stylesheet.xsl')
    transform = ET.XSLT(xslt)
    newdom = transform(dom)
    print(ET.tostring(newdom, pretty_print=True))
    Due to Python's byte objects, print's internal logic and things that are not yet clear to me (and Python3.x limitations on lxml side ?), this script outputs a "byte object string", that I'll clean (the ugliest possible way) with sed :
    • changing \n into [newline]
    • changing \t into [tab]
    • changing \' into '
  4. Don't forget to move all the related .xsl files to the work directory
  5. Then run the script : python3 ./script.py | sed -r 's_^..(.*).$_\1_g' | sed -r 's_\\n_\n_g' | sed -r 's_\\t_\t_g' | sed -r "s_\\\'_'_g" > output.html
  6. Enjoy !

How to compute date and time ?

import datetime
from datetime import datetime, timedelta
timeFormat = '%Y-%m-%d %H:%M:%S'
now = datetime.now()
print(now.strftime(timeFormat))
future = now + timedelta(seconds=500)
print(future.strftime(timeFormat))

How to install packages for Python3 with pip ?

apt install python3-pip; pip3 install requests