PIP - Httqm's Docs

pip list

module1 (x.y.z)
module2 (x.y.z)
module3 (x.y.z)

pip freeze

module1==x.y.z
module2==x.y.z
module3==x.y.z

pip show getopt wstools

Name: wstools
Version: 0.4.3
Summary: wstools
Home-page: https://github.com/pycontribs/wstools.git
Author: Gregory Warnes, kiorky, sorin
Author-email: Gregory.R.Warnes@Pfizer.com, kiorky@cryptelium.net, sorin.sbarnea+os@gmail.com
License: UNKNOWN
Location: /usr/lib/python2.7/dist-packages
Requires:

and finally :

python

Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> help("modules")

Please wait a moment while I gather a list of all available modules...

BaseHTTPServer      argparse            ihooks              samba
Bastion             array               imaplib             sched
CDROM               ast                 imghdr              secretstorage
CGIHTTPServer       asynchat            imp                 select

It is pretty common that servers aren't directly connected to the Internet, which makes it impractical to download + install packages with tools like pip3 or apt. The workaround is then to make an "offline install" :

from a machine that has internet access, download the required packages and their dependencies :

pip3 download ansible

Collecting ansible
  Downloading https://files.pythonhosted.org/packages/e8/b2/c10b82573bb494d9e0764b5c75eb7be1e649082435280f2220976d1a3b33/ansible-2.9.11.tar.gz (14.2MB)
    100% |████████████████████████████████| 14.3MB 94kB/s
  Saved ./ansible-2.9.11.tar.gz
Collecting PyYAML (from ansible)
  Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)
    100% |████████████████████████████████| 276kB 1.1MB/s
  Saved ./PyYAML-5.3.1.tar.gz
Collecting cryptography (from ansible)
  Downloading https://files.pythonhosted.org/packages/26/d6/e8b087b9ae062d737c67c3bf76e30333bda9295ca17205062e8ed2c872de/cryptography-3.0-cp35-abi3-manylinux1_x86_64.whl (2.7MB)
    100% |████████████████████████████████| 2.7MB 368kB/s
  Saved ./cryptography-3.0-cp35-abi3-manylinux1_x86_64.whl
Collecting jinja2 (from ansible)
  Downloading https://files.pythonhosted.org/packages/30/9e/f663a2aa66a09d838042ae1a2c5659828bb9b41ea3a6efa20a20fd92b121/Jinja2-2.11.2-py2.py3-none-any.whl (125kB)
    100% |████████████████████████████████| 133kB 2.2MB/s
  Saved ./Jinja2-2.11.2-py2.py3-none-any.whl
Collecting six>=1.4.1 (from cryptography->ansible)
  Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
  Saved ./six-1.15.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography->ansible)
  Downloading https://files.pythonhosted.org/packages/d1/fd/10e2050b3afcb03d78b2b33212c3ff17b3985ae25d0feb34c0c9eb417549/cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401kB)
    100% |████████████████████████████████| 409kB 1.1MB/s
  Saved ./cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting MarkupSafe>=0.23 (from jinja2->ansible)
  Using cached https://files.pythonhosted.org/packages/98/7b/ff284bd8c80654e471b769062a9b43cc5d03e7a615048d96f4619df8d420/MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl
  Saved ./MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography->ansible)
  Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
  Saved ./pycparser-2.20-py2.py3-none-any.whl
Successfully downloaded ansible PyYAML cryptography jinja2 six cffi MarkupSafe pycparser

send the downloaded files to the destination machine
install packages locally :
pip3 install path/to/downloadedPackage
- PyYAML is a dependency of ansible and it looks like pip3 install ./ansible-2.9.11.tar.gz is not able to install PyYAML from the local package (tries to download it instead). In such case, install dependencies first, then retry installing the main package.
- Looks like the downloaded packages (architecture, version, ... ?) match the machine they are downloaded from, but may not fit the destination machine and fail to install (to be investigated).

Situation

Full error message :

  ----------------------------------------
  Failed building wheel for ansible
  Running setup.py clean for ansible
  Running setup.py bdist_wheel for pycparser ... error
  Complete output from command /home/kevin/myGreatProject/myVirtualEnvironment/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-bz2ssx4w/pycparser/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-0z6zygi4 --python-tag cp37:


  error: invalid command 'bdist_wheel'

  ----------------------------------------

I got this error while installing Ansible but it can occur during the installation of any package since something is missing locally .

Solution

pip3 install wheel --upgrade
retry the command you were running in the first place

When playing with packages and versions, it may be safe to work inside a virtual environment.

List all available version of a package (source) :

Looks like there is no tool or command flag to do so, and the only option is to ask the installation of a non-existent version to read the expected information from the error message :

pip3 install ansible==

Collecting ansible==
  Could not find a version that satisfies the requirement ansible== (from versions: 1.0, 1.1, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.6, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.6.8, 1.6.9, 1.6.10, 1.7, 1.7.1, 1.7.2, 1.8, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.9.0.1, 1.9.1, 1.9.2, 1.9.3, 1.9.4, 1.9.5, 1.9.6, 2.0.0.0, 2.0.0.1, 2.0.0.2, 2.0.1.0, 2.0.2.0, 2.1.0.0, 2.1.1.0, 2.1.2.0, 2.1.3.0, 2.1.4.0, 2.1.5.0, 2.1.6.0, 2.2.0.0, 2.2.1.0, 2.2.2.0, 2.2.3.0, 2.3.0.0, 2.3.1.0, 2.3.2.0, 2.3.3.0, 2.4.0.0, 2.4.1.0, 2.4.2.0, 2.4.3.0, 2.4.4.0, 2.4.5.0, 2.4.6.0, 2.5.0a1, 2.5.0b1, 2.5.0b2, 2.5.0rc1, 2.5.0rc2, 2.5.0rc3, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.5.6, 2.5.7, 2.5.8, 2.5.9, 2.5.10, 2.5.11, 2.5.12, 2.5.13, 2.5.14, 2.5.15, 2.6.0a1, 2.6.0a2, 2.6.0rc1, 2.6.0rc2, 2.6.0rc3, 2.6.0rc4, 2.6.0rc5, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.6.6, 2.6.7, 2.6.8, 2.6.9, 2.6.10, 2.6.11, 2.6.12, 2.6.13, 2.6.14, 2.6.15, 2.6.16, 2.6.17, 2.6.18, 2.6.19, 2.7.0.dev0, 2.7.0a1, 2.7.0b1, 2.7.0rc1, 2.7.0rc2, 2.7.0rc3, 2.7.0rc4, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.7.5, 2.7.6, 2.7.7, 2.7.8, 2.7.9, 2.7.10, 2.7.11, 2.7.12, 2.7.13, 2.8.0a1, 2.8.0b1, 2.8.0rc1, 2.8.0rc2, 2.8.0rc3, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.8.5, 2.9.0b1, 2.9.0rc1, 2.9.0rc2, 2.9.0rc3)
No matching distribution found for ansible==

Improved version :

pip3 install ansible== 2>&1 | tr '), ' '\n' | grep -E '^[0-9]' | sort -nr | less

2.9.0rc3
2.9.0rc2
2.9.0rc1
2.9.0b1
2.8.5
2.8.4
2.8.3

Install a specific version :

From above :

pip3 install ansible==2.7.9

Situation

While trying to setup Molecule, a pip3 install -U [packages] command ends on :

building 'psutil._psutil_linux' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/psutil
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/home/bob/virtualenvs/projectName/include -I/usr/include/python3.7m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.7/psutil/_psutil_common.o
psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory	the line above the error 
 #include <Python.h>
          ^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1			the error

Details

As said here : Sometimes it's the line above this error you should look at. It should tell you what package is missing.

Solution

apt install python3-dev

Check :

dpkg -S Python.h

libpython3.7-dev:amd64: /usr/include/python3.7m/Python.h

Situation

I don't have the exact sequence of events and commands that led me here, but what I read while fixing this emphasize that PEBKAC :

while actually trying to use youtube-dl, it displayed my current version was out-of-date and I should upgrade (which I did)
the upgrade process suggested to upgrade pip too, and even offered the command line to do so. I copy-pasted it and executed it ()

after that, every youtube-dl command failed on :

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl", line 6, in <module>
    from youtube_dl import main
ImportError: cannot import name 'main'

Details

This StackOverflow thread has some solutions that worked for others, but did not fix things directly for me (guess I f*cked things a little more than others ). But I think this helped anyway cleaning a little of the mess I put.
While trying solutions suggested by the thread above, I also had apt commands returning errors like :
```
/usr/bin/mandb: can't search directory /usr/local/man: Permission denied
```
which is pretty unexpected since this was run as root. Looks like permissions were pretty f*cked up already :
ll /usr/local/man
```
lrwxrwxrwx 1 root staff 9 Oct 11  2016 /usr/local/man -> share/man
```
ll -d /usr/local/share/man
```
drwx--S---  3 root staff 4096 Mar 19 11:14 /usr/local/share/man	
```
Solution :
chmod -R a+rX /usr/local/share/man

Solution

This GitHub issue details the reasons of this error (the PEBKAC part, notably) and hints to debug / fix.

Unless explicitly specified, all commands below were run as a non-root user.

First, identify the full absolute path of the executable script that you are running :
which youtube-dl
```
/usr/local/bin/youtube-dl
```

Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script :

/usr/local/bin/youtube-dl

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl", line 6, in <module>
    from youtube_dl import main
ImportError: cannot import name 'main'

work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script :
head /usr/local/bin/youtube-dl
```
#!/usr/bin/python3
```

Once you know which Python is running pip, you can run python -m pip to invoke pip :

python -m pip

(returns the 'usage' page, but no error)

python3 -m pip

  
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 639, in __init__
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 695, in add_entry
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2012, in find_on_path
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/dist-packages/youtube_dl-2019.3.18.dist-info'

The last line is quite surprising and also recalls something seen minutes ago. The permissions were actually kinda weird :

as root :

ll /usr/local/lib/python3.5/dist-packages


drwx--S--- 6 root staff 4.0K Mar 19 08:52 youtube_dl
drwx--S--- 2 root staff 4.0K Mar 19 08:52 youtube_dl-2019.3.18.dist-info

as root again :
chmod -R a+rX /usr/local/lib/python3.5/dist-packages/
then, as non-root :
youtube-dl
```
(the 'usage' page )
```

This is highly context-specific and not exactly a "clean" solution, but maybe this will give you hints to find the root cause on your side... If anything, this issue,at least, led me to some best practices :

Best practices (source) :

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that "it's your fault and we won't help", but rather to help users to identify what went wrong, so that they can work out what to do next more easily.

Only ever use your system package manager to upgrade the system pip. The system installed pip is owned by the distribution, and if you don't use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says "you should upgrade with pip install -U pip" - that's true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don't. We're working with them on this, but it's not going to happen soon (remember, we're looking at cases where people are upgrading old versions of pip here, so patches to new versions won't help).
Never use sudo with pip. This follows on from the first point. If you think you need to use sudo, you're probably trying to modify a distribution-owned file. See point 1.
Prefer to use --user. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are PATH issues you need to be aware of here. We'll cover these later. Put simply, it's possible to follow this advice, and still hit problems, because you're not actually running the wrapper you installed as --user.

Situation

While trying to install mps-youtube (sudo pip3 install mps-youtube), I get the error :

...
MANY lines describing the exceptions
...
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

Details

This is because the installed version of pip (probably from the Debian packages) is too old.

Solution

apt remove python-pip python3-pip
proxyString="Kevin:p4ssw0rd@proxy:3128"; export http_proxy="http://$proxyString"; export https_proxy="https://$proxyString"
cd /tmp && wget https://bootstrap.pypa.io/get-pip.py

python get-pip.py

Collecting pip
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 781kB/s
Installing collected packages: pip
Successfully installed pip-9.0.1

python3 get-pip.py

Collecting pip
  Using cached pip-9.0.1-py2.py3-none-any.whl
Collecting wheel
  Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB)
    100% |████████████████████████████████| 51kB 718kB/s
Installing collected packages: pip, wheel
Successfully installed pip-9.0.1 wheel-0.30.0

You can now retry installing :

/usr/local/bin/pip3 install mps-youtube

Collecting mps-youtube
  Downloading mps_youtube-0.2.7.1-py3-none-any.whl (74kB)
    100% |████████████████████████████████| 81kB 974kB/s
Requirement already satisfied: pafy!=0.4.0,!=0.4.1,!=0.4.2,>=0.3.82 in /usr/lib/python3/dist-packages (from mps-youtube)
Installing collected packages: mps-youtube
Successfully installed mps-youtube-0.2.7.1

PIP - the Package Installer for Python

pip : How to list the installed modules ?

What is PIP ?

pip3 : how to make an "offline install" ?

pip3 : error: invalid command 'bdist_wheel'

Situation

Solution

pip3 : how to list all available versions of a package and install a specific version ?

List all available version of a package (source) :

Install a specific version :

pip3 ends on error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Situation

Details

Solution

Error after upgrading pip : ImportError: cannot import name 'main'

Situation

Details

Solution

Best practices (source) :

pip ends on error TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

Situation

Details

Solution

How to install packages for Python3 with pip ?