Installing libraries and packages¶
Python packages are a set of python modules, while python librariesare a group of python functions aimed to carry out specialtasks. There are over 137,000 python libraries and over 235,000 pythonpackages. These libraries and packages can ease a developer’sexperience and avoid the need to re-invent the wheel, as the sayinggoes. Henceforth, the word package(s) will be used when referring toboth packages and libraries. While there are distinctions withinpython, the words packages and libraries are frequently interchanged,and the process for installing them is generally the same.
Cartoon by Randall Munroe, https://xkcd.com/1987/
There are several ways that Python packages can be installed andmanaged. ARC support staff can install packages that can be madeavailable to anyone who loads the appropriate module, whileindividuals can install packages for their own use. There are two mainroutes for users to install packages. One is to install directly intothe user’s personal python library using the pip command. In thatcase, all packages (for a given version of python) are stored in thesame directory. However, this approach can result in conflicts withpackage version requirements.
Sometimes one application needs a particular version of a package buta different application needs another version. Since the requirementsconflict, installing either version will leave one application unableto run. This situation can be resolved by using virtualenvironments. A virtual environment is a semi-isolated Pythonenvironment that allows packages to be installed for use by aparticular application or for a particular project.
Common tools used for Python package installation and environment management include:
- pip – the Python package installer, can be used on its own or within a virtual environment
- venv – an environment manager within which pip is used to install packages
- virtualenv – an environment manager within which pip is used to install packages
- conda – an environment manager and a package installer, it does not rely on pip to install packages
You should pick one method and use it exclusively to avoid mixingtypes of installations.
Python packages are installed for the specific version of Python thatis in use during installation. If you switch from using a module forone version of Python to a different one with where either the majoror minor version changes, then you will have to re-install anypackages/libraries in order to make them available in the library ofthe new version of Python. You only need to install Python packagesonce for each cluster on which you wish to use the library and,separately, for each version of Python that you use. Please note,Python packages should be installed using the command line from alogin node, not from within Jupyter Notebook or the JupyterLab app.
A brief description of how to use each tool to install packages andmanage virtual environments, along with a description of where you canexpect to find installed packages, is provided below.
Pip¶
To install a Python package into your personal library using pip,enter the following command, replacing
$ pip install --user <package_name>The --user tag will, by default, place packages in
$HOME/.local/lib/python?.?/site-packageswhere ?.? indicates the versioning of the Python release. Thelibrary will then be available to you for this and future sessions.
You can install a specific version of a package by giving the packagename followed by == and the version number. For example:
pip install --user tensorflow==2.5.0Venv¶
venv is the standard Python tool for creating virtual environments,and has been part of Python since version 3.3. Starting with Python3.4, it defaults to installing pip into all created virtualenvironments. Installing packages into an active venv is done viathe pip command, as described above.
Virtual environments are created as follows:
$ python -m venv /path/to/new/virtual/environmentAlternatively, you can change into the directory of the project youare working on and simply provide a name for the virtual environmentin place of the full path:
$ cd /path/to/my/project$ python -m venv myenvTo activate a virtual environment, enter:
$ source myenv/bin/activateIf you are not in your project directory, then you must provide thefull path to the virtual environment you specified when creating thevirtual environment:
$ source /path/to/my/project/myenv/bin/activateWhen you are in an active virtual environment, you will see the nameof the environment in the prompt. The PATH environment variable isupdated so that the virtual environment’s bin directory is at thebeginning:
(myenv) $ which pip python~/my_project/myvenv/bin/pip~/my_project/myvenv/bin/pythonAt this point, you would use the pip command to install any neededpackages. Packages that you install using pip while in a virtualenvironment will be placed in the myenv folder, isolated from theglobal Python installation, and only available to you from within thevirtual environment.
You can deactivate a virtual environment by typing deactivate inyour terminal.
Virtualenv¶
virtualenv is a third party alternative (and predecessor) tovenv. It comes installed with the Developer Python modules. However,it is not installed for the Anaconda modules. If you would like towork within a virtual environment using virtualenv with one of theAnaconda modules, you will need to install it.
Install virtualenv with pip:
pip install --user virtualenvCreate a virtual environment for a project:
$ cd project_folder$ virtualenv <env_name>Similar to the way venv works with Python 3, virtualenv myenv willcreate a folder in the current directory which will contain the Pythonexecutable files and a copy of the pip library which you can use toinstall other packages. To begin using the virtual environment, it hasto be activated:
$ source myenv/bin/activateThe name of the current virtual environment will now appear inparenthesis to the left of the prompt to let you know that it’sactive.
When done working in the virtual environment, simply deactivate it:
(myvenv) $ deactivateConda¶
Conda is both a package installer, like pip, and an environmentmanager, like venv and virtualenv. While pip, venv, andvirtualenv are for Python, conda is language agnostic and workswith other languages as well as Python.
To create a virtual environment for Python with conda, enter thefollowing:
$ conda create --name conda-env pythonwhere conda-env can be replaced with whatever name you choose foryour virtual environment. Also, -n can be used in place of --name.This environment will use the same version of Python as your currentshell’s Python interpreter. To specify a different version of Python,specify the version number when creating your virtual environment asfollows:
$ conda create -n conda-env python=3.7You can install additional packages when creating an environment, byspecifying them after the environment name. You can also specify whichversions of packages you’d like to install.
$ conda create -n conda-env python=3.7 numpy=1.16.1 requests=2.19.1It’s recommended to install all packages that you want to include inan environment at the same time in order to avoid dependencyconflicts.
You can then activate your conda environment as follows:
$ conda activate conda-env(conda-env) $As when using other virtual environment programs, the name of thecurrent virtual environment will now appear in parenthesis to the leftof the prompt to let you know that it’s active. When you are finishedworking in the environment, simply enter $ conda deactivate and yournormal prompt will return.
Virtual environments created with conda reside, by default, in theenvs directory found in the following path: /home/$USER/.conda/envs
