Pip tools are a set of third party command line tools designed to help with Python 3 dependency management by keeping dependencies up to date, even when you have them pinned.
You can install it by running the following command:
python -m pip install pip-toolsYou can then run the following command to automatically create a requirements.txt file based on your setup.py or requirements.in file:
python -m piptools compileIf your existing requirements.txt file contains the complete and proper set of dependencies, no changes will be made, even if updates are available. However, you can force pip-compile to update all packages in an existing requirements.txt by running the following command:
pip-compile --upgradeYou can even update specific packages to the latest version (or any specific version) by running the following command:
pip-compile --upgrade-packageSo far so good, but pip tools (like pip itself) does not do Python dependency resolution, which means that upgrading is a crapshoot: you may pull in dependencies or transitive dependencies (ie., dependencies of dependencies) that don’t work together, thereby corrupting your environment.
Advanced Dependency Management
Unfortunately, most real world Python projects are more complicated than those supported by pip tools. For example, pip tools currently provides no way to:
- Distinguish packages from the dependencies they pull in
- Create a set dependencies and versions that are known to be compatible
- Update a virtual environment efficiently with a single command
- Support multiple configurations of dependencies that aren’t just subsets of the original. For example, one configuration for each environment: development, test/CI/CD, production, etc.
If you require any of these features, consider the ActiveState Platform, which offers advanced dependency management for Python projects.
Distinguish Packages from Dependencies
Packages refer to those Python libraries that are required to run your application. Dependencies are libraries that the packages require to be able to run. There are other dependencies, as well, including transitive dependencies (which are dependencies of dependencies) and Operating System or OS-level dependencies (which are dependencies required to run the application on the specified OS).
For each Python project the ActiveState Platform provides:
- A complete Bill of Materials (BoM) view showing:
- The Python version
- Packages
- Direct dependencies, as well as transitive dependencies
- OS-level dependencies
- Shared dependencies (ie., OpenSSL)
As a result, you can easily distinguish between top level packages you explicitly included in your project versus their dependencies that were pulled in automatically.
Dependency Resolution
Python dependency resolution refers to the ability for a package manager (like pip) to be able to not only pull in all the dependencies required by the packages in your project, but also be able to ensure they are all compatible with each other. Unfortunately, neither pip nor pip tools are able to accomplish this.
By way of comparison, the ActiveState Platform will not only resolve all dependencies to ensure they work together but will also flag unresolvable conflicts and even suggest a manual workaround to the problem.
By following the instructions and selecting a more recent version of pandas will resolve the conflict. Note that when upgrading a package, the ActiveState Platform will show you all the cascading configuration changes so you can know the ramifications of any change to your environment BEFORE you commit to the update.
Virtual Environment Creation & Updating
As most Python software engineers know, the best practice is to install each of your projects in a virtual environment in order to isolate dependencies and prevent conflicts.
The ActiveState Platform provides natives support for virtual environments, which means that whenever you install a project using our command line interface, the State Tool will create a virtual environment (similar to venv, virtualenv or pipenv) on your local system and populate it with the runtime environment, including:
- The version of Python required by your project
- All the packages and dependencies required to run your application
The runtime environment is pre-compiled from source code by the ActiveState Platform (including linked C libraries) in a reproducible manner. As a result, deployment happens much faster than if every dependency was independently retrieved from a repository (like PyPI or GitHub) in a serial manner.
Updates to the local environment happen in the same: a single command downloads and installs all updated dependencies.
Multiple Python Environment Configurations
A separate configuration of the Python runtime environment is often required for each environment the application is to be developed, tested and run in. The most common way to do this is to create a default configuration (typically for the development environment) and have the other environments simply inherit the dependencies with minor differences.
However, problems arise when the configuration is targeted at a different operating system or platform, such as Python 2, which might arise when doing migration work. Simple inheritance is not enough.
The ActiveState Platform provides the flexibility to create multiple child configurations from a parent configuration and:
- Inherit all, some or no updates from the parent configuration
- Completely change the child configuration, including OS, platform and packages/dependencies, as required
Ready to see for yourself? You can try the ActiveState Platform by signing up for a free account using your email or GitHub credentials. Or sign up for a free demo and let us show you how you can better manage dependencies.
Watch this video to learn how to use the ActiveState Platform to create a Python 3.9 environment, and then use the Platform’s CLI (State Tool)to install and manage it.
