Call Us: 888.421.1155
Blog

Python Dependency Management

Python virtual environments allow us to isolate applications in a common system, but what happens when we need to update one of the application’s packages without affecting dependent packages...

Introduction

Python virtual environments allow us to isolate applications in a common system, but what happens when we need to update one of the application’s packages without affecting dependent packages in the same application? Also, what of situations where we have conflicting support libraries, i.e. two or more packages expecting different versions of a third? Tools such as pipdeptree are instrumental in identifying package dependencies, but rather than solving these issues by hand let’s look at a python build tool: pip-tools.

At The Beep: A Time API

We’re going to write a simple application to demonstrate the benefits of a build tool: a single API endpoint that will return the current time as a message. To keep the sample app as simple as possible we will use only a single 3rd-party dependency: FastAPI

Assuming you have created a fresh directory for the app, we’ll set up the local (virtual) environment using venv then install FastAPI.

$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip install fastapi
...
Installing collected packages: typing-extensions, starlette, pydantic, fastapi

You can confirm that the dependent packages were installed in your virtual environment like so (your versions might not match mine exactly):

(.venv) $ pip freeze
fastapi==0.65.1
pydantic==1.8.2
starlette==0.14.2
typing-extensions==3.10.0.0

Typically, you would save this to a `requirements.txt` file and ensure deterministic builds within a CI/CD pipeline; but let’s ignore this for now.

Next, create a single main.py file for the app:

# main.py
from datetime import datetime
from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
    return {"message": f"The time is {datetime.now()}"}

A lot is happening behind that single decorator, but generally speaking anytime the root endpoint of your server receives a GET request, it’ll return a JSON response with the current time.

Here we encounter our first issue: how to test the app locally. FastAPI is built on Starlette, an ASGI (*Asynchronous Server Gateway Interface*) framework. Typical WSGI servers such as Gunicorn won’t work with FastAPI. We instead need an ASGI server like Uvicorn or Daphne. We can take advantage of many optional FastAPI features by installing the fastapi[all] extra:

(.venv) $ pip install fastapi[all]
...
Installing collected packages: six, rx, promise, h11, graphql-core, click, websockets, watchgod, uvloop, uvicorn, urllib3, pyyaml, python-dotenv, MarkupSafe, idna, httptools, graphql-relay, dnspython, chardet, certifi, aniso8601, ujson, requests, python-multipart, orjson, jinja2, itsdangerous, graphene, email-validator, async-generator, async-exit-stack, aiofiles

Of course, this has drastically increased the number of packages required to support the app (an additional 32 beyond the initial 4):

(.venv) $ pip freeze
aiofiles==0.5.0
aniso8601==7.0.0
async-exit-stack==1.0.1
async-generator==1.10
certifi==2021.5.30
chardet==4.0.0
click==7.1.2
dnspython==2.1.0
email-validator==1.1.2
fastapi==0.65.1
graphene==2.1.8
graphql-core==2.3.2
graphql-relay==2.0.1
h11==0.12.0
httptools==0.1.2
idna==2.10
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==2.0.1
orjson==3.5.3
promise==2.3
pydantic==1.8.2
python-dotenv==0.17.1
python-multipart==0.0.5
PyYAML==5.4.1
requests==2.25.1
Rx==1.6.1
six==1.16.0
starlette==0.14.2
typing-extensions==3.10.0.0
ujson==4.0.2
urllib3==1.26.5
uvicorn==0.13.4
uvloop==0.15.2
watchgod==0.7
websockets==8.1

Now we only need to start the local server by specifying your application:

(.venv) $ uvicorn main:app

Finally, you can test with httpie (or curl) from another terminal:

$ http :8000
HTTP/1.1 200 OK
content-length: 52
content-type: application/json
date: Tue, 01 Jun 2021 17:23:53 GMT
server: uvicorn

{
    "message": "The time is 2021-06-01 13:23:54.419989"
}

Deterministic Builds

As mentioned above, the typical process for managing virtual environment builds is to freeze the pip libraries into a `requirements.txt` file: 36 pinned packages to worry about. Do we really care about the particular version of pydantic installed? Unlikely, so long as it is supported by the installed version of FastAPI and consistent throughout all environment builds. This responsibility should instead be delegated to the build tool.

pip-tools: Compile and Sync

Remember that part of the design was to minimize the number of 3rd-party packages needed by the application. Beyond FastAPI, the only additional requirements were due to a need for local server testing. Let’s assume for now that the production server setup is beyond the scope of the app, and that your local environment has a global ASGI server already configured for this endpoint. (Don’t worry; we’ll return to expand on this later.)

The app has a single dependency: FastAPI. Let’s define that in a new package file, requirements.in:

# requirements.in
fastapi

With a package file defining the packages (not necessarily the versions) the app requires, we can now use the `pip-compile` build tool from pip-tools to generate the pinned (read: locked) versions in a requirements.txt`:

(.venv) $ pip install pip-tools
(.venv) $ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
fastapi==0.65.1
    # via -r requirements.in
pydantic==1.8.2
    # via fastapi
starlette==0.14.2
    # via fastapi
typing-extensions==3.10.0.0
    # via pydantic

Compared to the pip freeze version above, we can see that the versions identified (latest at the time of writing) are identical, but we can also see *why* the build tool included each package: either from a defined requirement else as a dependency for another package. You can verify this with pipdeptree:

(.venv) $ pipdeptree --packages fastapi
fastapi==0.65.1
  - pydantic [required: >=1.6.2,<2.0.0,!=1.8.1,!=1.8,!=1.7.3,!=1.7.2,!=1.7.1,!=1.7, installed: 1.8.2]
    - typing-extensions [required: >=3.7.4.3, installed: 3.10.0.0]
  - starlette [required: ==0.14.2, installed: 0.14.2]

Running pip-sync will ensure your virtual environment contains only the specific pinned packages defined in the generated requirements.txt file. (It will not remove your local pip-tools package, however; that would just be rude.)

dev-requirements.in

pip-tools allows for “layered” requirements for environment-specific dependencies. Let’s keep the assumption that the production server setup is outside scope, but it’s time to revisit the need for a local ASGI server. In this case, we’ll create a second package file, dev-requirements.in:

# dev-requirements.in
-c requirements.txt
fastapi[all]

We constrain the local packages with the generated requirements.txt file, ensuring that the local environment cannot have a different version of the same production package:

(.venv) $ pip-compile dev-requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile dev-requirements.in
#
aiofiles==0.5.0
    # via fastapi
aniso8601==7.0.0
    # via graphene
async-exit-stack==1.0.1
    # via fastapi
async-generator==1.10
    # via fastapi
certifi==2021.5.30
    # via requests
chardet==4.0.0
    # via requests
click==7.1.2
    # via uvicorn
dnspython==2.1.0
    # via email-validator
email-validator==1.1.2
    # via fastapi
fastapi[all]==0.65.1
    # via
    #   -c requirements.txt
    #   -r dev-requirements.in
graphene==2.1.8
    # via fastapi
graphql-core==2.3.2
    # via
    #   graphene
    #   graphql-relay
graphql-relay==2.0.1
    # via graphene
h11==0.12.0
    # via uvicorn
httptools==0.1.2
    # via uvicorn
idna==2.10
    # via
    #   email-validator
    #   requests
itsdangerous==1.1.0
    # via fastapi
jinja2==2.11.3
    # via fastapi
markupsafe==2.0.1
    # via jinja2
orjson==3.5.3
    # via fastapi
promise==2.3
    # via
    #   graphql-core
    #   graphql-relay
pydantic==1.8.2
    # via
    #   -c requirements.txt
    #   fastapi
python-dotenv==0.17.1
    # via uvicorn
python-multipart==0.0.5
    # via fastapi
pyyaml==5.4.1
    # via
    #   fastapi
    #   uvicorn
requests==2.25.1
    # via fastapi
rx==1.6.1
    # via graphql-core
six==1.16.0
    # via
    #   graphene
    #   graphql-core
    #   graphql-relay
    #   python-multipart
starlette==0.14.2
    # via
    #   -c requirements.txt
    #   fastapi
typing-extensions==3.10.0.0
    # via
    #   -c requirements.txt
    #   pydantic
ujson==4.0.2
    # via fastapi
urllib3==1.26.5
    # via requests
uvicorn[standard]==0.13.4
    # via fastapi
uvloop==0.15.2
    # via uvicorn
watchgod==0.7
    # via uvicorn
websockets==8.1
    # via uvicorn

Once again, we have 36 packages all pinned the same as before, but generated in a dev-requirements.txt file by the build tool instead of by us. This is a huge benefit with nested dependencies. Consider the package `six`; in our dependency tree we see that different version requirements exist for each of the packages:

(.venv) $ pipdeptree --reverse --packages six
six==1.16.0
  - graphene==2.1.8 [requires: six>=1.10.0,<2]
  - graphql-core==2.3.2 [requires: six>=1.10.0]
    - graphene==2.1.8 [requires: graphql-core>=2.1,<3]
    - graphql-relay==2.0.1 [requires: graphql-core>=2.2,<3]
      - graphene==2.1.8 [requires: graphql-relay>=2,<3]
  - graphql-relay==2.0.1 [requires: six>=1.12]
    - graphene==2.1.8 [requires: graphql-relay>=2,<3]
  - promise==2.3 [requires: six]
    - graphql-core==2.3.2 [requires: promise>=2.3,<3]
      - graphene==2.1.8 [requires: graphql-core>=2.1,<3]
      - graphql-relay==2.0.1 [requires: graphql-core>=2.2,<3]
        - graphene==2.1.8 [requires: graphql-relay>=2,<3]
    - graphql-relay==2.0.1 [requires: promise>=2.2,<3]
      - graphene==2.1.8 [requires: graphql-relay>=2,<3]
  - python-multipart==0.0.5 [requires: six>=1.4.0]

Many applications quickly form dependency trees that can become a nightmare to traverse when attempting to update even a single dependency, let alone multiple packages. The build tool makes dependency management a trivial task: updating libraries as necessary per the requirements defined in the package file(s), ensuring deterministic builds in all environments with the generated lock file(s).

The JBS Quick Launch Lab

Free Qualified Assessment

Quantify what it will take to implement your next big idea!

Our assessment session will deliver tangible timelines, costs, high-level requirements, and recommend architectures that will work best. Let JBS prove to you and your team why over 24 years of experience matters.

Get Your Assessment