Can packaging improve Django deployments?

DjangoCon Europe 2018

Markus Zapke-Gründemann

About me

How can
packaging
Django projects
make deployments
easier, faster
and more reliable?

Key topics

  1. The quest
  2. Packaging a Django project
  3. Installing a packaged Django project
  4. Packaging tools comparison
  5. Summary

My quest for
reproducible
and deterministic
deployments

Packaging
a Django project

Project structure I

              
  myproject
  ├── manage.py
  ├── MANIFEST.in # NEW
  ├── myproject
  │   ├── __init__.py
  │   ├── apps # NEW
  │   │   ├── __init__.py
  │   │   └── myapp
  │   │       ├── __init__.py
  │   │       ├── apps.py
  │   │       └── ...
  │   └── config # NEW
  │       ├── __init__.py
  │       ├── settings.py
  │       ├── urls.py
  │       └── wsgi.py
  ├── setup.cfg # NEW
  └── setup.py # NEW
              
            

manage.py

              
 1 │ #!/usr/bin/env python
 2 │ import os
 3 │ import sys
 4 │
 5 │ if __name__ == "__main__":
 6 │     os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.config.settings")
 7 │     try:
 8 │         from django.core.management import execute_from_command_line
 9 │     except ImportError as exc:
10 │         raise ImportError(
11 │             "Couldn't import Django. Are you sure it's installed and "
12 │             "available on your PYTHONPATH environment variable? Did you "
13 │             "forget to activate a virtual environment?"
14 │         ) from exc
15 │     execute_from_command_line(sys.argv)
              
            

settings.py I

              
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.staticfiles',
    'myproject.apps.myapp.apps.MyappConfig',
]

ROOT_URLCONF = 'myproject.config.urls'

WSGI_APPLICATION = 'myproject.config.wsgi.application'
              
            

Project structure II

              
  myproject
  ├── manage.py
  ├── MANIFEST.in
  ├── myproject
  │   ├── __init__.py
  │   ├── apps
  │   │   ├── __init__.py
  │   │   └── myapp
  │   ├── config
  │   ├── locale # NEW
  │   ├── static # NEW
  │   └── templates # NEW
  ├── setup.cfg
  └── setup.py
              
            

settings.py II

              
LOCALE_PATHS = (
    os.path.join(BASE_DIR, 'locale'),
)
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
            ],
        },
    },
]
              
            

requirements.txt
vs.
setup.py & setup.cfg

"Old" setup.py

              
import os
import re
from codecs import open
from setuptools import find_packages, setup

BASE_DIR = os.path.dirname(os.path.abspath(__file__))

def read(*paths):
    """Build a file path from *paths and return the contents."""
    with open(os.path.join(*paths), 'r', 'utf-8') as f:
        return f.read()

with open('myproject/__init__.py') as f:
    contents = f.read()
    metadata = dict(re.findall(r'__([a-z]+)__\s*=\s*[\'"]([^\'"]*)[\'"]', contents))
    metadata['description'] = re.search(r'^"{3}(.*)"{3}', contents, re.DOTALL).group(1)

setup(
    name='myproject',
    version=metadata['version'],
    description=metadata['description'],
    long_description=read(BASE_DIR, 'README.rst'),
    author=metadata['author'],
    author_email=metadata['email'],
    include_package_data=True,
    install_requires=['arrow', 'bleach', 'Django', 'django-allauth', 'gunicorn', 'psycopg2', 'pytz', 'wagtail'],
    license=metadata['license'],
    url=metadata['url'],
    packages=find_packages(exclude=['tests*']),
    zip_safe=False,
    classifiers=[
        'Development Status :: 5 - Production/Stable',
        'Environment :: Web Environment',
        'Framework :: Django',
        'Programming Language :: Python',
    ],
)
              
            

setup.cfg I

              
[metadata]
name = myproject
version = 1.2.0
description = Lorem ipsum
long_description = file: README.rst
author = My Company
author_email = development@example.com
license = Other/Proprietary License
classifiers =
    Development Status :: 5 - Production/Stable
    Environment :: Web Environment
    Framework :: Django
    Programming Language :: Python
              
            
Configuring setup() using setup.cfg files

setup.cfg II

              
[options]
zip_safe = False
include_package_data = True
packages = myproject
install_requires =
    arrow==0.12.1
    bleach==2.1.3
    Django==2.0.5
    django-allauth==0.36.0
    gunicorn==19.8.1
    psycopg2==2.7.4
    pytz==2018.4
    wagtail==2.0.1
python_requires = >=3.6,<3.7

[bdist_wheel]
universal = 1
              
            

"New" setup.py

              
from setuptools import setup

setup()
              
            

bumpversion

MANIFEST.in

              
include *.rst
graft myproject
prune myproject/media
prune myproject/static_root
exclude *.yml
exclude manage.py
global-exclude *.py[co]
global-exclude __pycache__
              
            

check-manifest

What about manage.py?

              
[options.entry_points]
console_scripts =
    site-admin = django.core.management:execute_from_command_line
              
            

Building the package

              
                python setup.py bdist_wheel
              
              
                ls dist
              
              
                myproject-1.2.0-py2.py3-none-any.whl
              
            

Installing a packaged Django project

pip-tools

pip-tools commands

              
                pip-compile --output-file constraints.txt
              
              
                pip-compile --generate-hashes --output-file constraints.txt
              
              
                pip-sync constraints.txt
              
            

constraints.txt

              
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file constraints.txt setup.py
#
arrow==0.12.1
beautifulsoup4==4.6.0     # via wagtail
bleach==2.1.3
defusedxml==0.5.0         # via python3-openid
django-allauth==0.36.0
django-modelcluster==4.1  # via wagtail
django-taggit==0.22.2     # via wagtail
django-treebeard==4.3     # via wagtail
django==2.0.5             # via django-allauth, django-treebeard, wagtail
djangorestframework==3.8.2  # via wagtail
draftjs-exporter==2.0.0   # via wagtail
gunicorn==19.8.1
html5lib==0.999999999     # via bleach, wagtail
oauthlib==2.1.0           # via requests-oauthlib
olefile==0.45.1           # via pillow
pillow==4.0.0             # via wagtail
psycopg2==2.7.4
python-dateutil==2.7.3    # via arrow
python3-openid==3.1.0     # via django-allauth
pytz==2018.4
requests-oauthlib==0.8.0  # via django-allauth
requests==2.13.0          # via django-allauth, requests-oauthlib, wagtail
six==1.11.0               # via bleach, html5lib, python-dateutil
unidecode==0.4.21         # via wagtail
wagtail==2.0.1
webencodings==0.5.1       # via html5lib
willow==1.1               # via wagtail
              
            

Installing for development

              
                python -m pip install -c constraints.txt -e .
              
            

Installing development tools

              
                [options.extras_require]
                dev =
                    bumpversion
                    django-debug-toolbar
                    flake8
              
              
                python -m pip install -c constraints.txt -e .[dev]
              
            

How to serve the package?

  • From the file system
  • Using any HTTP server
  • Using devpi

Installing on any server

              
                python -m pip install -c constraints.txt myproject==1.2.0
              
              
                python -m pip install -c constraints.txt \
                    -f /path/to/wheel myproject==1.2.0
              
              
                python -m pip install -c constraints.txt \
                    --extra-index-url https://example.com myproject==1.2.0
              
            

How to change settings?

Running a WSGI server

              
                gunicorn myproject.config.wsgi
              
            

Packaging tools comparison

  • Use npm/yarn for JavaScript dependency management
  • Python packages are the lowest common denominator
  • conda can be used instead of pip to install a packaged Django project
  • Use tools like pex or snapcraft
  • Use platform package managers if necessary
  • Use Docker if necessary

Summary

  • Hosting solution independent
  • Uses tools you already know and use to install your dependencies - prevents NIH syndrome
  • Improves deployment to many servers
  • Same release is used everywhere: Dev, CI, Staging, Production
  • Easy rollback
  • A built distribution requires no build step
  • Avoids to install tools like git, gcc, gettext, Node.js etc.

Thank you!

Resources

slides.keimlink.de

keimlink

@keimlink

markus@keimlink.de

This work is licensed under a Creative Commons Attribution 4.0 International License

Built with cookiecutter-reveal.js, reveal.js, reveal.js-menu, Font Awesome and devicons.