Quickstart: permanent install on server/laptop

This document walks you through setting up your own devpi-server instance, controlling it with supervisor on a UNIX-like system or launchd on Mac OS X and (optionally) serving it through nginx on a UNIX-like system. It also shows how to create a first user and index.

Note that the the pypi-mirroring quickstart already discusses the devpi-server --start|--log|--stop background-server control options which you might use to integrate with existing init.d or similar infrastructure.

Warning

If you intend to expose your server to the internet, you should read about securing it properly.

Installing devpi-server

Install or upgrade devpi-server:

pip install -U devpi-server
# if you want the web interface
pip install -U devpi-web

And let’s check the version:

$ devpi-server --version
4.0.0

generating example config files for supervisor/nginx/cron/launchd

Use devpi-server --gen-config to write out several example config files into the gen-config directory (relative to where you invoke it from). The files come with some defaults to integrate with your own permanent hosting infrastructure. When --gen-config is present no server is started and no network request is performed because the sole purpose is to help you get started with properly running and deploying a devpi-server in your company or on your laptop.

If you have improvements or additions to the example configs please share them with a Merge/Pull Request on the configuration template files and –gen-config implemenation along with a little doc section.

Here is an example run:

$ devpi-server --port 4040 --serverdir ~/mydevpiserver --gen-config
wrote gen-config/supervisor-devpi.conf
wrote gen-config/nginx-devpi.conf
wrote gen-config/crontab
wrote gen-config/net.devpi.plist
wrote gen-config/devpi.service

Below we look at the config files in a bit of detail.

nginx-devpi.conf: nginx site config

The nginx site configuration as generated by the above example run:

$ cat gen-config/nginx-devpi.conf
server {
    server_name localhost $hostname "";
    listen 80;
    gzip             on;
    gzip_min_length  2000;
    gzip_proxied     any;
    gzip_types       text/html application/json;

    proxy_read_timeout 60s;
    client_max_body_size 64M;

    # set to where your devpi-server state is on the filesystem
    root /tmp/home/mydevpiserver;

    # try serving static files directly
    location ~ /\+f/ {
        # workaround to pass non-GET/HEAD requests through to the named location below
        error_page 418 = @proxy_to_app;
        if ($request_method !~ (GET)|(HEAD)) {
            return 418;
        }

        expires max;
        try_files /+files$uri @proxy_to_app;
    }
    # try serving docs directly
    location ~ /\+doc/ {
        try_files $uri @proxy_to_app;
    }
    location / {
        # workaround to pass all requests to / through to the named location below
        error_page 418 = @proxy_to_app;
        return 418;
    }
    location @proxy_to_app {
        proxy_pass http://localhost:4040;
        proxy_set_header X-outside-url $scheme://$host:$server_port;
        proxy_set_header X-Real-IP $remote_addr;
        expires -1;  # no-cache
    }
}

The devpi specific settings are found in the location directives where we first try to serve static files through nginx before deferring to the devpi-server server instance. The X-outside URL settings drive the devpi-server to generate correct URLs because it might otherwise assume it is exposed at http://localhost:4040 and generate URLs accordingly.

supervisor site configuration

Let’s look at the supervisor configuration files. The supervisor-devpi.conf one presumes you have a master supervisor configuration on your system and and just drop the include config file into a directory like /etc/supervisor/conf.d/ on Ubuntu 12.04. Here are the contents:

$ cat gen-config/supervisor-devpi.conf

[program:devpi-server]
command=/home/hpk/venv/0/bin/devpi-server --port 4040 --serverdir /tmp/home/mydevpiserver
user = hpk
priority = 999
startsecs = 5
redirect_stderr = True
autostart = True

If supervisord is running on your system you may, after copying the config file to the supervisor conf.d directory, reload supervisord:

configuration directory

crontab / start at bootup

If you want to have things running at system startup and you are using a standard cron, a modified copy of your user crontab has been amended which you may inspect:

$ cat gen-config/crontab
@reboot /home/hpk/venv/0/bin/devpi-server --start --port 4040 --serverdir /tmp/home/mydevpiserver

and install with:

crontab TARGETDIR/etc/crontab

launchd / start at bootup (on Mac OS X)

Although cron can be used on Mac OS X, the preferred service manager on Mac OS X is launchd. With launchd, you will be able to use the launchctl command to start, stop, and monitor your server like any other Mac OS X service. An example property list file for your launch agent has been generated for you:

$ cat gen-config/net.devpi.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>net.devpi</string>
    <key>ProgramArguments</key>
    <array>
            <string>/home/hpk/venv/0/bin/devpi-server</string>
            <string>--port</string>
            <string>4040</string>
            <string>--serverdir</string>
            <string>/tmp/home/mydevpiserver</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

You can install this agent for your user account (recommended) by running:

mkdir -p ~/Library/LaunchAgents
cp gen-config/net.devpi.plist ~/Library/LaunchAgents

You can also install to any of the system locations listed in the FILES section of launchd(8). However, this isn’t recommended unless you know what you’re doing.

devpi should now launch upon the next reboot. You can tell launchd to launch it right now by running the following:

launchctl load ~/Library/LaunchAgents/net.devpi.plist
launchctl start net.devpi

The argument passed to launchctl start corresponds to the Label keyword in the plist file, but also matches the file name by convention.

To manually stop the devpi server started by launchd:

launchctl stop net.devpi

Please see the man pages launchd(8), launchd.plist(5), and launchctl(1) on your system for more information on launchd.

Installing devpi server and client

When started afresh, devpi-server will not contain any users or indexes except for the root user and the root/pypi index (see using root/pypi index) which represents and caches https://pypi.python.org packages. Let’s start a server for the purposes of this tutorial in the background:

$ devpi-server --port 4040 --start
2016-05-13 17:49:39,277 INFO  NOCTX Loading node info from /tmp/home/.devpi/server/.nodeinfo
2016-05-13 17:49:39,278 INFO  NOCTX generated uuid: 4078a2f1449d45d999ee4eedbdeb0171
2016-05-13 17:49:39,278 INFO  NOCTX wrote nodeinfo to: /tmp/home/.devpi/server/.nodeinfo
2016-05-13 17:49:39,279 INFO  NOCTX DB: Creating schema
2016-05-13 17:49:39,313 INFO  [Wtx-1] setting password for user u'root'
2016-05-13 17:49:39,313 INFO  [Wtx-1] created user u'root' with email None
2016-05-13 17:49:39,313 INFO  [Wtx-1] created root user
2016-05-13 17:49:39,313 INFO  [Wtx-1] created root/pypi index
2016-05-13 17:49:39,324 INFO  [Wtx-1] fswriter0: committed: keys: u'.config',u'root/.config'
starting background devpi-server at http://localhost:4040
/tmp/home/.devpi/server/.xproc/devpi-server$ /home/hpk/venv/0/bin/devpi-server --port 4040
process u'devpi-server' started pid=23936
devpi-server process startup detected
logfile is at /tmp/home/.devpi/server/.xproc/devpi-server/xprocess.log

The --start option will run a server in the background which you can further control with the “background server options”, see output of devpi-server -h at the end.

The server will listen on http://localhost:4040 and also serve the devpi web interface if you have devpi-web installed.

In order to manage users and indices on our fresh server let’s also install the devpi-client package:

$ pip install -U --pre -q devpi-client

You can install this client software on different hosts if you configured nginx.

Connecting to the server

If you configured nginx, you can use the server_name variable content for connecting to the server, instead of the localhost. For purposes of this tutorial, we use the URL http://localhost:4040 of the background tutorial server we started above:

$ devpi use http://localhost:4040
using server: http://localhost:4040/ (not logged in)
no current index: type 'devpi use -l' to discover indices
~/.pydistutils.cfg     : http://localhost:4040/alice/dev/+simple/
~/.pip/pip.conf        : http://localhost:4040/alice/dev/+simple/
~/.buildout/default.cfg: http://localhost:4040/alice/dev/+simple/
always-set-cfg: no

At this point we are not connected to any index, just to the root server. And we are not logged in.

setting the root password

The first thing to do is to set a password for the root user. For that we first need to login:

$ devpi login root --password ''
logged in 'root', credentials valid for 10.00 hours

and can then change it:

$ devpi user -m root password=123
/root changing password: 123
user modified: root

Let’s verify we don’t have any other users:

$ devpi user -l
root

The root user can modify any index and any user configuration. As we don’t plan to work further with the root user, we can log off:

$ devpi logoff
login information deleted

Registering a new user

Let’s register ourselves a new regular non-root user:

$ devpi user -c alice password=456  email=alice@example.com
user created: alice

and then login:

$ devpi login alice --password=456
logged in 'alice', credentials valid for 10.00 hours

Alice can now create her new dev index:

$ devpi index -c dev bases=root/pypi
http://localhost:4040/alice/dev:
  type=stage
  bases=root/pypi
  volatile=True
  acl_upload=alice
  mirror_whitelist=
  pypi_whitelist=

and use it

$ devpi use alice/dev
current devpi index: http://localhost:4040/alice/dev (logged in as alice)
~/.pydistutils.cfg     : http://localhost:4040/alice/dev/+simple/
~/.pip/pip.conf        : http://localhost:4040/alice/dev/+simple/
~/.buildout/default.cfg: http://localhost:4040/alice/dev/+simple/
always-set-cfg: no

Our alice/dev index derives from root/pypi which makes all pypi.python.org releases available.

automatically setting pip/easy_install config files

You can cause devpi to set $HOME configuration files which will cause pip and easy_install to use our in-use index server:

$ devpi use --set-cfg alice/dev
current devpi index: http://localhost:4040/alice/dev (logged in as alice)
~/.pydistutils.cfg     : http://localhost:4040/alice/dev/+simple/
~/.pip/pip.conf        : http://localhost:4040/alice/dev/+simple/
~/.buildout/default.cfg: http://localhost:4040/alice/dev/+simple/
always-set-cfg: no

This will modify or create common configuration files in your home directory so that subsequent pip or easy_install invocations will work against the user/indexname index. You can configure devpi to perform this configuration modification:

$ devpi use --always-set-cfg=yes
current devpi index: http://localhost:4040/alice/dev (logged in as alice)
~/.pydistutils.cfg     : http://localhost:4040/alice/dev/+simple/
~/.pip/pip.conf        : http://localhost:4040/alice/dev/+simple/
~/.buildout/default.cfg: http://localhost:4040/alice/dev/+simple/
always-set-cfg: yes

This will imply --set-cfg on all subsequent devpi use ... operations.

Installing, uploading, testing and releasing

You may now continue with install, test and release activities as described in the release process quickstart.

Stopping the server

Let’s not forget to stop our background tutorial server:

$ devpi-server --stop
2016-05-13 17:49:47,785 INFO  NOCTX Loading node info from /tmp/home/.devpi/server/.nodeinfo
2016-05-13 17:49:47,786 INFO  NOCTX wrote nodeinfo to: /tmp/home/.devpi/server/.nodeinfo
killed server pid=23936