Concepts

Goals

devpi goals are to provide a fast and reliable package cache to pypi as well as a mechanism to share packages, in various states of development, amongst users and developers, prior to pushing to the outside world.

This implies that users can:

  • Work unaffected if PyPI fails
  • store “closed source” packages internally, that can be accessed like any other packages as if they were residing on PyPI.

With this understanding, let’s go over a short overview of the devpi-server and devpi-client.

Overview

The devpi consists of two parts:

  • A devpi server responsible for providing a PyPI index cache and to manager sub-indices. The devpi server is compatible with most python installer tools.
  • The devpi client provides the user interface (command line) to interface with the devpi server.

The devpi server has a default, read-only, index name /root/pypi. When a package is not found in the index hierachy, it will will query pypi in a attempt to locate it. If located, the devpi server will cache all versions of that package, and periodically check the change logs to determine if new versions have been in released, in which case, the new package versions will be automatically downloaded into the server cache.

The diagram depicts the topology of a master server as well as their index structures:

../_images/devpi-topology.png

The Devpi Server

The devpi-server consist of a single process running on a system. The system can be local to your machine or on a centralized server. The server, by default, normally listens to port 3141 (PI = 3.141592653589793)

By default, the server has one index only:

  • /root/pypi is a special index which acts a PyPI cache. This index is read only, that is, no package can be uploaded to that index.

Access Control

To be documented

The Devpi Client

To be documented

Indexes

  • An index is a container or repository where you can store packages, test results and documentation.

  • An index derives from one or multiple base indices:

    For instance, imagine Emilie creates her own production index (called /emilie/prod)) and decide to have a developement index (/emilie/dev) which derives from her production index (a base). The latter is used to upload temporary packages currently under development.

    The structure could look something like this:

    $ devpi getjson http://localhost:3141/emilie
    {
        "result": {
            "email": "edoe@mydomain.net",
            "indices": {
                "dev": {
                    "bases": [
                        "/emilie/prod"
                    ],
                    "type": "stage",
                    "volatile": true
                },
                "prod": {
                    "bases": [],
                    "type": "stage",
                    "volatile": false
                }
            },
            "username": "emilie"
        },
        "status": 200,
        "type": "userconfig"
    }
    
    note:

    Important note about inheritance:

    When inheriting from an index, the packages of that index are not copied into the child index, making it susceptible to changes in the parent index. When looking for package foo, the devpi-server will traverse the inheritance tree if this package is not found in the current index.

  • A index is volatile if packages can be overridden with the same version (which would make sense for a development index). Otherwise, attempting to upload [2] a package with the same version would result in an error. See the Non Volatile Indexes for rules specific to this index type.

  • A user can have/create as many indices as he/she wants.

  • A user can use the index from another user as the base for one of her/his index:

    Suppose that Sophie has the same index hierarchy as Emilie (/sophie/prod -> /sophie/dev) but wants to experiment with a package Emilie is currently working on and stored in /emilie/dev, Sophie could create a sub index /sophie/fiddle, using both her and Emilie’s dev index [3], which would look like:

    $ devpi getjson http://localhost:3141/sophie
    {
        "result": {
            "email": "sober@mydomain.net",
            "indices": {
                "dev": {
                    "bases": [
                        "/sophie/prod"
                    ],
                    "type": "stage",
                    "volatile": true
                },
                "fiddle": {
                    "bases": [
                        "/sophie/dev",
                        "/emilie/dev"
                    ],
                    "type": "stage",
                    "volatile": true
                },
                "prod": {
                    "bases": [],
                    "type": "stage",
                    "volatile": false
                }
            },
            "username": "sophie"
        },
        "status": 200,
        "type": "userconfig"
    }
    

Non Volatile Indexes

As introduced earlier, a volatile index is an index that can be modified/deleted at will by its owner.

A non volatile index is an index that can not be modified in a destructive manner. This implies that:

  • A non volatile index can not be deleted.
  • If a project is created, it can not be deleted.
  • If a version is uploaded or pushed, it can not be be removed or overriden.

Furthermore, a non volatile index should not use a volatile index as one of its bases.

Non volatile indexes should be used as common package repositories between users, either for staging or production.

Mirror Indexes

These indexes mirror externally stored packages. By default root/pypi is such an index, which mirrors https://pypi.python.org/simple.

You can’t upload or push any packages to mirror indexes. They update themselves whenever they are used. For example when you try to install a package via pip.

By default the info for a package is cached for 30 minutes, after that the original is queried again. This can be adjusted per mirror index.

Package releases are downloaded on demand from the original location and cached indefinitely from then on.

Mirror indexes can’t have bases from which they inherit. They are commonly used as a base in regular indexes though.

The data produced by exporting the server state doesn’t include mirrored releases, only the settings of the mirror index.

The default settings of root/pypi look like this:

$ devpi index root/pypi
http://localhost:3141/root/pypi:
  type=mirror
  volatile=False
  custom_data=
  mirror_cache_expiry=1800
  mirror_url=https://pypi.python.org/simple/
  mirror_web_url_fmt=https://pypi.python.org/pypi/{name}
  title=PyPI

Server End Points

Footnotes

[1]this is true, when you run devpi and the devpi server on you local machine.
[2]Uploading refers to storing a package in an index whereas pushing refers to transferring a package from one index to another. In the example above, Emilie would upload package foo to /emilie/dev and later on push the package from /emilie/dev to /emilie/prod.
[3]

A preferred option in this case would be to temporarily modify the base of sophie’s dev index:

devpi index -m dev bases=/sophie/prod,/emilie/dev volatile=True