Distributing Python Modules - University of California, Berkeley

Distributing Python Modules

Greg Ward

April 14, 2001 E-mail: gward@

Abstract This document describes the Python Distribution Utilities ("Distutils") from the module developer's point-of-view, describing how to use the Distutils to make Python modules and extensions easily available to a wider audience with very little overhead for build/release/install mechanics.

Contents

1 Introduction

2

2 Concepts & Terminology

2

2.1 A simple example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.2 General Python terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.3 Distutils-specific terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Writing the Setup Script

5

3.1 Listing whole packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3.2 Listing individual modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.3 Describing extension modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Extension names and packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Extension source files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Preprocessor options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Library options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Other options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.4 Listing scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.5 Listing additional files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 Writing the Setup Configuration File

10

5 Creating a Source Distribution

12

5.1 Specifying the files to distribute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.2 Manifest-related options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6 Creating Built Distributions

14

6.1 Creating dumb built distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

6.2 Creating RPM packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

6.3 Creating Windows installers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

7 Reference

18

7.1 Installing modules: the install command family . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

install data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 install scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 7.2 Creating a source distribution: the sdist command . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1 Introduction

In the past, Python module developers have not had much infrastructure support for distributing modules, nor have Python users had much support for installing and maintaining third-party modules. With the introduction of the Python Distribution Utilities (Distutils for short) in Python 1.6, this situation should start to improve.

This document only covers using the Distutils to distribute your Python modules. Using the Distutils does not tie you to Python 1.6, though: the Distutils work just fine with Python 1.5.2, and it is reasonable (and expected to become commonplace) to expect users of Python 1.5.2 to download and install the Distutils separately before they can install your modules. Python 1.6 (or later) users, of course, won't have to add anything to their Python installation in order to use the Distutils to install third-party modules.

This document concentrates on the role of developer/distributor: if you're looking for information on installing Python modules, you should refer to the Installing Python Modules manual.

2 Concepts & Terminology

Using the Distutils is quite simple, both for module developers and for users/administrators installing third-party modules. As a developer, your responsibilities (apart from writing solid, well-documented and well-tested code, of course!) are:

? write a setup script (`setup.py' by convention)

? (optional) write a setup configuration file

? create a source distribution

? (optional) create one or more built (binary) distributions

Each of these tasks is covered in this document. Not all module developers have access to a multitude of platforms, so it's not always feasible to expect them to create a multitude of built distributions. It is hoped that a class of intermediaries, called packagers, will arise to address this need. Packagers will take source distributions released by module developers, build them on one or more platforms, and release the resulting built distributions. Thus, users on the most popular platforms will be able to install most popular Python module distributions in the most natural way for their platform, without having to run a single setup script or compile a line of code.

2.1 A simple example

The setup script is usually quite simple, although since it's written in Python, there are no arbitrary limits to what you can do with it.1 If all you want to do is distribute a module called foo, contained in a file `foo.py', then your setup script can be as little as this:

1But be careful about putting arbitrarily expensive operations in your setup script; unlike, say, Autoconf-style configure scripts, the setup script may be run multiple times in the course of building and installing your module distribution. If you need to insert potentially expensive processing steps into the Distutils chain, see section ?? on extending the Distutils.

2

2 Concepts & Terminology

from distutils.core import setup setup(name="foo",

version="1.0", py_modules=["foo"])

Some observations:

? most information that you supply to the Distutils is supplied as keyword arguments to the setup() function ? those keyword arguments fall into two categories: package meta-data (name, version number) and information

about what's in the package (a list of pure Python modules, in this case) ? modules are specified by module name, not filename (the same will hold true for packages and extensions) ? it's recommended that you supply a little more meta-data, in particular your name, email address and a URL for

the project (see section 3 for an example)

To create a source distribution for this module, you would create a setup script, `setup.py', containing the above code, and run:

python setup.py sdist

which will create an archive file (e.g., tarball on UNIX, ZIP file on Windows) containing your setup script, `setup.py', and your module, `foo.py'. The archive file will be named `Foo-1.0.tar.gz' (or `.zip'), and will unpack into a directory `Foo-1.0'. If an end-user wishes to install your foo module, all she has to do is download `Foo-1.0.tar.gz' (or `.zip'), unpack it, and--from the `Foo-1.0' directory--run

python setup.py install

which will ultimately copy `foo.py' to the appropriate directory for third-party modules in their Python installation. This simple example demonstrates some fundamental concepts of the Distutils: first, both developers and installers have the same basic user interface, i.e. the setup script. The difference is which Distutils commands they use: the sdist command is almost exclusively for module developers, while install is more often for installers (although most developers will want to install their own code occasionally). If you want to make things really easy for your users, you can create one or more built distributions for them. For instance, if you are running on a Windows machine, and want to make things easy for other Windows users, you can create an executable installer (the most appropriate type of built distribution for this platform) with the bdist wininst command. For example:

python setup.py bdist_wininst

will create an executable installer, `Foo-1.0.win32.exe', in the current directory. Currently (Distutils 0.9.2), the only other useful built distribution format is RPM, implemented by the bdist rpm command. For example, the following command will create an RPM file called `Foo-1.0.noarch.rpm':

python setup.py bdist_rpm

2.1 A simple example

3

(This uses the rpm command, so has to be run on an RPM-based system such as Red Hat Linux, SuSE Linux, or Mandrake Linux.) You can find out what distribution formats are available at any time by running

python setup.py bdist --help-formats

2.2 General Python terminology

If you're reading this document, you probably have a good idea of what modules, extensions, and so forth are. Nevertheless, just to be sure that everyone is operating from a common starting point, we offer the following glossary of common Python terms:

module the basic unit of code reusability in Python: a block of code imported by some other code. Three types of modules concern us here: pure Python modules, extension modules, and packages.

pure Python module a module written in Python and contained in a single `.py' file (and possibly associated `.pyc' and/or `.pyo' files). Sometimes referred to as a "pure module."

extension module a module written in the low-level language of the Python implementation: C/C++ for Python, Java for JPython. Typically contained in a single dynamically loadable pre-compiled file, e.g. a shared object (`.so') file for Python extensions on UNIX, a DLL (given the `.pyd' extension) for Python extensions on Windows, or a Java class file for JPython extensions. (Note that currently, the Distutils only handles C/C++ extensions for Python.)

package a module that contains other modules; typically contained in a directory in the filesystem and distinguished from other directories by the presence of a file ` init .py'.

root package the root of the hierarchy of packages. (This isn't really a package, since it doesn't have an ` init .py' file. But we have to call it something.) The vast majority of the standard library is in the root package, as are many small, standalone third-party modules that don't belong to a larger module collection. Unlike regular packages, modules in the root package can be found in many directories: in fact, every directory listed in sys.path can contribute modules to the root package.

2.3 Distutils-specific terminology

The following terms apply more specifically to the domain of distributing Python modules using the Distutils:

module distribution a collection of Python modules distributed together as a single downloadable resource and meant to be installed en masse. Examples of some well-known module distributions are Numeric Python, PyXML, PIL (the Python Imaging Library), or mxDateTime. (This would be called a package, except that term is already taken in the Python context: a single module distribution may contain zero, one, or many Python packages.)

pure module distribution a module distribution that contains only pure Python modules and packages. Sometimes referred to as a "pure distribution."

non-pure module distribution a module distribution that contains at least one extension module. Sometimes referred to as a "non-pure distribution."

distribution root the top-level directory of your source tree (or source distribution); the directory where `setup.py' exists and is run from

4

2 Concepts & Terminology

3 Writing the Setup Script

The setup script is the centre of all activity in building, distributing, and installing modules using the Distutils. The main purpose of the setup script is to describe your module distribution to the Distutils, so that the various commands that operate on your modules do the right thing. As we saw in section 2.1 above, the setup script consists mainly of a call to setup(), and most information supplied to the Distutils by the module developer is supplied as keyword arguments to setup(). Here's a slightly more involved example, which we'll follow for the next couple of sections: the Distutils' own setup script. (Keep in mind that although the Distutils are included with Python 1.6 and later, they also have an independent existence so that Python 1.5.2 users can use them to install other module distributions. The Distutils' own setup script, shown here, is used to install the package into Python 1.5.2.)

#!/usr/bin/env python

from distutils.core import setup

setup(name="Distutils", version="1.0", description="Python Distribution Utilities", author="Greg Ward", author_email="gward@", url="", packages=['distutils', 'mand'],

)

There are only two differences between this and the trivial one-file distribution presented in section 2.1: more metadata, and the specification of pure Python modules by package, rather than by module. This is important since the Distutils consist of a couple of dozen modules split into (so far) two packages; an explicit list of every module would be tedious to generate and difficult to maintain. Note that any pathnames (files or directories) supplied in the setup script should be written using the UNIX convention, i.e. slash-separated. The Distutils will take care of converting this platform-neutral representation into whatever is appropriate on your current platform before actually using the pathname. This makes your setup script portable across operating systems, which of course is one of the major goals of the Distutils. In this spirit, all pathnames in this document are slash-separated (MacOS programmers should keep in mind that the absence of a leading slash indicates a relative path, the opposite of the MacOS convention with colons). This, of course, only applies to pathnames given to Distutils functions. If you, for example, use standard python functions such as glob.glob or os.listdir to specify files, you should be careful to write portable code instead of hardcoding path separators:

glob.glob(os.path.join('mydir', 'subdir', '*.html')) os.listdir(os.path.join('mydir', 'subdir'))

3.1 Listing whole packages

The packages option tells the Distutils to process (build, distribute, install, etc.) all pure Python modules found in each package mentioned in the packages list. In order to do this, of course, there has to be a correspondence between package names and directories in the filesystem. The default correspondence is the most obvious one, i.e. package distutils is found in the directory `distutils' relative to the distribution root. Thus, when you say packages = ['foo'] in your setup script, you are promising that the Distutils will find a file `foo/ init .py' (which might be spelled differently on your system, but you get the idea) relative to the directory where your setup script lives. (If you break this promise, the Distutils will issue a warning but process the broken package anyways.)

5

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download