Python policy — различия между версиями

Текущая версия на 10:53, 16 марта 2020

Содержание

1 Tools
- 1.1 bdist_rpm and bdist_rpm5
- 1.2 Py2pack
2 Rules
3 Rpm macro
- 3.1 Deprecated Macros
4 pyc/pyo policy
- 4.1 egg-info
- 4.2 Automated setup
5 Example spec file
6 Rebuild policy
7 Policy in other distributions

Tools

bdist_rpm and bdist_rpm5

The fastest way to pack a Python module for ROSA is to use a standard bdist_rpm python class:

python setup.py bdist_rpm

Prior to rosa2019.1 and switch to rpm4, one should use bdist-rpm5 instead:

python setup.py bdist_rpm5

If everything goes fine, you will find ready-to-use SRPM inside the dist folder which contains spec file conforming to ROSA Python policy (and the rest of this document is just for your interest then).

If the tool fails to automatically detect values for some tags (license, description, etc.), it will set these fields to 'UNKNOWN' value.

Please look for 'UNKNOWN' words in the generated spec and fix them manually.

Note that bdist_rpm5 worked only for Python2, while bdist_rpm works fine for both Python2 and Python3. However, starting from rosa2019.1 platform for most modules there is no need to build Python2 versions anymore, only Python3 ones should be left.

Py2pack

Py2pack script can be used to generate RPM spec file from Python modules using Python Package Index (PyPI). Currently py2pack can only generate spec files for OpenSUSE and Fedora, but one may use generated spec as a basis for the ROSA one.

Rules

Naming

The name of a package with Python2 module 'foo' that provides an extension for Python2 library must be of the form python2-foo, with foo being lower case. For packages with Python3 modules, 'python3' prefix should be used - python3-foo. If the upstream name 'foo' already contains "python", that should be dropped from the name.

It is recommended to name the project with python module as python-foo and define python3-foo and python2-foo subpackages. Python2 subpackages are not required and should only be built if the module is used by software which is not ported to Python2.

Rationale: using a prefix is serving as a name space, and clearly shows how to find python modules.

Having everything in lower case is easier to read, and offers a consistent sorting.

Note: User-level applications written in Python do not need to follow these naming rules.

Build Requires

'python2' or 'python3' requirement is added automatically once the rpmbuild detects you are using python to build your package. If spec file is created using bdist_rpm, setuptools dependency is also added.
All other dependencies should be added manually.
Arch dependent packages (compiled extensions) should require (at least) python2-devel (python3-devel for Python3 packages).

Requires

Mandatory requirement python(abi) = VERSION is added automatically

Rpm macro

The following macros are available for Python packagers:

Macro (Python2)	Analogue for Python3	Description
%py2_ver	%py3_ver	Get Python version (as reported by invoking 'python -V')
%py2_incdir	%py3_incdir	Path to Python include files (necessary to compile C extensions)
%python2_sitelib	%python3_sitelib	Directory for the installation of third-party extensions which are platform-independent (written in pure Python)
%python2_sitearch	%python3_sitearch	Directory for the installation of third-party extensions which are platform-dependent (C extensions)
%py2_platlibdir	N/A yet	Platform-dependent directory for the installation for the standard library
%py2_purelibdir	N/A yet	Platform-independent directory for the installation for the standard library
%py2_prefix	%py3_prefix	Site-specific directory prefix where the platform independent Python files are installed (sys.prefix)
%py2_compile(O)	N/A yet	Create .pyc files, dropping existing ones, if any. If -O option is specified, the same is performed for .pyo files.

%py_(pure|plat)(site|lib)dir are intended to be used in order to place pure Python modules (that are architecture independent) in the same place on all hardware platforms, while putting compiled C extensions to arch-specific folders. For example, pure Python modules will go to /usr/lib/python on both x86 and x86-64, while compiled extensions will go to /usr/lib/python/ on x86 and /usr/lib64/python/ on x86-64. This allows packaging software consisting of pure Python modules only as noarch packages suitable for installing on all platforms.

For those familiar with Python distutils - %py_(pure|plat)(site|lib)dir correspond to results of distutils.sysconfig.get_python_lib() with appropriate parameter (plat_specific, standard _lib).

See http://archives.mandrivalinux.com/cooker/2006-01/msg01404.php for more discussions.

Deprecated Macros

Starting with rosa2019.1 platform, all macros with unversioned %py_ prefix are not allowed to be used - one should explicitly specify either %py2_ or %py3_.

The following macros are also available but marked as deprecated and not recommended:

Macro	Description
%py2_libdir	The same as %py2_purelibdir
%py2_sitedir	The same as %py2_puresitedir
%py2_platsitedir	%python2_sitearch}
%py2_puresitedir	The same as %python2_sitelib
%python2_version	The same as %py2_ver
%py_requires(d)	Insert build dependency on python; if '-d' option is specified then add build dependency on python-devel, as well. For now, it is recommended to explicitly specify BuildRequires: python and BuildRequires: python-devel.

pyc/pyo policy

Files should be compiled before rpm creation and included into it as they will otherwise be created if package is run as root and will not be removed on package removal.

Starting from rosa2019.1 platform, bytecompiled files for Python3 are created automatically, but not forget to include them to the package. For python3, all such files are pushed into __pycache__ folder so you may need to add the following line to your spec:

%{python3_sitelib}/__pycache_/*

egg-info

If egg-info files or directories are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime. So the %files section should contain something like this:

%{py_platsitedir}/%{name}-*.egg-info

Automated setup

If you use automated setup and invoke something like 'python setup.py install', you may ask python to record names of installed files and then use the recorded list in the %files section:

%install
...
python3 setup.py install --root=%{buildroot} --record=FILELIST
...

%files -f FILELIST
...

Note, however, that automatically created file lists are not completely reliable - they don't contain byte-compiled files and t the same time may contain duplicates - more particular, it can contain a directory followed by the whole directory contents, such as:

/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg-info
/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg-info/top_level.txt
/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg-info/PKG-INFO
...

However, for modern rpm it is enough to mention /usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg-info only. You can filter out extra records using grep:

cat FILELIST | grep -v "egg-info/" > NEW_FILELIST
mv NEW_FILELIST FILELIST

Example spec file

Here we only create subpackage for python3. If you also need python2 - please create the same subpackage with python2- prefix.

%define module	mymodule

Summary: 	Example module
Name: 	 	python-%{module}
Version: 	1.0
Release: 	1
Source:         %{module}-%{version}.tar.gz
URL:		http://mypage.org/mymodule
License:	Apache License
Group:		Development/Python
BuildArch:	noarch

%description
Example Python module

#---------------------------------------------------------------------------
%package     -n python3-%{module}
Summary: 	Example module
BuildRequires:	python3-devel
BuildRequires:	python3-setuptools

%description -n python3-%{module}
Example Python module

%files -n python3-%{module}
%{python3_sitelib}/*

#---------------------------------------------------------------------------

%prep
%setup -qn %{module}-%{version}

%build
%py3_build

%install
%py3_install

Rebuild policy

It is known that Python is not completely backward compatible (so if you update e.g. from 2.6 to 2.7, some modules may become broken; the more so for Python3 updates). Due to this 'python(abi)' dependency is set to particular Python version.

As a result, when Python itself is updated all dependent packages should be rebuilt to handle pick up the new version.

Policy in other distributions

@@ Строка 3: / Строка 3: @@
 == Tools ==
-=== bdist_rpm5 ===
+=== bdist_rpm and bdist_rpm5 ===
-The fastest way to pack a Python2 module for ROSA is to use bdist_rpm5 class extending standard [http://docs.python.org/distutils/builtdist.html#creating-rpm-packages bdist_rpm] and intended to create RPMv5 packages for ROSA.
+The fastest way to pack a Python module for ROSA is to use a standard [http://docs.python.org/distutils/builtdist.html#creating-rpm-packages bdist_rpm] python class:
-To use it, you should specify bdistr_rpm5 option for setup.py:
+<pre>
+python setup.py bdist_rpm
+</pre>
+Prior to rosa2019.1 and switch to rpm4, one should use bdist-rpm5 instead:
 <pre>
 python setup.py bdist_rpm5
@@ Строка 17: / Строка 21: @@
 Please look for 'UNKNOWN' words in the generated spec and fix them manually.
-bdist_rpm5 for Python3 does not exist yet.
+Note that bdist_rpm5 worked only for Python2, while bdist_rpm works fine for both Python2 and Python3. However, starting from rosa2019.1 platform for most modules there is no need to build Python2 versions anymore, only Python3 ones should be left.
 === Py2pack ===
@@ Строка 25: / Строка 29: @@
 === Naming ===
-The name of a package with Python2 module 'foo' that provides an extension for Python2 library must be of the form {{pkg|python-foo}}, with foo being lower case. For packages with Python3 modules, 'python3' prefix should be used - {{pkg|python3-foo}}. If the upstream name 'foo' already contains "python", that should be dropped from the name.
+The name of a package with Python2 module 'foo' that provides an extension for Python2 library must be of the form {{pkg|python2-foo}}, with foo being lower case. For packages with Python3 modules, 'python3' prefix should be used - {{pkg|python3-foo}}. If the upstream name 'foo' already contains "python", that should be dropped from the name.
+It is recommended to name the project with python module as {{pkg|python-foo}} and define {{pkg|python3-foo}} and {{pkg|python2-foo}} subpackages. Python2 subpackages are not required and should only be built if the module is used by software which is not ported to Python2.
 Rationale: using a prefix is serving as a name space, and clearly shows how to find python modules.
@@ Строка 38: / Строка 44: @@
 === Build Requires ===
-* 'python' or 'python3' requirement is added automatically once the rpmbuild detects you are using python to build your package. If spec file is created using bdist_rpm5, python-setuptool dependency is also added.
+* 'python2' or 'python3' requirement is added automatically once the rpmbuild detects you are using python to build your package. If spec file is created using bdist_rpm, setuptools dependency is also added.
 * All other dependencies should be added manually.
-* Arch dependent packages (compiled extensions) should require (at least) python-devel (python3-devel for Python3 packages).
+* Arch dependent packages (compiled extensions) should require (at least) python2-devel (python3-devel for Python3 packages).
 === Requires ===
-* Mandatory requirement python(abi) = VERSION is added automatically (by pythondeps.sh script)
+* Mandatory requirement python(abi) = VERSION is added automatically
 == Rpm macro ==
@@ Строка 51: / Строка 57: @@
 ! Macro (Python2) !! Analogue for Python3 !! Description
 |-
-| {{macro|%py_ver}} || {{macro|%py3_ver}} || Get Python version (as reported by invoking 'python -V')
+| {{macro|%py2_ver}} || {{macro|%py3_ver}} || Get Python version (as reported by invoking 'python -V')
 |-
-| {{macro|%py_incdir}} || {{macro|%py3_incdir}} || Path to Python include files (necessary to compile C extensions)
+| {{macro|%py2_incdir}} || {{macro|%py3_incdir}} || Path to Python include files (necessary to compile C extensions)
 |-
-| {{macro|%py_puresitedir}} || {{macro|%py3_puresitedir}} || Directory for the installation of third-party extensions which are platform-independent (written in pure Python)
+| {{macro|%python2_sitelib}} || {{macro|%python3_sitelib}} || Directory for the installation of third-party extensions which are platform-independent (written in pure Python)
 |-
-| {{macro|%py_platsitedir}} || {{macro|%py3_platsitedir}} || Directory for the installation of third-party extensions which are platform-dependent (C extensions)
+| {{macro|%python2_sitearch}} || {{macro|%python3_sitearch}} || Directory for the installation of third-party extensions which are platform-dependent (C extensions)
 |-
-| {{macro|%py_platlibdir}} || N/A yet || Platform-dependent directory for the installation for the standard library
+| {{macro|%py2_platlibdir}} || N/A yet || Platform-dependent directory for the installation for the standard library
 |-
-| {{macro|%py_purelibdir}} || N/A yet || Platform-independent directory for the installation for the standard library
+| {{macro|%py2_purelibdir}} || N/A yet || Platform-independent directory for the installation for the standard library
 |-
-| {{macro|%py_prefix}} || {{macro|%py3_prefix}} || Site-specific directory prefix where the platform independent Python files are installed (sys.prefix)
+| {{macro|%py2_prefix}} || {{macro|%py3_prefix}} || Site-specific directory prefix where the platform independent Python files are installed (sys.prefix)
 |-
-| {{macro|%py_compile(O)}} || N/A yet || Create .pyc files, dropping existing ones, if any. If -O option is specified, the same is performed for .pyo files.
+| {{macro|%py2_compile(O)}} || N/A yet || Create .pyc files, dropping existing ones, if any. If -O option is specified, the same is performed for .pyo files.
 |}
-'''%py_(pure|plat)(site|lib)dir''' are intended to be used instead of single '''%py_(site|lib)dir''' in order to place pure Python modules (that are architecture independent) in the same place on all hardware platforms, while putting compiled C extensions to arch-specific folders. For example, pure Python modules will go to /usr/lib/python on both x86 and x86-64, while compiled extensions will go to /usr/lib/python/ on x86 and /usr/lib64/python/ on x86-64. This allows packaging software consisting of pure Python modules only as noarch packages suitable for installing on all platforms.
+'''%py_(pure|plat)(site|lib)dir''' are intended to be used in order to place pure Python modules (that are architecture independent) in the same place on all hardware platforms, while putting compiled C extensions to arch-specific folders. For example, pure Python modules will go to /usr/lib/python on both x86 and x86-64, while compiled extensions will go to /usr/lib/python/ on x86 and /usr/lib64/python/ on x86-64. This allows packaging software consisting of pure Python modules only as noarch packages suitable for installing on all platforms.
 For those familiar with [http://docs.python.org/distutils/apiref.html#module-distutils.sysconfig Python distutils] - '''%py_(pure|plat)(site|lib)dir''' correspond to results of ''distutils.sysconfig.get_python_lib()'' with appropriate parameter (plat_specific, standard _lib).
@@ Строка 75: / Строка 81: @@
 === Deprecated Macros ===
+Starting with rosa2019.1 platform, all macros with unversioned %py_ prefix are not allowed to be used - one should explicitly specify either %py2_ or %py3_.
 The following macros are also available but marked as deprecated and not recommended:
 {|class='wikitable'
 ! Macro !! Description
 |-
-| {{macro|%py_libdir}} || The same as %py_purelibdir
+| {{macro|%py2_libdir}} || The same as %py2_purelibdir
 |-
-| {{macro|%py_sitedir}} || The same as %py_puresitedir
+| {{macro|%py2_sitedir}} || The same as %py2_puresitedir
 |-
-| {{macro|%python_sitearch}} || The same as %py_platsitedir
+| {{macro|%py2_platsitedir}} || The same as {{macro|%python2_sitearch}
 |-
-| {{macro|%python_sitelib}} || The same as %py_puresitedir
+| {{macro|%py2_puresitedir}} || The same as {{macro|%python2_sitelib}}
 |-
-| {{macro|%python_version}} || The same as %py_ver
+| {{macro|%python2_version}} || The same as %py2_ver
 |-
 | {{macro|%py_requires(d)}} || Insert build dependency on python; if '-d' option is specified then add build dependency on python-devel, as well.<br/> For now, it is recommended to explicitly specify BuildRequires: python and  BuildRequires: python-devel.
@@ Строка 93: / Строка 101: @@
 == pyc/pyo policy ==
-Files should be compiled before rpm creation and includes as they will otherwise be created if a package is run as root, and therefore created weird bug ( ie, module removed but still here, unowned files ).
+Files should be compiled before rpm creation and included into it as they will otherwise be created if package is run as root and will not be removed on package removal.
-To compile the .py moudles, add the following macros to the beginning of the spec:
+Starting from rosa2019.1 platform, bytecompiled files for Python3 are created automatically, but not forget to include them to the package. For python3, all such files are pushed into {{File|__pycache__}} folder so you may need to add the following line to your spec:
 <pre>
-%define python_compile_opt python -O -c "import compileall; compileall.compile_dir('.')"
+%{python3_sitelib}/__pycache_/*
-%define python_compile     python -c "import compileall; compileall.compile_dir('.')"
-</pre>
-Then in the %install section:
-<pre>
-dir -d %{buildroot}%{py_platsitedir}/%{name}
-%python_compile_opt
-%python_compile
-install *.pyc *.pyo %{buildroot}%{py_platsitedir}/%{name}
-</pre>
-This will create a directory for the software under the platform sitedir, compile the python modules, then copy them to the directory created before. You may use 'pushd directory' before that code and 'popd' after the code if the python modules are in a directory below the root of the tarball.
-And finally in the %files section:
-<pre>
-%{py_platsitedir}/%{name}/*.pyc
-%{py_platsitedir}/%{name}/*.pyo
 </pre>
@@ Строка 127: / Строка 116: @@
 === Automated setup ===
-If you use automated setup and invoke something like 'python setup.py install', you may set PYTHONDONTWRITEBYTECODE to zero or empty value to keep all files generated during installation. In addition, you may ask python to record names of installed files and then use the recorded list in the %files section:
+If you use automated setup and invoke something like 'python setup.py install', you may ask python to record names of installed files and then use the recorded list in the %files section:
 <pre>
 %install
 ...
-PYTHONDONTWRITEBYTECODE= python setup.py install --root=%{buildroot} --record=FILELIST
+python3 setup.py install --root=%{buildroot} --record=FILELIST
 ...
@@ Строка 139: / Строка 128: @@
 </pre>
-Note, however, that automatically created file lists may contain duplicates from rpm5 point of view - more particular, it can contain a directory followed by the whole directory contents, sich as:
+Note, however, that automatically created file lists are not completely reliable - they don't contain byte-compiled files and t the same time may contain duplicates - more particular, it can contain a directory followed by the whole directory contents, such as:
 <pre>
 /usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg-info
@@ Строка 147: / Строка 136: @@
 </pre>
-However, for modern rpm5 it is enough to mention '''/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg-info''' only. You can filter out extra records using {{cmd|grep}}:
+However, for modern rpm it is enough to mention '''/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg-info''' only. You can filter out extra records using {{cmd|grep}}:
 <pre>
 cat FILELIST | grep -v "egg-info/" > NEW_FILELIST
@@ Строка 154: / Строка 143: @@
 == Example spec file ==
+Here we only create subpackage for python3. If you also need python2 - please create the same subpackage with python2- prefix.
 <pre>
 %define module	mymodule
@@ Строка 165: / Строка 156: @@
 License:	Apache License
 Group:		Development/Python
-BuildRequires:	python-devel
 BuildArch:	noarch
 %description
 Example Python module
+#---------------------------------------------------------------------------
+%package     -n python3-%{module}
+Summary: 	Example module
+BuildRequires:	python3-devel
+BuildRequires:	python3-setuptools
+%description -n python3-%{module}
+Example Python module
+%files -n python3-%{module}
+%{python3_sitelib}/*
+#---------------------------------------------------------------------------
 %prep
-%setup -q
+%setup -qn %{module}-%{version}
-%install
+%build
-PYTHONDONTWRITEBYTECODE=  python setup.py install --root=%{buildroot} --record=FILELIST
+%py3_build
-%files -f FILELIST
+%install
-%doc *.txt
+%py3_install
 </pre>
 == Rebuild policy  ==
-It is known that Python is not completely backward compatible (so if you update e.g. from 2.6 to 2.7, some modules may become broken). Due to this 'python(abi)' dependency is set to particular Python version.
+It is known that Python is not completely backward compatible (so if you update e.g. from 2.6 to 2.7, some modules may become broken; the more so for Python3 updates). Due to this 'python(abi)' dependency is set to particular Python version.
 As a result, when Python itself is updated all dependent packages should be rebuilt to handle pick up the new version.
@@ Строка 190: / Строка 194: @@
 * [http://www.debian.org/doc/packaging-manuals/python-policy/ Debian python packaging policy]
 * [http://en.opensuse.org/openSUSE:Packaging_Python OpenSUSE python packaging policy]
-<hr>
-{{Примечание|This Policy is based on the [http://wiki.mandriva.com/en/Development/Tasks/Packaging/Policies/Python Mandriva Python Packaging Policy].}}