Build python pyproject package #1146

Closed
opened 2023-09-05 19:11:26 +02:00 by danmur97 · 2 comments
danmur97 commented 2023-09-05 19:11:26 +02:00 (Migrated from github.com)

Problem to solve

nixpkgs have functions that handle the construction of python packages and python environments i.e. buildPythonPackage, buildEnv. All the python packages from nixpkgs are built from these helper functions.

The core problems by using makePythonPypiEnvironment are:

  • is limited to only pypi dependencies (cannot include local deps)
  • does not safely integrate with other environments.
  • since packages are built using pypi and poetry they relay on packaged binaries bundled on the wheels (that can be a problem since binary dependencies are not handled entirely by nix)

It is true that all of this problems can be overtaken and "fixed" by modifying PATH and other env setup but this makes that:

  • using a python package that depends on binaries is not as straightforward as using the nixpkgs version of the package
  • integrating pypi-environments (e.g. dev and runtime) with local deps and nixpkgs python packages results in posible package version collisions. With nixpkgs buildEnv this can be detected using ignoreCollisions = false;.

Intended users

Python developers

Permissions and Security

Proposal

  • The idea is to enable the building of python pkgs and environments throw the nixpkgs standard functions.
  • Also is desirable to adjust makePythonPypiEnvironment to be usable by this new python builder

Test plan

What does success look like, and how can we measure that?

/label ~"type::feature"

### Problem to solve `nixpkgs` have functions that handle the construction of python packages and python environments i.e. `buildPythonPackage`, `buildEnv`. All the python packages from `nixpkgs` are built from these helper functions. The core problems by using `makePythonPypiEnvironment` are: - is limited to only pypi dependencies (cannot include local deps) - does not safely integrate with other environments. - since packages are built using `pypi` and `poetry` they relay on packaged binaries bundled on the wheels (that can be a problem since binary dependencies are not handled entirely by nix) It is true that all of this problems can be overtaken and "fixed" by modifying PATH and other env setup but this makes that: - using a python package that depends on binaries is not as straightforward as using the `nixpkgs` version of the package - integrating pypi-environments (e.g. dev and runtime) with local deps and `nixpkgs` python packages results in posible package version collisions. With `nixpkgs` buildEnv this can be detected using `ignoreCollisions = false;`. ### Intended users Python developers ### Permissions and Security ### Proposal - The idea is to enable the building of python pkgs and environments throw the `nixpkgs` standard functions. - Also is desirable to adjust `makePythonPypiEnvironment` to be usable by this new python builder ### Test plan ### What does success look like, and how can we measure that? ### Links / references /label ~"type::feature"
dsalaza4 commented 2023-09-05 19:29:48 +02:00 (Migrated from github.com)

Some thoughts regarding this:

  1. Provided packages for makePythonPypiEnvironment should be fetched from PyPi instead of relying on the ones built on nixpkgs in order to avoid compilations and having to deal with several nixpkgs versions.
  2. If local dependencies are an actual necessity, let's try to reuse the fetch logic used in (1) to fetch them.
Some thoughts regarding this: 1. Provided packages for `makePythonPypiEnvironment` should be fetched from PyPi instead of relying on the ones built on nixpkgs in order to avoid compilations and having to deal with several nixpkgs versions. 2. If local dependencies are an actual necessity, let's try to reuse the fetch logic used in (1) to fetch them.
danmur97 commented 2023-09-19 18:38:41 +02:00 (Migrated from github.com)

integration support for pypi will be handled on this issue: https://github.com/fluidattacks/makes/issues/1161

integration support for pypi will be handled on this issue: https://github.com/fluidattacks/makes/issues/1161
This discussion has been locked. Commenting is limited to contributors.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
justin/makes#1146
No description provided.