Makes breaks when provenances are too large #1128

Closed
opened 2023-08-10 05:53:08 +02:00 by dsalaza4 · 0 comments
dsalaza4 commented 2023-08-10 05:53:08 +02:00 (Migrated from github.com)

When running jobs that have huge provenance files, the CLI function _nix_hashes receives too many arguments and breaks due to ARG_MAX limitations (depending on the OS).

Here is where the bug happens

github.com/fluidattacks/makes@2244b2bb1e/src/cli/main/cli.py (L329-L340)

Specifically in the *paths argument provided to the shell (line 334).

Let's make _nix_hashes receive bytes from stdin and then call nix-store using xargs so it does not hit the ARG_MAX limit.

I found this bug by running m gitlab:fluidattacks/universe@trunk /docs/generate/criteria, as it currently depends on several hundred derivations for building the entire criteria section of the website.

When running jobs that have huge provenance files, the CLI function `_nix_hashes` receives too many arguments and breaks due to `ARG_MAX` limitations (depending on the OS). Here is where the bug happens https://github.com/fluidattacks/makes/blob/2244b2bb1e0a4e9b78e7e3d16ae3a43841b01328/src/cli/main/cli.py#L329-L340 Specifically in the `*paths` argument provided to the shell (line 334). Let's make `_nix_hashes` receive bytes from stdin and then call `nix-store` using `xargs` so it does not hit the `ARG_MAX` limit. I found this bug by running `m gitlab:fluidattacks/universe@trunk /docs/generate/criteria`, as it currently depends on several hundred derivations for building the entire criteria section of the website.
This discussion has been locked. Commenting is limited to contributors.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
justin/makes#1128
No description provided.