Implement chunks builtin for makeDerivationParallel #995

Closed
opened 2023-01-13 21:45:30 +01:00 by dsalaza4 · 0 comments
dsalaza4 commented 2023-01-13 21:45:30 +01:00 (Migrated from github.com)

makeDerivationParallel is great for local parallelization in just one machine.

Many times, the next step for improving performance is distributing workloads into several machines.

A chunks builtin would be great for this:

{
  chunks,
  inputs,
  makeDerivation,
  makeDerivationParallel,
}: let
numbers = [0 1 2 3 4 5 6 7 8 9];
myChunks =  chunks numbers 3; # [[0 1 2 3] [4 5 6] [7 8 9]]

buildNumber = n: makeDerivation {
  name = "build-number-${n}";
  env.envNumber = n;
  builder = ''
    echo "$envNumber"
    touch "$out"
  '';
};
in
  makeDerivationParallel {
    dependencies = builtins.map buildNumber (inputs.nixpkgs.lib.lists.elemAt myChunks 0);
    name = "build-numbers-0";
  }

With this logic one would be able to easily distribute workloads into several CI jobs, achieving distributed parallelization.

`makeDerivationParallel` is great for local parallelization in just one machine. Many times, the next step for improving performance is distributing workloads into several machines. A `chunks` builtin would be great for this: ```nix { chunks, inputs, makeDerivation, makeDerivationParallel, }: let numbers = [0 1 2 3 4 5 6 7 8 9]; myChunks = chunks numbers 3; # [[0 1 2 3] [4 5 6] [7 8 9]] buildNumber = n: makeDerivation { name = "build-number-${n}"; env.envNumber = n; builder = '' echo "$envNumber" touch "$out" ''; }; in makeDerivationParallel { dependencies = builtins.map buildNumber (inputs.nixpkgs.lib.lists.elemAt myChunks 0); name = "build-numbers-0"; } ``` With this logic one would be able to easily distribute workloads into several CI jobs, achieving distributed parallelization.
This discussion has been locked. Commenting is limited to contributors.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
justin/makes#995
No description provided.