Allow users to customize outputs list #1223

Open
opened 2023-12-16 03:26:45 +01:00 by Ten0 · 7 comments
Ten0 commented 2023-12-16 03:26:45 +01:00 (Migrated from github.com)

I'm considering using this project.

IIUC, I currently have two ways to generate an "output" for the CLI:

  1. use a builtin, and that will generate an arbirary number of targets depending on some magic in makes' own code
  2. use an extension, which always means creating a folder for the target, and having a main.nix in that folder.

What I want to achieve is the following directory structure:

project1
        /service1/<main or makes>.nix
        /service2/<main or makes>.nix
project2
        /service1/<main or makes>.nix
makes/service.nix

and I want my possible "OUTPUT"s in the CLI to look like this:

/project1/service1/build
/project1/service1/test
/project1/service1/run
/project1/service1/deploy
/project1/service2/build
/project1/service2/test
/project1/service2/run
/project1/service2/deploy
/project2/service1/build
/project2/service1/test
/project2/service1/run
/project2/service1/deploy

where the build/test/run/deploy outputs behavior are all defined by my /makes/service.nix

That means I need to be able to customize:

  • The list of targets that are made available from a <main or makes>.nixs (so that with a single nix file in a service folder I can declare the multiple targets that I need)
  • The arguments that are passed to the <main or makes>.nixs (to be able to pass import ./makes/service.nix)

My understanding is that this is not possible currently, and instead any such service.nix would have to be declared as a builtin in makes itself, and then I would put only the configuration of that in some makes.nix, which also means I could only instantiate one of each of those per makes.nix, whereas my ideal syntax would probably be:

# /project1/service1/<main or makes>.nix
{ args, service, pkgs, ... }: service { name = "a"; }
# /makes/service.nix
{ args, pkgs, makeScript, deployContainerImage, outputAttrSet, ... }:
let
  build = { /* ... */ };
  docker = pkgs.dockerTools.buildLayeredImage { /* ... */ };
  # Where args are the arguments pased as [ARGS...] to the CLI and may be used here
in
outputAttrSet {
  inherit build;
  deploy = deployContainerImage {
    images = [{
      src = docker;
      registry = { /* ... */ };
    }];
  };
  test = { /* ... */};
  run = makeScript { /* ... */ };
}

(where outputAttrSet would convert from the attrset of output to probably [{ name, derivation }], allowing for further customization if required)

Is it actually already possible to achieve something like this? If so, how? If not, is this something that could maybe be made possible in the future or are there any clear blockers?

Thanks,

I'm considering using this project. IIUC, I currently have two ways to generate an "output" for the CLI: 1. use a `builtin`, and that will generate an arbirary number of targets depending on some magic in `makes`' own code 2. use an `extension`, which *always* means creating a folder for the target, and having a `main.nix` in that folder. What I want to achieve is the following directory structure: ``` project1 /service1/<main or makes>.nix /service2/<main or makes>.nix project2 /service1/<main or makes>.nix makes/service.nix ``` and I want my possible "OUTPUT"s in the CLI to look like this: ``` /project1/service1/build /project1/service1/test /project1/service1/run /project1/service1/deploy /project1/service2/build /project1/service2/test /project1/service2/run /project1/service2/deploy /project2/service1/build /project2/service1/test /project2/service1/run /project2/service1/deploy ``` where the build/test/run/deploy outputs behavior are all defined by my `/makes/service.nix` That means I need to be able to customize: - The *list* of targets that are made available from a `<main or makes>.nix`s (so that with a single nix file in a service folder I can declare the multiple targets that I need) - The arguments that are passed to the `<main or makes>.nix`s (to be able to pass `import ./makes/service.nix`) My understanding is that this is not possible currently, and instead any such `service.nix` would have to be declared as a builtin in `makes` itself, and then I would put only the configuration of that in some `makes.nix`, which also means I could only instantiate one of each of those per `makes.nix`, whereas my ideal syntax would probably be: ```nix # /project1/service1/<main or makes>.nix { args, service, pkgs, ... }: service { name = "a"; } ``` ```nix # /makes/service.nix { args, pkgs, makeScript, deployContainerImage, outputAttrSet, ... }: let build = { /* ... */ }; docker = pkgs.dockerTools.buildLayeredImage { /* ... */ }; # Where args are the arguments pased as [ARGS...] to the CLI and may be used here in outputAttrSet { inherit build; deploy = deployContainerImage { images = [{ src = docker; registry = { /* ... */ }; }]; }; test = { /* ... */}; run = makeScript { /* ... */ }; } ``` (where `outputAttrSet` would convert from the attrset of output to probably `[{ name, derivation }]`, allowing for further customization if required) Is it actually already possible to achieve something like this? If so, how? If not, is this something that could maybe be made possible in the future or are there any clear blockers? Thanks,
dsalaza4 commented 2023-12-19 21:05:01 +01:00 (Migrated from github.com)

Hey @Ten0,

For main.nix files, you can set extendingMakesDirs = ["/"]; as described in the documentation. With this setting Makes will name CLI jobs according to absolute paths within the repository. That is, for file /project1/service1/run/main.nix, the CLI output would be /project1/service1/run.

With makes.nix things are a little more complicated. makes.nix is an interface between:

  1. evaluators: default configurations that builtins receive (This is what you declare in makes.nix.
  2. arguments: Makes' builtins.

This is why, when declaring builtins from makes.nix files, CLI outputs do not follow a naming pattern based on directory structure like main.nix files but rather group all jobs for the same builtin.

Example:

For the following makes.nix file

{
  formatPython,
  ...
}: {
  formatPython = {
    project1.targets = ["/project1"];
    project2.targets = ["/project2"];
    project3.targets = ["/project3"];
  };
}

We get the following CLI outputs:

/formatPython/project1
/formatPython/project2
/formatPython/project3

With some boilerplate code you could still use all builtins from main.nix files, granting directory-based CLI outputs.

Example:

/project1/service1/format/main.nix

{
  formatNix,
  makeScript,
  projectPath,
  ...
}: let
  bin = formatNix {
    name = "project1-service1";
    targets = [(projectPath "/project1/service1")];
  };
in
  makeScript {
    entrypoint = "format-nix-for-project1-service1";
    name = "project1-service1-format";
    searchPaths.bin = [bin];
  }

From that main.nix you can do anything that nix allows so you can orchestrate everything from a makes/services.nix file:

/project1/service1/format/main.nix

{
  formatNix,
  makeScript, 
  projectPath,
  ...
}: let
  config = import (projectPath "/makes/services.nix") {
    inherit formatNix;
    inherit makeScript;
    inherit projectPath;
  };
in
# customFormatNix is a more generic function that runs formatNix as explained before
config.customFormatNix {
  name = config.project1.service1.format.name; 
  targets = config.project1.service1.targets;
}
Hey @Ten0, For `main.nix` files, you can set `extendingMakesDirs = ["/"];` as described in the [documentation](https://makes.fluidattacks.com/api/builtins/framework-configuration/?h=exten#extendingmakesdirs). With this setting Makes will name CLI jobs according to absolute paths within the repository. That is, for file `/project1/service1/run/main.nix`, the CLI output would be `/project1/service1/run`. With `makes.nix` things are a little more complicated. `makes.nix` is an interface between: 1. [evaluators](https://github.com/fluidattacks/makes/tree/main/src/evaluator/modules): default configurations that builtins receive (This is what you declare in `makes.nix`. 2. [arguments](https://github.com/fluidattacks/makes/tree/main/src/args): Makes' builtins. This is why, when declaring builtins from `makes.nix` files, CLI outputs do not follow a naming pattern based on directory structure like `main.nix` files but rather group all jobs for the same builtin. Example: For the following `makes.nix` file ```nix { formatPython, ... }: { formatPython = { project1.targets = ["/project1"]; project2.targets = ["/project2"]; project3.targets = ["/project3"]; }; } ``` We get the following CLI outputs: ``` /formatPython/project1 /formatPython/project2 /formatPython/project3 ``` With some boilerplate code you could still use all builtins from `main.nix` files, granting directory-based CLI outputs. Example: `/project1/service1/format/main.nix` ```nix { formatNix, makeScript, projectPath, ... }: let bin = formatNix { name = "project1-service1"; targets = [(projectPath "/project1/service1")]; }; in makeScript { entrypoint = "format-nix-for-project1-service1"; name = "project1-service1-format"; searchPaths.bin = [bin]; } ``` From that `main.nix` you can do anything that nix allows so you can orchestrate everything from a `makes/services.nix` file: `/project1/service1/format/main.nix` ```nix { formatNix, makeScript, projectPath, ... }: let config = import (projectPath "/makes/services.nix") { inherit formatNix; inherit makeScript; inherit projectPath; }; in # customFormatNix is a more generic function that runs formatNix as explained before config.customFormatNix { name = config.project1.service1.format.name; targets = config.project1.service1.targets; } ```
dsalaza4 commented 2023-12-19 21:13:17 +01:00 (Migrated from github.com)

You could even declare everything in the services.nix file and simply import it from the main.nix:

makesArgs: let
  config = import (makesArgs.projectPath "/makes/services.nix") {inherit makesArgs;};
in
config.project1.service1.format.run
You could even declare everything in the `services.nix` file and simply import it from the `main.nix`: ```nix makesArgs: let config = import (makesArgs.projectPath "/makes/services.nix") {inherit makesArgs;}; in config.project1.service1.format.run ```
Ten0 commented 2023-12-19 22:38:12 +01:00 (Migrated from github.com)

Thank you for your answer! 🙂

So it looks like my understanding was correct: to generate target paths, I'm tied by either the builtins (which I can't customize) or the "one folder with a main.nix per output" approach (which would force me to create one directory per output with a main file that would specify a bunch of boilerplate and finally .target_name(=folder_name), which woudn't fit at all with my current directory structure or be one I'd like anyway) 🙁

Is there any technical constraint that prevents building a system that has a more flexible interface similar to the one I described above? Having the root expression that evaluates to just at-most-one-incantation-of-each-type-of-builtin doesn't allow for flexible program structure compared to something along the lines of [{ name, thing_to_do }].

At first glance it looks like we could evaluate only the part of the current folder's expressions that are related to outputs names when starting the CLI, and then only when actually building, evaluate the derivation attribute of each { name, derivation } that the root expression outputs, but there may be big drawbacks that I'm missing.

Would a design where we add an outputs builtin that takes as parameter [{ name, derivation }] work or is there some constraint that prevents passing such complex inputs to builtins? (like, they have to get evaluated before they get passed or something along those lines)

To be very clear: I love the idea of the project of having a CI system globally powered by nix, with a CLI that enables choosing the targets we're interested in, especially making full use of caching across all users of the organization for every single derivation. The potential of it looks insane! However, if it seems that the only way to efficiently add outputs to the CLI is to make incantations in the makes.nix after PR-ing a new builtin to this repository, then it seems pretty clear to me that that doesn't scale unless you happen to also be the company that manages the Makes project: we can't have every company use this and PR every single customization they need as a global builtin in the project:

  • Some builtins would be too custom for integration.
  • The integration friction would be too high.
  • That's too many global builtins and it becomes an absolute mess. (Actually there's already too many global builtins dedicated to tools and languages I don't use for my taste)

Thanks a lot 🙂

Thank you for your answer! 🙂 So it looks like my understanding was correct: to generate target paths, I'm tied by either the builtins (which I can't customize) or the "one folder with a main.nix per output" approach (which would force me to create one directory per output with a main file that would specify a bunch of boilerplate and finally `.target_name`(=`folder_name`), which woudn't fit at all with my current directory structure or be one I'd like anyway) 🙁 Is there any technical constraint that prevents building a system that has a more flexible interface similar to the one I described [above](https://github.com/fluidattacks/makes/issues/1223#issue-2044569874)? Having the root expression that evaluates to just at-most-one-incantation-of-each-type-of-builtin doesn't allow for flexible program structure compared to something along the lines of `[{ name, thing_to_do }]`. At first glance it looks like we could evaluate only the part of the current folder's expressions that are related to outputs names when starting the CLI, and then only when actually building, evaluate the `derivation` attribute of each `{ name, derivation }` that the root expression outputs, but there may be big drawbacks that I'm missing. Would a design where we add an `outputs` builtin that takes as parameter `[{ name, derivation }]` work or is there some constraint that prevents passing such complex inputs to builtins? (like, they have to get evaluated before they get passed or something along those lines) To be very clear: I love the idea of the project of having a CI system globally powered by nix, with a CLI that enables choosing the targets we're interested in, *especially* making full use of caching across all users of the organization for every single derivation. The potential of it looks insane! However, **if** it seems that the only way to efficiently add outputs to the CLI is to make incantations in the `makes.nix` after PR-ing a new builtin to this repository, then it seems pretty clear to me that that doesn't scale unless you happen to also be the company that manages the `Makes` project: we can't have every company use this and PR every single customization they need as a global builtin in the project: - Some builtins would be too custom for integration. - The integration friction would be too high. - That's too many global builtins and it becomes an absolute mess. (Actually there's already too many global builtins dedicated to tools and languages I don't use for my taste) Thanks a lot 🙂
dsalaza4 commented 2024-12-17 02:06:54 +01:00 (Migrated from github.com)

I will start working on a first iteration that allows to declare jobs = {} within makes.nix so we can begin moving away from main.nix files.

I will start working on a first iteration that allows to declare `jobs = {}` within `makes.nix` so we can begin moving away from `main.nix` files.
dsalaza4 commented 2024-12-20 17:57:35 +01:00 (Migrated from github.com)

@Ten0 Hey! If you're still interested, take a look at the new jobs = {} essential. It basically allows to describe jobs directly in makes.nix files. https://makes.fluidattacks.tech/configuration/essentials/#jobs

@Ten0 Hey! If you're still interested, take a look at the new `jobs = {}` essential. It basically allows to describe jobs directly in `makes.nix` files. https://makes.fluidattacks.tech/configuration/essentials/#jobs
Ten0 commented 2024-12-20 22:32:54 +01:00 (Migrated from github.com)

Hey! Thanks for letting me know! That's indeed very interesting!
I may give this another look in the coming weeks. 👍️

Hey! Thanks for letting me know! That's indeed very interesting! I may give this another look in the coming weeks. 👍️
dsalaza4 commented 2024-12-20 22:54:20 +01:00 (Migrated from github.com)
  • Deprecate main.nix files in Universe
  • Deprecate main.nix support from Makes CLI
  • Deprecate main.nix support from Makes backend
- [ ] Deprecate `main.nix` files in Universe - [ ] Deprecate `main.nix` support from Makes CLI - [ ] Deprecate `main.nix` support from Makes backend
This discussion has been locked. Commenting is limited to contributors.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
justin/makes#1223
No description provided.