Command orchestrator #502

Closed
opened 2021-09-07 16:27:17 +02:00 by blaggacao · 4 comments
blaggacao commented 2021-09-07 16:27:17 +02:00 (Migrated from github.com)

Now, as I see it, makes is an in-tree job tree with a cli storefront.

Having such a sound framework for in-tree jobs, I wonder what could be a well-defined interface up north to a job orchestrator.

That can be both, a scheduler, such as nomad or k8s, as well as a workflow orchestrator.

For this issue, I'm more interested in exploring possible interfaces to zeebe.io as a workflow automation tool.

Do I understand correctly, that currently each job's interface is basically cli args to the respective bash script & outputs to stdout of the same?

Also: are jobs currently limited to bash scripts? Can I user other languagues like rust, etc?

Now, as I see it, makes is an _in-tree job tree with a cli storefront_. Having such a sound framework for in-tree jobs, I wonder what could be a well-defined interface up north to a job orchestrator. That can be both, a scheduler, such as nomad or k8s, as well as a workflow orchestrator. For this issue, I'm more interested in exploring possible interfaces to zeebe.io as a workflow automation tool. Do I understand correctly, that currently each job's interface is basically cli args to the respective bash script & outputs to stdout of the same? Also: are jobs currently limited to bash scripts? Can I user other languagues like rust, etc?
kamadorueda commented 2021-09-08 23:24:04 +02:00 (Migrated from github.com)

@blaggacao You may want to read this first: https://github.com/fluidattacks/makes/issues/180#issuecomment-915563837

Having such a sound framework for in-tree jobs, I wonder what could be a well-defined interface up north to a job orchestrator.

That can be both, a scheduler, such as nomad or k8s, as well as a workflow orchestrator.

We as a Framework currently support these providers: https://github.com/fluidattacks/makes#providers-comparison

Which means someone can get Makes running on Gitlab/Github/Travis/Locally-on-a-laptop at the very least (and generally over anything that supports containers)


For this issue, I'm more interested in exploring possible interfaces to zeebe.io as a workflow automation tool.

⚠️ Experimental stuff alert

I'm currently exploring being able to generate configuration files automatically for Gitlab (https://github.com/fluidattacks/makes/issues/180), this is how it looks:

https://gitlab.com/fluidattacks/product/-/blob/master/makes/makes/pipeline/makes.nix

And this is what it generates:

https://gitlab.com/fluidattacks/product/-/blob/master/makes/makes/gitlab-ci.yaml

Gitlab is able to understand this big YAML and produce a CI/CD with jobs/workflows and that fancy stuff:

  • image

Now. Why I mention this? because if zeebee.io is able to receive some written configuration as input, we could create a module on Makes that generates such configuration.

This is how support for Gitlab (again, experimental) is implemented:

People just configure dependencies/stages/etc between jobs, and voila

@blaggacao You may want to read this first: https://github.com/fluidattacks/makes/issues/180#issuecomment-915563837 > Having such a sound framework for in-tree jobs, I wonder what could be a well-defined interface up north to a job orchestrator. > That can be both, a scheduler, such as nomad or k8s, as well as a workflow orchestrator. We as a Framework currently support these providers: https://github.com/fluidattacks/makes#providers-comparison Which means someone can get Makes running on Gitlab/Github/Travis/Locally-on-a-laptop at the very least (and generally over anything that supports containers) --- > For this issue, I'm more interested in exploring possible interfaces to zeebe.io as a workflow automation tool. ⚠️ Experimental stuff alert I'm currently exploring being able to generate configuration files automatically for Gitlab (https://github.com/fluidattacks/makes/issues/180), this is how it looks: https://gitlab.com/fluidattacks/product/-/blob/master/makes/makes/pipeline/makes.nix And this is what it generates: https://gitlab.com/fluidattacks/product/-/blob/master/makes/makes/gitlab-ci.yaml Gitlab is able to understand this big YAML and produce a CI/CD with jobs/workflows and that fancy stuff: - ![image](https://user-images.githubusercontent.com/47480384/132587346-fb047908-7b2d-41c8-b219-f9fd87175abb.png) Now. Why I mention this? because if zeebee.io is able to receive some written configuration as input, we could create a module on Makes that generates such configuration. This is how support for Gitlab (again, experimental) is implemented: - https://github.com/fluidattacks/makes/tree/main/src/evaluator/modules/pipelines People just configure dependencies/stages/etc between jobs, and voila
blaggacao commented 2021-09-09 00:23:11 +02:00 (Migrated from github.com)

I figured that as a job runner, makes is actually perfectly suited for a command and control messaging pattern.

I'm musing about a tiny wrapper-worker that just maps command messages (coming from zeebe, apache airflow, whatever) to makes jobs and wires them the message payload.

I wonder if the framework could take a stance on the way that "wiring" should/could be done. E.g. a simple translation of a message struct to command line args?

Nevermind though, this is just Q5 ideas of mine.

I figured that as a job runner, makes is actually perfectly suited for a command and control messaging pattern. I'm musing about a tiny wrapper-worker that just maps command messages (coming from zeebe, apache airflow, whatever) to makes jobs and wires them the message payload. I wonder if the framework could take a stance on the way that "wiring" should/could be done. E.g. a simple translation of a message struct to command line args? Nevermind though, this is just Q5 ideas of mine.
kamadorueda commented 2021-09-09 02:52:09 +02:00 (Migrated from github.com)

I figured that as a job runner, makes is actually perfectly suited for a command and control messaging pattern.

I had to read about that and I agree that yes, it is well suited for C&C

I'm musing about a tiny wrapper-worker that just maps command messages (coming from zeebe, apache airflow, whatever) to makes jobs and wires them the message payload.

I wonder if the framework could take a stance on the way that "wiring" should/could be done. E.g. a simple translation of a message struct to command line args?

Preferably message -> makes-command, yes,
so it resembles the format $ m "${src}" "${output}" "${cli_args_if_any[@]}"

The framework guarantees you stability of this CLI interface.
You can also pin Makes CLI and Makes framework to a commit for maximum stability.

You can point to your source code repository with "${src}" (example github:owner/repo@rev).
and:

  • whatever is built is done in a Nix sandbox, isolated from the file system
  • whatever is executed will be done in the context of a fresh copy of "${src}" stored in a temporary directory

TODO: support private "${src}"s ?
github.com/fluidattacks/makes@939a40963d/src/cli/main/main.py#L64-L106


In the case of apache airflow the most essential case would be a BashOperator that triggers $ m ...


TODO: In the case that storing the output were important:

  • $out for derivations
  • $STATE for non-derivations

We can add something to the module system that abstracts putting those results on different backends:

  • a S3 bucket
  • a specific path in the file system
  • etc

This part is easy to implement


TODO: rename $STATE to $out so it's more intuitive?


Nevermind though, this is just Q5 ideas of mine.

I'm enjoying and learning with the conversation

> I figured that as a job runner, makes is actually perfectly suited for a command and control messaging pattern. I had to read about that and I agree that yes, it is well suited for C&C > I'm musing about a tiny wrapper-worker that just maps command messages (coming from zeebe, apache airflow, whatever) to makes jobs and wires them the message payload. > I wonder if the framework could take a stance on the way that "wiring" should/could be done. E.g. a simple translation of a message struct to command line args? Preferably `message -> makes-command`, yes, so it resembles the format `$ m "${src}" "${output}" "${cli_args_if_any[@]}"` The framework guarantees you stability of this CLI interface. You can also pin Makes CLI and Makes framework to a commit for maximum stability. You can point to your source code repository with `"${src}"` (example github:owner/repo@rev). and: - whatever is built is done in a Nix sandbox, isolated from the file system - whatever is executed will be done in the context of a fresh copy of `"${src}"` stored in a temporary directory --- TODO: support private "${src}"s ? https://github.com/fluidattacks/makes/blob/939a40963d820ad35a296d00cabe43cdce16eae7/src/cli/main/__main__.py#L64-L106 --- In the case of apache airflow the most essential case would be a [BashOperator](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html) that triggers `$ m ..`. --- TODO: In the case that storing the output were important: - $out for derivations - [$STATE](https://github.com/fluidattacks/makes/blob/main/src/args/make-script/template.sh#L25) for non-derivations We can add something to the module system that abstracts putting those results on different backends: - a S3 bucket - a specific path in the file system - etc This part is easy to implement --- TODO: rename $STATE to $out so it's more intuitive? --- > Nevermind though, this is just Q5 ideas of mine. I'm enjoying and learning with the conversation
dsalaza4 commented 2024-12-11 23:17:47 +01:00 (Migrated from github.com)

We want Makes to be core-lean, we will focus on providing:

  1. Basic builtins for populating pristine environments (makeScript, projectPath, fromJson, toJson, ...)
  2. A base container for being able to run such environments on systems that support containers (k8s, nomad, etc)
  3. A plugin interface that allows to import flakes or default.nix from other repositories that implement custom functionality (deployNomad, deployk8s, lintRust, etc...)
We want Makes to be core-lean, we will focus on providing: 1. Basic builtins for populating pristine environments (makeScript, projectPath, fromJson, toJson, ...) 2. A base container for being able to run such environments on systems that support containers (k8s, nomad, etc) 3. A plugin interface that allows to import flakes or default.nix from other repositories that implement custom functionality (deployNomad, deployk8s, lintRust, etc...)
This discussion has been locked. Commenting is limited to contributors.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
justin/makes#502
No description provided.