Command orchestrator #502
Labels
No labels
dependencies
difficulty::high
difficulty::low
difficulty::medium
good first issue
help wanted
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
justin/makes#502
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Now, as I see it, makes is an in-tree job tree with a cli storefront.
Having such a sound framework for in-tree jobs, I wonder what could be a well-defined interface up north to a job orchestrator.
That can be both, a scheduler, such as nomad or k8s, as well as a workflow orchestrator.
For this issue, I'm more interested in exploring possible interfaces to zeebe.io as a workflow automation tool.
Do I understand correctly, that currently each job's interface is basically cli args to the respective bash script & outputs to stdout of the same?
Also: are jobs currently limited to bash scripts? Can I user other languagues like rust, etc?
@blaggacao You may want to read this first: https://github.com/fluidattacks/makes/issues/180#issuecomment-915563837
We as a Framework currently support these providers: https://github.com/fluidattacks/makes#providers-comparison
Which means someone can get Makes running on Gitlab/Github/Travis/Locally-on-a-laptop at the very least (and generally over anything that supports containers)
⚠️ Experimental stuff alert
I'm currently exploring being able to generate configuration files automatically for Gitlab (https://github.com/fluidattacks/makes/issues/180), this is how it looks:
https://gitlab.com/fluidattacks/product/-/blob/master/makes/makes/pipeline/makes.nix
And this is what it generates:
https://gitlab.com/fluidattacks/product/-/blob/master/makes/makes/gitlab-ci.yaml
Gitlab is able to understand this big YAML and produce a CI/CD with jobs/workflows and that fancy stuff:
Now. Why I mention this? because if zeebee.io is able to receive some written configuration as input, we could create a module on Makes that generates such configuration.
This is how support for Gitlab (again, experimental) is implemented:
People just configure dependencies/stages/etc between jobs, and voila
I figured that as a job runner, makes is actually perfectly suited for a command and control messaging pattern.
I'm musing about a tiny wrapper-worker that just maps command messages (coming from zeebe, apache airflow, whatever) to makes jobs and wires them the message payload.
I wonder if the framework could take a stance on the way that "wiring" should/could be done. E.g. a simple translation of a message struct to command line args?
Nevermind though, this is just Q5 ideas of mine.
I had to read about that and I agree that yes, it is well suited for C&C
Preferably
message -> makes-command, yes,so it resembles the format
$ m "${src}" "${output}" "${cli_args_if_any[@]}"The framework guarantees you stability of this CLI interface.
You can also pin Makes CLI and Makes framework to a commit for maximum stability.
You can point to your source code repository with
"${src}"(example github:owner/repo@rev).and:
"${src}"stored in a temporary directoryTODO: support private "${src}"s ?
github.com/fluidattacks/makes@939a40963d/src/cli/main/main.py#L64-L106In the case of apache airflow the most essential case would be a BashOperator that triggers
$ m ...TODO: In the case that storing the output were important:
We can add something to the module system that abstracts putting those results on different backends:
This part is easy to implement
TODO: rename $STATE to $out so it's more intuitive?
I'm enjoying and learning with the conversation
We want Makes to be core-lean, we will focus on providing: