Show HN: Kestra – Initiate-Offer Airflow Different

Website • Twitter • Linked In • Discord • Documentation Demo Play with our demo app! What is Kestra ? Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines. 🔀 Any kind of workflow: Workflows can start simple and progress to more complex systems with branching,…

Show HN: Kestra – Initiate-Offer Airflow Different

Kestra workflow orchestrator

Github star
Last Version
Docker pull
Artifact Hub
Kestra infinitely scalable orchestration and scheduling platform
Github discussions
Code Cov
Github Actions

Net put
Linked In

modern data orchestration and scheduling platform


Play with our demo app!

What is Kestra ?

Kestra is an infinitely scalable orchestration and scheduling platform, constructing, running, scheduling, and monitoring millions of advanced pipelines.

  • 🔀 Any abolish of workflow: Workflows can open straightforward and growth to extra advanced systems with branching, parallel, dynamic projects, float dependencies
  • 🎓‍ Easy to be taught: Flows are in straightforward, descriptive language outlined in YAML;u originate no longer must be a developer to construct a brand original float.
  • 🔣 Easy to elongate: Plugins are all over the situation in Kestra, many are readily accessible from the Kestra core workforce, but that you would be in a position to construct one with out problems.
  • 🆙 Any triggers: Kestra is event-primarily based mostly at heart—that you would be in a position to dwelling off an execution from API, agenda, detection, occasions
  • 💻 A prosperous user interface: The built-in web interface allows you to construct, run, and video show your total flows—no have confidence to deploy your flows, ethical edit them.
  • Revel in limitless scalability: Kestra is built spherical high cloud native technologies—scale to millions of executions stress-free.

Instance float:

identification: my-first-float

  - style: FILE
    name: uploaded
    description: A Csv file to be uploaded thru API or UI

  - identification: archive
    style: io.kestra.plugin.gcp.gcs.Add
    description: Archive the file on Google Cloud Storage bucket
    from: "{{ inputs.uploaded }}"
    to: "gs://my_bucket/archives/{{ execution.identification }}.csv"

  - identification: csvReader
    style: io.kestra.plugin.serdes.csv.CsvReader
    from: "{{ inputs.uploaded }}"

  - identification: fileTransform
    style: io.kestra.plugin.scripts.nashorn.FileTransform
    description: This job will anonymize the contactName with a custom nashorn script (javascript over jvm). This disclose that you in a position to address custom transformation or remapping within the ETL ability
    from: "{{ outputs.csvReader.uri }}"
    script: |
      if (row['contactName']) {

  - identification: avroWriter
    style: io.kestra.plugin.serdes.avro.AvroWriter
    description: This file will convert the file from Kestra interior storage to avro. Again, we handling ETL since the conversion is completed by Kestra earlier than loading the knowledge in BigQuery. This let you have confidence some control earlier than loading and to reject impolite knowledge as soon as imaginable.
    from: "{{ outputs.fileTransform.uri }}"
    schema: |
        "style": "fable",
        "name": "Root",
            { "name": "contactTitle", "type": ["null", "string"] },
            { "name": "postalCode", "type": ["null", "long"] },
            { "name": "entityId", "type": ["null", "long"] },
            { "name": "country", "type": ["null", "string"] },
            { "name": "region", "type": ["null", "string"] },
            { "name": "address", "type": ["null", "string"] },
            { "name": "fax", "type": ["null", "string"] },
            { "name": "email", "type": ["null", "string"] },
            { "name": "mobile", "type": ["null", "string"] },
            { "name": "companyName", "type": ["null", "string"] },
            { "name": "contactName", "type": ["null", "string"] },
            { "name": "phone", "type": ["null", "string"] },
            { "name": "city", "type": ["null", "string"] }

  - identification: load
    style: io.kestra.plugin.gcp.bigquery.Load
    description: Simply load the generated from avro job to BigQuery
      useAvroLogicalTypes: correct
    destinationTable: kestra-prd.demo.customer_copy
    layout: AVRO
    from: "{{outputs.avroWriter.uri }}"
    writeDisposition: WRITE_TRUNCATE

  - identification: combination
    style: io.kestra.plugin.gcp.bigquery.Ask
    description: Combination some knowledge from loaded files
    createDisposition: CREATE_IF_NEEDED
    destinationTable: kestra-prd.demo.agg
    sql: |
      SELECT good enough.categoryName, p.productName, c.companyName, s.orderDate, SUM(d.quantity) AS quantity, SUM(d.unitPrice d.quantity r.substitute) as totalEur
      FROM `kestra-prd.demo.salesOrder` AS s
      INNER JOIN `kestra-prd.demo.orderDetail` AS d ON s.entityId=d.orderId
      INNER JOIN `kestra-prd.demo.buyer` AS c ON c.entityId=s.customerId
      INNER JOIN `kestra-prd.demo.product` AS p ON p.entityId=d.productId
      INNER JOIN `kestra-prd.demo.class` AS good enough ON good enough.entityId=p.categoryId
      INNER JOIN `kestra-prd.demo.rates` AS r ON AND"USD"
      GROUP BY 1, 2, 3, 4
    timePartitioningField: orderDate
    writeDisposition: WRITE_TRUNCATE

Getting Started

To score a native reproduction up and running, please apply these straightforward steps.


Originate decided you have confidence already put aside in:

Begin Kestra

  • Download the manufacture file here and put aside it aside with the name docker-manufacture.yml, for linux and macos, that you would be in a position to run wget
  • Bustle docker-manufacture pull
  • Bustle docker-manufacture up -d
  • Initiate http://localhost: 8080 on your browser
  • Conform to this tutorial to construct your first float.
  • Learn the documentation to know


Kestra is built on plugin systems. You need to salvage your plugin to have confidence interaction alongside with your provider; alternatively, that you would be in a position to apply straightforward steps to develop your score plugin. Right here are the authentic plugins that are readily accessible:

This list is increasing rapid as we’re actively building extra plugins, and we welcome contributions!

Neighborhood Strengthen

Join our community whenever you wish support, are searching to chat or have confidence any other questions for us:

  • GitHub – Dialogue forums and updates from the Kestra workforce
  • Twitter – For the total most up-to-date Kestra news
  • Discord – Join the dialog! Accumulate the total most up-to-date updates and chat to the devs


Glimpse the open disorders for a checklist of proposed facets (and known disorders) or see on the mission board.

Growing domestically & Contributing

We love contributions wonderful or runt, test out our guide on open.

Glimpse our Plugin Developer Recordsdata for constructing Kestra plugins.


Apache 2.0 © Kestra Technologies

Read More

Charlie Layers

Charlie Layers

Fill your life with experiences so you always have a great story to tellBio: About: