Show HN: Kestra – Initiate-Offer Airflow Different

Net put •
Twitter •
Linked In •
Discord •
Documentation

Demo

Play with our demo app!

What is Kestra ?

Kestra is an infinitely scalable orchestration and scheduling platform, constructing, running, scheduling, and monitoring millions of advanced pipelines.

🔀 Any abolish of workflow: Workflows can open straightforward and growth to extra advanced systems with branching, parallel, dynamic projects, float dependencies
🎓‍ Easy to be taught: Flows are in straightforward, descriptive language outlined in YAML;u originate no longer must be a developer to construct a brand original float.
🔣 Easy to elongate: Plugins are all over the situation in Kestra, many are readily accessible from the Kestra core workforce, but that you would be in a position to construct one with out problems.
🆙 Any triggers: Kestra is event-primarily based mostly at heart—that you would be in a position to dwelling off an execution from API, agenda, detection, occasions
💻 A prosperous user interface: The built-in web interface allows you to construct, run, and video show your total flows—no have confidence to deploy your flows, ethical edit them.
⏩ Revel in limitless scalability: Kestra is built spherical high cloud native technologies—scale to millions of executions stress-free.

Instance float:

identification: my-first-float
namespace: my.company.groups

inputs:
  - style: FILE
    name: uploaded
    description: A Csv file to be uploaded thru API or UI

projects:
  - identification: archive
    style: io.kestra.plugin.gcp.gcs.Add
    description: Archive the file on Google Cloud Storage bucket
    from: "{{ inputs.uploaded }}"
    to: "gs://my_bucket/archives/{{ execution.identification }}.csv"

  - identification: csvReader
    style: io.kestra.plugin.serdes.csv.CsvReader
    from: "{{ inputs.uploaded }}"

  - identification: fileTransform
    style: io.kestra.plugin.scripts.nashorn.FileTransform
    description: This job will anonymize the contactName with a custom nashorn script (javascript over jvm). This disclose that you in a position to address custom transformation or remapping within the ETL ability
    from: "{{ outputs.csvReader.uri }}"
    script: |
      if (row['contactName']) {
        row['contactName']="*".repeat(row['contactName'].length);
      }

  - identification: avroWriter
    style: io.kestra.plugin.serdes.avro.AvroWriter
    description: This file will convert the file from Kestra interior storage to avro. Again, we handling ETL since the conversion is completed by Kestra earlier than loading the knowledge in BigQuery. This let you have confidence some control earlier than loading and to reject impolite knowledge as soon as imaginable.
    from: "{{ outputs.fileTransform.uri }}"
    schema: |
      {
        "style": "fable",
        "name": "Root",
        "fields": 
          [
            { "name": "contactTitle", "type": ["null", "string"] },
            { "name": "postalCode", "type": ["null", "long"] },
            { "name": "entityId", "type": ["null", "long"] },
            { "name": "country", "type": ["null", "string"] },
            { "name": "region", "type": ["null", "string"] },
            { "name": "address", "type": ["null", "string"] },
            { "name": "fax", "type": ["null", "string"] },
            { "name": "email", "type": ["null", "string"] },
            { "name": "mobile", "type": ["null", "string"] },
            { "name": "companyName", "type": ["null", "string"] },
            { "name": "contactName", "type": ["null", "string"] },
            { "name": "phone", "type": ["null", "string"] },
            { "name": "city", "type": ["null", "string"] }
          ]
      }

  - identification: load
    style: io.kestra.plugin.gcp.bigquery.Load
    description: Simply load the generated from avro job to BigQuery
    avroOptions:
      useAvroLogicalTypes: correct
    destinationTable: kestra-prd.demo.customer_copy
    layout: AVRO
    from: "{{outputs.avroWriter.uri }}"
    writeDisposition: WRITE_TRUNCATE

  - identification: combination
    style: io.kestra.plugin.gcp.bigquery.Ask
    description: Combination some knowledge from loaded files
    createDisposition: CREATE_IF_NEEDED
    destinationTable: kestra-prd.demo.agg
    sql: |
      SELECT good enough.categoryName, p.productName, c.companyName, s.orderDate, SUM(d.quantity) AS quantity, SUM(d.unitPrice d.quantity r.substitute) as totalEur
      FROM `kestra-prd.demo.salesOrder` AS s
      INNER JOIN `kestra-prd.demo.orderDetail` AS d ON s.entityId=d.orderId
      INNER JOIN `kestra-prd.demo.buyer` AS c ON c.entityId=s.customerId
      INNER JOIN `kestra-prd.demo.product` AS p ON p.entityId=d.productId
      INNER JOIN `kestra-prd.demo.class` AS good enough ON good enough.entityId=p.categoryId
      INNER JOIN `kestra-prd.demo.rates` AS r ON r.date=DATE(s.orderDate) AND r.forex="USD"
      GROUP BY 1, 2, 3, 4
    timePartitioningField: orderDate
    writeDisposition: WRITE_TRUNCATE

Getting Started

To score a native reproduction up and running, please apply these straightforward steps.

Necessities

Originate decided you have confidence already put aside in:

Begin Kestra

Download the manufacture file here and put aside it aside with the name docker-manufacture.yml, for linux and macos, that you would be in a position to run wget https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-manufacture.yml
Bustle docker-manufacture pull
Bustle docker-manufacture up -d
Initiate http://localhost: 8080 on your browser
Conform to this tutorial to construct your first float.
Learn the documentation to know
- Manufacture your flows
- Deploy Kestra
- Spend our terraform provider
- Manufacture your score plugins

Plugins

Kestra is built on plugin systems. You need to salvage your plugin to have confidence interaction alongside with your provider; alternatively, that you would be in a position to apply straightforward steps to develop your score plugin. Right here are the authentic plugins that are readily accessible:

This list is increasing rapid as we’re actively building extra plugins, and we welcome contributions!

Neighborhood Strengthen

Join our community whenever you wish support, are searching to chat or have confidence any other questions for us:

GitHub – Dialogue forums and updates from the Kestra workforce
Twitter – For the total most up-to-date Kestra news
Discord – Join the dialog! Accumulate the total most up-to-date updates and chat to the devs

Roadmap

Glimpse the open disorders for a checklist of proposed facets (and known disorders) or see on the mission board.

Growing domestically & Contributing

We love contributions wonderful or runt, test out our guide on open.

Glimpse our Plugin Developer Recordsdata for constructing Kestra plugins.