Show HN: Kestra – Initiate-Offer Airflow Different

27
[favorite_button]
Show HN: Kestra – Initiate-Offer Airflow Different
Advertisements


Kestra workflow orchestrator

License
Commits-per-month
Github star
Last Version
Docker pull
Artifact Hub
Kestra infinitely scalable orchestration and scheduling platform
Discord
Github discussions
Twitter
Code Cov
Github Actions

Net put
Twitter
Linked In
Discord
Documentation

modern data orchestration and scheduling platform

Demo

Play with our demo app!

What is Kestra ?

Kestra is an infinitely scalable orchestration and scheduling platform, constructing, running, scheduling, and monitoring millions of advanced pipelines.

  • 🔀 Any abolish of workflow: Workflows can open straightforward and growth to extra advanced systems with branching, parallel, dynamic projects, float dependencies
  • 🎓‍ Easy to be taught: Flows are in straightforward, descriptive language outlined in YAML;u originate no longer must be a developer to construct a brand original float.
  • 🔣 Easy to elongate: Plugins are all over the situation in Kestra, many are readily accessible from the Kestra core workforce, but that you would be in a position to construct one with out problems.
  • 🆙 Any triggers: Kestra is event-primarily based mostly at heart—that you would be in a position to dwelling off an execution from API, agenda, detection, occasions
  • 💻 A prosperous user interface: The built-in web interface allows you to construct, run, and video show your total flows—no have confidence to deploy your flows, ethical edit them.
  • Revel in limitless scalability: Kestra is built spherical high cloud native technologies—scale to millions of executions stress-free.

Instance float:

Advertisements

identification: my-first-float
namespace: my.company.groups

inputs:
  - style: FILE
    name: uploaded
    description: A Csv file to be uploaded thru API or UI

projects:
  - identification: archive
    style: io.kestra.plugin.gcp.gcs.Add
    description: Archive the file on Google Cloud Storage bucket
    from: "{{ inputs.uploaded }}"
    to: "gs://my_bucket/archives/{{ execution.identification }}.csv"

  - identification: csvReader
    style: io.kestra.plugin.serdes.csv.CsvReader
    from: "{{ inputs.uploaded }}"

  - identification: fileTransform
    style: io.kestra.plugin.scripts.nashorn.FileTransform
    description: This job will anonymize the contactName with a custom nashorn script (javascript over jvm). This disclose that you in a position to address custom transformation or remapping within the ETL ability
    from: "{{ outputs.csvReader.uri }}"
    script: |
      if (row['contactName']) {
        row['contactName']="*".repeat(row['contactName'].length);
      }

  - identification: avroWriter
    style: io.kestra.plugin.serdes.avro.AvroWriter
    description: This file will convert the file from Kestra interior storage to avro. Again, we handling ETL since the conversion is completed by Kestra earlier than loading the knowledge in BigQuery. This let you have confidence some control earlier than loading and to reject impolite knowledge as soon as imaginable.
    from: "{{ outputs.fileTransform.uri }}"
    schema: |
      {
        "style": "fable",
        "name": "Root",
        "fields": 
          [
            { "name": "contactTitle", "type": ["null", "string"] },
            { "name": "postalCode", "type": ["null", "long"] },
            { "name": "entityId", "type": ["null", "long"] },
            { "name": "country", "type": ["null", "string"] },
            { "name": "region", "type": ["null", "string"] },
            { "name": "address", "type": ["null", "string"] },
            { "name": "fax", "type": ["null", "string"] },
            { "name": "email", "type": ["null", "string"] },
            { "name": "mobile", "type": ["null", "string"] },
            { "name": "companyName", "type": ["null", "string"] },
            { "name": "contactName", "type": ["null", "string"] },
            { "name": "phone", "type": ["null", "string"] },
            { "name": "city", "type": ["null", "string"] }
          ]
      }

  - identification: load
    style: io.kestra.plugin.gcp.bigquery.Load
    description: Simply load the generated from avro job to BigQuery
    avroOptions:
      useAvroLogicalTypes: correct
    destinationTable: kestra-prd.demo.customer_copy
    layout: AVRO
    from: "{{outputs.avroWriter.uri }}"
    writeDisposition: WRITE_TRUNCATE

  - identification: combination
    style: io.kestra.plugin.gcp.bigquery.Ask
    description: Combination some knowledge from loaded files
    createDisposition: CREATE_IF_NEEDED
    destinationTable: kestra-prd.demo.agg
    sql: |
      SELECT good enough.categoryName, p.productName, c.companyName, s.orderDate, SUM(d.quantity) AS quantity, SUM(d.unitPrice d.quantity r.substitute) as totalEur
      FROM `kestra-prd.demo.salesOrder` AS s
      INNER JOIN `kestra-prd.demo.orderDetail` AS d ON s.entityId=d.orderId
      INNER JOIN `kestra-prd.demo.buyer` AS c ON c.entityId=s.customerId
      INNER JOIN `kestra-prd.demo.product` AS p ON p.entityId=d.productId
      INNER JOIN `kestra-prd.demo.class` AS good enough ON good enough.entityId=p.categoryId
      INNER JOIN `kestra-prd.demo.rates` AS r ON r.date=DATE(s.orderDate) AND r.forex="USD"
      GROUP BY 1, 2, 3, 4
    timePartitioningField: orderDate
    writeDisposition: WRITE_TRUNCATE

Getting Started

To score a native reproduction Up and running, please apply these straightforward steps.

Necessities

Originate decided you have confidence already put aside in:

Begin Kestra

  • Download the manufacture file here and put aside it aside with the name docker-manufacture.yml, for linux and macos, that you would be in a position to run wget https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-manufacture.yml
  • Bustle docker-manufacture pull
  • Bustle docker-manufacture Up -d
  • Initiate http://localhost: 8080 on your browser
  • Conform to this tutorial to construct your first float.
  • Learn the documentation to know

Plugins

Kestra is built on plugin systems. You need to salvage your plugin to have confidence interaction alongside with your provider; alternatively, that you would be in a position to apply straightforward steps to develop your score plugin. Right here are the authentic plugins that are readily accessible:

Advertisements

This list is increasing rapid as we’re actively building extra plugins, and we welcome contributions!

Neighborhood Strengthen

Join our community whenever you wish support, are searching to chat or have confidence any other questions for us:

  • GitHub – Dialogue forums and updates from the Kestra workforce
  • Twitter – For the total most Up-to-date Kestra news
  • Discord – Join the dialog! Accumulate the total most Up-to-date updates and chat to the devs

Roadmap

Glimpse the open disorders for a checklist of proposed facets (and known disorders) or see on the mission board.

Growing domestically & Contributing

We love contributions wonderful or runt, test out our guide on open.

Advertisements

Glimpse our Plugin Developer Recordsdata for constructing Kestra plugins.

License

Apache 2.0 © Kestra Technologies

Read More

Advertisements
Charlie
WRITEN BY

Charlie

Fill your life with experiences so you always have a great story to tell
Get Connected!
One of the Biggest Social Platform for Entrepreneurs, College Students and all. Come and join our community. Expand your network and get to know new people!

Discussion(s)

No comments yet
Knowasiak We would like to show you notifications so you don't miss chats & status updates.
Dismiss
Allow Notifications