Kaitai Struct: A brand contemporary manner to construct parsers for binary structures

85
Kaitai Struct: A brand contemporary manner to construct parsers for binary structures

[Kaitai logo]

A brand contemporary manner to construct parsers for binary structures.

Finding out and writing binary formats is laborious, particularly if
it’s an interchange structure that ought to work across a mess of
platforms and languages.

Personal you ever found your self writing repetitive,
error-susceptible and laborious-to-debug code that reads binary data structures
from files or network streams and a technique or the opposite represents them in reminiscence for
more straightforward entry?

Kaitai Struct tries to rep this job more straightforward — you handiest must
listing the binary structure once after which each person can exhaust it from their
programming languages — hostile-language, hostile-platform.

Kaitai Struct is a declarative language used to listing varied
binary data structures, specified by files or in reminiscence: i.e. binary
file formats, network circulate packet formats, etc.

The predominant opinion is that a explicit structure is described in Kaitai
Struct language (.ksy file) after which is also compiled with
ksc into offer files in a single of the supported programming
languages. These modules will encompass a generated code for a parser
that can read the described data development from a file or circulate and give
entry to it in a advantageous, easy-to-comprehend API.

The exhaust of KS in your challenge

On the total, the exhaust of formats described in KS in your challenge entails the
following steps:

  • Characterize the structure — i.e. construct a .ksy file
  • Utilize a visualizer to debug the structure and be clear it parses data properly
  • Assemble the .ksy file precise into a target language offer file and encompass
    that file into your challenge
  • Add the KS runtime library to your explicit language into your
    challenge (don’t misfortune, it’s exiguous and it’s there largely to rep clear
    readability of generated code)
  • Utilize the generated class(es) to parse your binary file or circulate and
    entry its elements

Overview out the documentation for additional info.

meta:
  identification: tcp_segment
  endian: be
seq:
  - identification: src_port
    kind: u2
  - identification: dst_port
    kind: u2
  - identification: seq_num
    kind: u4
  - identification: ack_num
    kind: u4
public class TcpSegment extends KaitaiStruct {
    // ...
    inner most void _read() throws IOException {
        this.srcPort=_io.readU2be();
        this.dstPort=_io.readU2be();
        this.seqNum=_io.readU4be();
        this.ackNum=_io.readU4be();
    }
    // ...

Put in thoughts this uncomplicated .ksy structure description file that
describes the header of a GIF image file:

meta:
  identification: gif
  file-extension: gif
  endian: le
seq:
  - identification: header
    kind: header
  - identification: logical_screen
    kind: logical_screen
kinds:
  header:
    seq:
      - identification: magic
        contents: 'GIF'
      - identification: version
        dimension: 3
  logical_screen:
    seq:
      - identification: image_width
        kind: u2
      - identification: image_height
        kind: u2
      - identification: flags
        kind: u1
      - identification: bg_color_index
        kind: u1
      - identification: pixel_aspect_ratio
        kind: u1

It announces that a GIF file in most cases has a .gif extension and makes exhaust of
miniature-endian integer encoding. The file itself begins with two
blocks: first comes header after which comes logical_screen:

  • “Header” consists of a “magic” string of 3 bytes (“GIF”) that
    identifies that it’s a GIF file starting after which there are 3 extra
    bytes that identify the structure version (87a or 89a).
  • “Logical track descriptor” is a block of integers:
    • image_width and image_height are 2-byte unsigned ints
    • flags, bg_color_index and pixel_aspect_ratio draw stop 1-byte
      unsigned ints each and each

This .ksy file is also compiled
into gif.cpp / Gif.cs / Gif.java
/ Gif.js / gif.nim / Gif.pm
/ Gif.php / gif.py / gif.rb and
then one can today load a .gif file and entry, as an illustration, its
width and top.

std::ifstream ifs("path/to/some.gif", std::ifstream::binary);
kaitai::kstream ks(&ifs);
gif_t g=gif_t(&ks);

std::cout image_width() image_height() 
Gif g=Gif.FromFile("path/to/some.gif");

Console.WriteLine("width=" + g.LogicalScreen.ImageWidth);
Console.WriteLine("height=" + g.LogicalScreen.ImageHeight);
file, err :=os.Open("path/to/some.gif")
g :=NewGif()
err=g.Read(kaitai.NewStream(file), g, g)

fmt.Printf("width=%dn", g.LogicalScreen.ImageWidth)
fmt.Printf("height=%dn", g.LogicalScreen.ImageHeight)
Gif g=Gif.fromFile("path/to/some.gif");

System.out.println("width=" + g.logicalScreen().imageWidth());
System.out.println("height=" + g.logicalScreen().imageHeight());
var g=new Gif(new KaitaiStream(someArrayBuffer));

console.log("width=" + g.logicalScreen.imageWidth);
console.log("height=" + g.logicalScreen.imageHeight);
let g=Gif.fromFile("path/to/some.gif")

echo "width=" & $g.logicalScreen.imageWidth
echo "height=" & $g.logicalScreen.imageHeight
my $g=Gif->from_file("path/to/some.gif");

print("width=", $g->logical_screen()->image_width(), "n");
print("top=", $g->logical_screen()->image_height(), "n");
$g=Gif::fromFile("path/to/some.gif");

print("width=" . $g->logicalScreen()->imageWidth() . "n");
print("top=" . $g->logicalScreen()->imageHeight() . "n");
g=Gif.from_file("path/to/some.gif")

print("width=%d" % (g.logical_screen.image_width))
print("top=%d" % (g.logical_screen.image_height))
g=Gif.from_file("path/to/some.gif")

puts "width=#{g.logical_screen.image_width}"
puts "top=#{g.logical_screen.image_height}"

For certain, this situation reveals handiest a very restricted subset of what Kaitai
Struct can blueprint. Please refer to the documentation for additional
insights.

2021-05-02 — Briefly the exhaust of GitHub Releases for compiler distributions

As of , JFrog Bintray (distribution provider where we hosted the compiler artifacts for years) has been sunset, so we moved all precise compiler variations to GitHub Releases in the kaitai_struct_compiler repository. The installation instructions below collect been updated accordingly. Building (unstable) builds that collect been hosted on Bintray (Linux .deb and Common .zip) aren’t accessible precise now (till we teach up the contemporary distribution system).

We’re currently setting up but any other repository to substitute Bintray, which is also accessible on a personalised domain functions.kaitai.io to be future-proof, so care for tuned!

The precise kaitai-struct-compiler variations are currently uploaded to https://github.com/kaitai-io/kaitai_struct_compiler/releases (see box above). Correct collect the .deb equipment and set up it:

curl -LO https://github.com/kaitai-io/kaitai_struct_compiler/releases/collect/0.9/kaitai-struct-compiler_0.9_all.deb
sudo succesful-rep set up ./kaitai-struct-compiler_0.9_all.deb

Requirements

  • .deb-based mostly Linux distribution (Debian, Ubuntu, etc)

There would possibly be a formula now accessible within Homebrew that you simply are going to be in a location to exhaust to set up
kaitai-struct-compiler:

brew set up kaitai-struct-compiler

Dwelling windows variations are accessible in as an MSI structure installer. When which that you simply can perchance perchance admire a conveyable version that requires no installation, collect our smartly-liked .zip ticket as a change.

Download — precise v0.9, 9.8 MiB

Download — newest pattern (unstable) ticket

“Common” builds are downloadable as
a .zip file that entails your total required
.jar files bundled and launcher scripts
for Linux / macOS / Dwelling windows systems. No
installation required, one can ravishing unpack
and flee it.

Download — precise v0.9, 9.5 MiB

Downloadnewest pattern (unstable) ticket — currently not accessible (see box above)

When you come by to ticket your instruments from offer, or ravishing must gape how KS works, the simplest manner to take a look at out the total challenge is to collect the predominant (umbrella) repository that already entails all other elements as submodules. Utilize:

git clone --recurse-submodules https://github.com/kaitai-io/kaitai_struct.git

When you already cloned the challenge and forgot --recurse-submodules, flee

git submodule update --init --recursive

Alternatively, one can take a look at out particular person sub-projects that consitute the Kaitai Struct suite. Watch the GitHub challenge page for main aspects.

Licensing

Kaitai Struct is free and begin-offer tool, licensed below the
following terms:

work in growth

Read More

Vanic
WRITTEN BY

Vanic

β€œSimplicity, patience, compassion.
These three are your greatest treasures.
Simple in actions and thoughts, you return to the source of being.
Patient with both friends and enemies,
you accord with the way things are.
Compassionate toward yourself,
you reconcile all beings in the world.”
― Lao Tzu, Tao Te Ching