2021-06-28

2021-09-30

structural types:

• VLQ
• length prefixed value array
• length prefixed byte array
• fixed length byte array
• fixed length value array

Parsing structure results to semantic type / value paris.

Purpose of structural types are to splice the bytes into values. Semantic types allow interpreting the values.

packing entity ids

Data segments could start with a dictionary of the 128 most common entity ids in the segment. The first bit of an entity id encoding in that segment then would tell wether the byte should be interpreted as a reference to this index or as a VLQ encoding of the stream id surrogate. If the value is not an index reference, the first VLQ is followed by the local entity id as VQL. Examples:

• 0000 0000: the first entity id in the dictionary.
• 0000 0001: the second entity id in the dictionary.
• 1000 0000 0000 0000: entity id 0/0 as stream id surrogate / local entity id pair.
• 1100 0001, 0000 0000, 0000 0000: entity id 256/0. The second bit specifies that the stream id surrogate continues on the second byte.

Using a simple VLQ pair to refer to stream id surrogate and local entity id pair could be enough in practice. That would most likely result in using only two bytes per attribute reference.

index transfer format

A dabric index could be encoded in a transfer format. The transfer format is expected to be read into an in memory data structure for usage. It does not have the row offset index as actual index segments.

The transfer file format has the following parts:

• stream id dictionary: a dictionary from small integer numbers to stream ids.
• column types array: value types of each column in the file.
• values: values in the index

value type definition format

A value type is identified by an entity id and has an associated structural type that defines the byte layout of the values of that type. The entity represents the semantic meaning of the values, while the structural type defines how the value should be parsed from a byte stream.

Value types can refer other value types. A date for example could consists of an array of three single byte integers and single byte integers are defined in the dabric prelude with entity id 0.

Structural type definitions have a textual syntax:

• n: VLQ
• b1: one byte value
• b2: two byte value
• b*: length prefixed byte array. the length is encoded as VLQ preceding the bytes.
• [2 n] array of two VLQ:s.
• [2 b] array of two one byte values
• [* b] length prefixed array of one byte values. The length is expressed as VLQ.
• [b b b] an array of three single byte values
• 0/0 type
• (0/0 b1) a single byte that has a semantic type denoted by the entity id 0/0
• (0/0 2 b1) an array of two single byte values with the semantic type 0/0
• Streams could be refered with their full id and entities with their aliases: dabric.net/ui8
• (person/person (person/name b*) {person/shoe-size b} a two item array

entity aliases

An entity could be given local aliases that uniquelly correspond to one local entity id. For example the single byte integer type could be given an alias string "ui8" in the dabric prelude and thus it could be refered as dabric.net/prelude/ui8.

This site is generated with zetgen