BDIO in Julia

This package provides an interface to read/write BDIO (Binary Data Input/Output) files in Julia.

Getting started

Here we write a file with two records:

A record of type BDIO_BIN_INT64LE that contains the first 1000 numbers. This record has user info 1.
A record of type BDIO_BIN_F64LE that contains a total of 2000 numbers: 1000 random numbers and then the same numbers with opposite sign. This record has user info 2. Note that writing data to a record can be done with more than one call to BDIO_write!.

The MD5 sum of both records is stored in the same BDIO file thanks to the calls to BDIO_write_hash!. These are written in special BDIO records themselves, with user info 7.

julia> using BDIO

julia> BDIO_set_user("alberto")
true

julia> BDIO_set_host("laptop")
true

julia> fb  = BDIO_open("foo.bdio", "w", "Test file")

julia> BDIO_start_record!(fb, BDIO_BIN_INT64LE, 1, true)
true

julia> BDIO_write!(fb, collect(1:1000))
true

julia> BDIO_write_hash!(fb)
true

julia> BDIO_start_record!(fb, BDIO_BIN_F64LE, 2, true)
true

julia> vec1  = randn(1000);

julia> vec2  = similar(vec1);

julia> vec2 .= .- vec1;

julia> BDIO_write!(fb, vec1)
true

julia> BDIO_write!(fb, vec2)
true

julia> BDIO_write_hash!(fb)
true

julia> BDIO_close!(fb)
0-element Array{BDIO.Record,1}

In total the file has 4 records (2 data records, 2 checksum records). We can see the contents using the lsbdio tool.

$ lsbdio foo.bdio 
ID     record type       size   uinf starts with                           long
0      header v 1          alberto@laptop           Wed Jul  1 09:31:24 2020   
1      record i64 le    8000 byte  1 1 2 3 4 5 6 7 8 9 10 11 12 13            x
2      record MD5-h       20 byte  7 F41BD4FBFC77E5D5ABDF7300F42C1591         -
3      record f64 le   16000 byte  2 1.378609e-01 -2.838856e-01               x
4      record MD5-h       20 byte  7 1A5636335356BB004E489AB5DAE89B5D         -

Now let's read the data. We travel the file using BDIO_seek! until we find the records that we are looking for: user info 1 for the integer data and user info 2 for the real data. Again note that reading a single record can be done in chunks with multiple calls to BDIO_read.

julia> using BDIO

julia> fb2 = BDIO_open("foo.bdio", "r")

julia> global isum = 0
0

julia> global fsum = 0.0
0.0

julia> while BDIO_seek!(fb2)
           if BDIO_get_uinfo(fb2) == 1
               idt = similar(Array{Int64,   1}, 100)
               for i = 1:10
                   BDIO_read(fb2, idt)
                   global isum += sum(idt)
               end
           elseif BDIO_get_uinfo(fb2) == 2
               fdt = similar(Array{Float64, 1}, 100)
               for i = 1:20
                   BDIO_read(fb2, fdt)
                   global fsum += sum(fdt)
               end
           end
       end

julia> println("Better be zero: ", fsum)
Better be zero: -1.3322676295501878e-15

julia> println("Better be zero: ", (2*isum - 1000*1001))
Better be zero: 0

Setting up global information

BDIO.BDIO_set_user — Function

BDIO_set_user(us::String)

Set user name globally for writing BDIO files.

Arguments

us: Set us as the user name when writing BDIO files.

Examples

julia> BDIO_set_user("alberto")

BDIO.BDIO_set_host — Function

BDIO_set_host(us::String)

Set host machine globally for writing BDIO files.

Arguments

us: Set us as the host when writing BDIO files.

Examples

julia> BDIO_set_host("HLRN")

Openning/closing `BDIO` files

BDIO.BDIO_open — Function

BDIO_open(fname::String, mode::String, protocol_info::String="")

Opens a BDIO file and returns the BDIO handle.

The file can be opened in several modes:

Write mode ("w"): The file is created and a header written. If the file exists an error is printed.
Delete mode ("d"): The file is created and a header written. If the file exists it is overwritten.
Append mode ("a"): The file is created if it does not exist, or opened for appending if the file exists.
Read mode ("r"): The file is opened for reading.

Arguments

fname: File name
mode: The mode in which the file is opened. See above.
protocol_info: Only used when the file is created (i.e. "w" or "d" modes) and labels the file.

Examples

julia> fb = BDIO_open("new_file.bdio", "w", "Test file")

Returns

A BDIOstream type.

BDIO.BDIO_close! — Function

BDIO_close!(fb::BDIOstream)

Closes the file associated with fb and clears the record database

Arguments

fb: A BDIOstream type. It must be associated with a file.

Examples

```julia-repl julia> fb = BDIOopen("newfile.bdio", "w", "Test file") julia> BDIO_close(fb)

Writing data to `BDIO` files

BDIO.BDIO_start_record! — Function

BDIO_start_record!(fb::BDIOstream, ifmt, iuinfo, long::Bool = false)

Start a new BDIO record at the end of the file.

Arguments

fb: A BDIOstream type. It must be associated with a file in either w or a mode
ifmt: Format. Currently the supported formats are
- BDIO_BIN_GENERIC: Generic binary data
- BDIO_ASC_EXEC: ASCII Executable (i.e. sh script)
- BDIO_BIN_INT32BE: 32-bit integer in Big Endian format
- BDIO_BIN_INT32LE: 32-bit integer in Little Endian format
- BDIO_BIN_INT64BE: 64-bit integer in Big Endian format
- BDIO_BIN_INT64LE: 64-bit integer in Little Endian format
- BDIO_BIN_F32BE: 32-bit float in Big Endian format
- BDIO_BIN_F32LE: 32-bit float in Little Endian format
- BDIO_BIN_F64BE: 64-bit float in Big Endian format
- BDIO_BIN_F64LE: 64-bit float in Little Endian format
- BDIO_ASC_GENERIC: ASCII text file
- BDIO_ASC_XML: Plain XML data
iuinfo: Integer in the range 0-15. A user specified label to help indentifying the record
long (optional): If true create a long record. To store more than 1048575 bytes of data ($\approx 1\, {\rm MB}$), a long record is required. Default value of false.

Examples

julia> fb = BDIO_open("new_file.bdio", "w", "Test file")
julia> BDIO_start_record!(fb, BDIO_BIN_F32, 2)

BDIO.BDIO_write! — Function

BDIO_write!(fb::BDIOstream,data,hash::Bool=true)

Write data to BDIO file to the end of the last record.

Arguments

fb: A BDIOstream type. It must be associated with a file in either w or a mode.
data: Data to write to file.
hash (optional): Bool. If true, data will be checksummed before writtento file. The hash can be stored in a BDIO record by calling BDIO_write_hash(...)

Examples

julia> # Write 1000 normal random numbers to file `randoms.bdio`
julia> fb = BDIO_open("randoms.bdio", "w", "File with random numbers")
julia> BDIO_start_record!(fb, BDIO_BIN_F32, 2)
julia> BDIO_write!(fb, randn(1000))

BDIO.BDIO_write_hash! — Function

BDIO_write_hash!(fb::BDIOstream)

Write the MD5 checksum of the actual record as a new record.

Arguments

fb: A BDIOstream type. It must be associated with a file in either w or a mode.

Examples

julia> # Write 1000 normal random numbers to file `randoms.bdio` with checksum
julia> fb = BDIO_open("randoms.bdio", "w", "File with random numbers")
julia> BDIO_start_record!(fb, BDIO.BDIO_BIN_F32, 2)
julia> BDIO_write!(fb, randn(1000))
julia> BDIO_write_hash!(fb)

Reading data from `BDIO` files

BDIO.BDIO_read — Function

BDIO_read(fb::BDIOstream, vdata::Vector, n::Int64 = 0)

Read data from BDIO file.

Arguments

fb: A BDIOstream type. It must be associated with a file in either w or a mode.
data[:]: A Vector of data to read.
n (optional): Integer. If present read n elements of data[:] from file

Examples

julia> # Real 1000 floats from the first record of file `randoms.bdio`
julia> fb = BDIO_open("randoms.bdio", "r")
julia> BDIO_seek!(fb)
julia> data = similar({Array, 1}, 1000)
julia> BDIO_read(fb, data)

BDIO.BDIO_seek! — Function

BDIO_seek!(fb::BDIOstream, icnt::Int = 1)

Move the read position backward/forward icnt records

Arguments

fb: A BDIOstream type. It must be associated with a file.
icnt (optional): number of records to move forward (if icnt>0) or backwards (icnt<0). If icnt=0 move to the first record. The default value is +1.

Examples

julia> # count number of records in a file.
julia> fb = BDIO_open("randoms.bdio", "r")
julia> count = 0
julia> while BDIO_seek!(fb)
julia>     count += 1
julia> end

Obtaining information about `BDIO` records

BDIO.BDIO_get_uinfo — Function

BDIO_get_uinfo(fb::BDIOstream)

Returns the user provided info of each record

Arguments

fb: A BDIOstream type. It must be associated with a file.

Examples

julia> # Write length of all records
julia> fb = BDIO_open("randoms.bdio", "r")
julia> while BDIO_seek!(fb)
julia>     count += 1
julia>     println("Record: ", count, " user info: ", BDIO_get_uinfo(fb))
julia> end

BDIO.BDIO_get_len — Function

BDIO_get_len(fb::BDIOstream)

Returns the len (in bytes) of the current record

Arguments

fb: A BDIOstream type. It must be associated with a file.

Examples

julia> # Write length of all records
julia> fb = BDIO_open("randoms.bdio", "r")
julia> while BDIO_seek!(fb)
julia>     count += 1
julia>     println("Record: ", count, " length: ", BDIO_get_len(fb), " bytes")
julia> end

BDIO.BDIO_get_fmt — Function

BDIO_get_fmt(fb::BDIOstream)

Returns the format of the current record

Arguments

fb: A BDIOstream type. It must be associated with a file.

Examples

julia> # Print all BDIO_BIN_GENERIC records
julia> fb = BDIO_open("randoms.bdio", "r")
julia> while BDIO_seek!(fb)
julia>     count += 1
julia>     if (BDIO_get_fmt(fb) == BDIO_BIN_GENERIC)
julia>         println("Record: ", count, " is BIN_GENERIC")
julia>     end
julia> end