Automatic serialization library. Currently it has following 'backends': TextArchive?, JsonArchive? and BinaryArchive?.

Some of most important features:

• can work with arrays and as well with storages
• highly customizable
• thread/exception safe
• tracking of references (classes/pointers)
• user defined types versioning
• importing of old classes
• user defined constructors
• user defined loaders/dumpers
• different archive classes

There is quite big suite of unit tests for serialization. You can see serializer features in usage there. Serializer works also with storages.

Serialization with Doost serializer.

---

Main goals for serialization library

• Simple usage of library by users. Provide reasonable defaults. "Less code is more".
• Possibility to serialize/deserialize to/from string and as well to/from streams
• Versioning for each class definition. When a class definition changed, older files can still be imported to the new version of the class.
• Deep pointer save and restore. That is, save and restore of pointers saves and restores the data pointed to.
• Proper restoration of pointers to shared data.
• Data Portability - Streams of bytes created on one platform should be readable on any other. (- Not very well tested)
• Possibility to change serialization format without necessity of changing user serialization code.
• Non-intrusive serialization if possible.
• Easy creation of new archive formats.

Simple example

Imagine you have struct which keeps information about user and his password. You want to save it to string 'serialized'. Serialization of such a case is as easy as writing following code:

import std.stdio;

import doost.util.serializer.Serializer;
import doost.util.serializer.archive.JsonArchive;

struct LoginData {
    string user;
    string password;
}

void main() {
    LoginData logingData = {"key","secret"};
	string serialized;

    auto serializer = new Serializer!(JsonArchive);
    serialized = serializer.dump(logingData);
	writefln(serialized);
}

This simple example should work very well for most cases. Although there are sometimes cases when you want to have better control on what is serialized

Intrusive version

There are cases where you will need better control over what is serialized. You might want for example to skip some members or to encode them, so they will not be visible. For such a case you can use intrusive version of serialization. In our example in first case we will just skip serialization of password, so it will not be visible after serialization. The only thing changed from previous example is struct LoginData?:

struct LoginData {
    string user;
    string password;

    void describeUdt(T)(T arch) {
        arch.describe(user, "user");
    }
}

(In D1 it was not possible to pass member field name into template describe, so there is a need to pass it explicitly by user.)

Now output doesn't contain field password.

In case we would like to encode field password before serialization, it is necessary to use two template functions inside 'LoginData?' struct:

struct LoginData {
    string user;
    string password;

	//This is "just for fun" symmetric encoding algorithm
	ubyte[] encode(ubyte[] arr, int val) {
        ubyte[] result;
        ubyte b;
        foreach(v; arr) {
            if (v+val>255) b = v + val - 255;
            else
            if (v+val<0) b = v + val + 255;
            else b = v + val;

            result~=b;
        }
        return result;
    }

    void loadUdt(T)(T arch) {
        arch.describe(login, login.stringof);
        ubyte[] hashed;
        arch.describe(hashed, "password");
        hashed = encode(hashed, -2);
        password = cast(string)hashed;
    }

    void dumpUdt(T)(T arch) {
        arch.describe(login, login.stringof);
        ubyte[] hashed = cast(ubyte[])password;
        ubyte[] pom = encode(hashed, 2);
        arch.describe(pom, "password");
    }
}

Storage concept

While serializing to string is quite useful, I would guess that it's not most common use case. Probably most common use case when serializing data will be reading and writing files. For example it might be saving configuration of program. For such a cases we will need something more: storage concept.

What is Storage? Basically Storage is kind of stream. It consists of elements, which have same type. Storages can be used for writing and reading, so they define methods for getting elements and putting them. Normally during serialization only reading-methods, and during deserialization only writing-methods will be used. Storages can be defined in a very different ways e.g. they can be connected with only one file, or they can be connected with two or more files. That's completely implementation specific.

Every type which can be used as storage must have defined following methods:

interface Storage(ELEMENTTYPE) {
    alias ELEMENTTYPE[] STORAGETYPE;

	//Tests for end of stream. Returns true for if element on position 'pos' is beyond stream end
    bool eos(uint pos = 0);
	
	//Gets one element from beginning of stream
    ELEMENTTYPE get();
	
	//Gets 'no' elements from beginning of stream
    STORAGETYPE get(uint no);

	//Puts one element to storage
    void put(ELEMENTTYPE element);
	//Puts sequence of elements to storage
    void put(STORAGETYPE sequence);

	//Peeks for element on position 'pos'
    ELEMENTTYPE peek(uint pos=0);
	//Peeks for 'n' consecutive elements from position 'pos'
    STORAGETYPE peek(uint pos, uint n);

	//Returns pointer to 'frame' which is part of the stream currently kept in memory buffer
    STORAGETYPE- frame();
}

This interface is also implemented for strings thanks to definition in 'doost.storage.Storage' module, so every string is fulfilling requirements for storage.

Let's try to serialize our class to file:

import std.stdio;

import doost.util.serializer.Serializer;
import doost.util.serializer.archive.JsonArchive;
import doost.storage.Storage;
import doost.storage.FileStorage;

struct LoginData {
    string user;
    string password;
}

void main() {
    LoginData loginData = {"key","secret"};
	string serialized;

    auto jsonstreamserializer = new Serializer!(JsonArchive, Storage!(char));
    auto storage = new FileStorage!(char)("login_data.txt");

    storage.open;
    auto res = jsonstreamserializer.dump(loginData, storage);
    storage.close;

    serialized = cast(char[])std.file.read("login_data.txt");
	writefln(serialized);
}

Using storages it should be possible to define e.g. NetworkStorage? which will deserialize objects directly from network stream of bytes.

Struct/Class Versioning

Library is designed to allow several versions of classes/structs to exist. This can be useful in case when class was extended with additional fields, but we still need to read old, serialized data. It is possible to achieve it in simple way. Let's assume that we changed our LoginData? into class, and added additional string field: division.

import std.stdio;

import doost.util.serializer.Serializer;
import doost.util.serializer.archive.JsonArchive;
import doost.storage.Storage;
import doost.storage.FileStorage;

struct LoginDataOld {
    string user;
    string password;
}

class LoginData {
    static uint versionUdt = 1;
    string user;
    string password;
    string division;

    void transformUdt(T)(T arch, ref LoginData val, uint ver) {
        switch(ver) {
            case 0: LoginDataOld v0;
                    arch.transform(v0);
                    val.user = v0.user;
                    val.password = v0.password;
                    val.division = "sales";
                    break;
            default: throw new Exception("Unknown VersionClass version.");
        }
    }
}

void main() {

    string loginDataOld = `{
        "LoginData" : {
            "user" : "key",
            "password" : "secret"
        }
    }`;

	string serialized;

    auto serializer = new Serializer!(JsonArchive);
    auto loginData = serializer.load!(LoginData)(loginDataOld);

    serialized = serializer.dump(loginData);
	writefln(serialized);
}

To import old data it is enough to:

1. Rename old class/struct definition e.g. suffixing it with word 'Old'
2. Add to new class: 'static uint versionUdt = 1;' with version of current class
3. Add to new class template method: 'void transformUdt(T)(T arch, ref LoginData? val, uint ver);'

That should be enough to transform old classes to new ones.

---

TO BE CONTINUED...

---

DEPENDENCIES

• DMD 1.x
• Library Std2

BUGS

• Serialization from base class is not possible

MISSING FEATURES OF DMD

• Creation of class/struct instance without default/public constructor: Object.factory() (?)
• Getting name of most derived class when having just base class: already fixed in 2.0 with typeid() giving most derived type (?)
• It's very easy to make error in templates describeUdt/loadUdt/dumpUdt/transformUdt as they are discovered using SFINAE - single syntax error makes it stop working.