Download Reference Manual
The Developer's Library for D
About Wiki Forums Source Search Contact

how to fill the structure from file?

Moderators: larsivi kris

Posted: 07/15/08 09:41:37

Hello. What is the best way to fill the structure

struct S {
 int a, b, c;
}

from file?

Author Message

Posted: 07/15/08 11:22:26

Doost has a serializer. I'm also working on one, although it can't yet serialize structs. I don't think there's anything in tango.

Posted: 07/17/08 06:25:22

purely the struct content, with no conversion, then use TypedStream?. Otherwise, DataStream? is usually quite effective

Posted: 08/03/08 05:32:15

I'm a D beginner, and this took me a little while to get right. This topic may be old, but maybe sometime this month another beginner like me might want to know this. In any case, I still have questions about this topic!

First of all, I have a file that is binary data. At the start of the file is an int saying how many structs are in the file, and following that is the data that would fill an array of those structs.

My first attempt was like this:

auto inputStream = new DataInput( new FileInput("data/Data") );
int howManyStructs = inputStream.getInt();
MyStruct[] structArray = new MyStruct[howManyStructs];
Stdout.format("array length: {}", structArray.length).newline;
int numberRead = inputStream.read(structArray);
Stdout.format("number read: {}", numberRead).newline;

This failed miserably, because the stream was now offset by an additional int.sizeof. The read() method expects there to be a leading int, skips that int, and then starts reading. So if I take that int from the stream and then call read, it skips an "int" and then starts reading. Compounding this was that if I didn't read that first int, how was I to know how big of an array to allocate? I could just allocate an arbitrarily large one, skip the getInt() call, and then just call read(), but that feels like a dodgy practise. Furthermore, the get() method isn't useful for this because the get() method takes the initial int as the number of bytes to read, not as the number of structs; I'd wind up under-reading by a very large amount. Now, if I could rewind the buffer after having done that getInt(), then read() would be great. But I don't know how to.

So I looked at DataStream?.d to see if they call a method similar to C's fread that I could use. Sure enough, it does. Which lead to my next approach:

auto file = new FileConduit("data/Data");
auto buffer = new Buffer(file);
int firstInt = 99998888; // just some dummy value
buffer.readExact(&firstInt, firstInt.sizeof);

Stdout.format("int read by Buffer: {}", firstInt).newline;
MyStruct[] structArray = new MyStruct[firstInt];
buffer.readExact(structArray.ptr, MyStruct.sizeof * firstInt);

I had wanted to use file.input.read() so that I wouldn't have to take the additional step of making buffer, but the problem with using file.input.read() is that it'd behave like DataInput?'s read() method (i.e. tries to put the entire file contents into the array), and that didn't really improve my situation from my first attempt. So in any case, what I ended up doing was making an fread-alike in D.

A question that I still have is whether I could've gotten around concocting such code to simulate fread, given the same binary data file. Because of the leading integer in the data file, TypedStream? couldn't be used. FileConduit? grants a seek() method, but the utility of that is not so great for me because I don't know how to make FileConduit? read a specific amount (e.g. just the leading int) and I end up needing to make a Buffer anyway. Is a nicer way possible within the tango library?

Posted: 08/03/08 16:02:49

You should be able to do something like this:

auto data = new DataFileInput ("myfile.dat");
auto type = new TypedInput!(MyStruct) (data);

auto array = new MyStruct [data.getInt];
foreach (ref element; array)
         type.read (element);

Another approach might be to read the entire file, sans header:

MyStruct[] array;

foreach (element; new TypedInput!(MyStruct) (new FileInput("myfile.dat")))
         array ~= element;

Hope that helps?

Posted: 08/05/08 02:21:34 -- Modified: 08/05/08 02:29:23 by
lower_bowl

Thanks for the reply!

The code you posted didn't work as is on my install, but a slight modification does:

    auto file = new FileInput ("data/Data");
    auto data = new DataFileInput(file);
    auto type = new TypedInput!(MyStruct) (data);

    auto array = new MyStruct [data.getInt];
    foreach (ref element; array)
             type.read (element);

DataFileInput? wouldn't accept a string path as its parameter, otherwise this is much nicer!

Posted: 09/19/08 15:31:12

in this file TypedInput? work fine.

module write;

import tango.io.Stdout;
import tango.io.stream.FileStream;
import tango.io.stream.DataFileStream;
import tango.io.stream.TypedStream;
import tango.io.Buffer;

struct Bata{
    int id;
    wchar[] name;
}

void main(){
    int len = 9;
    Bata x;
    auto file	= new FileOutput("1.dat");
    auto data	= new DataFileOutput(file);
    auto oot = new TypedOutput!(Bata)(data);
    for(int i=0;i<len;i++){
	x.name	= "char " ;
	x.id	= i;
	oot.write(x);
    }
    data.flush.close;
    oot.buffer.clear;
    oot.close;
    
    auto file2 = new FileInput ("1.dat");
    auto data2 = new DataFileInput(file2);
    auto type2 = new TypedInput!(Bata) (data2);
    
    Bata el;
    while(type2.read (el)){
	Stdout(el.id)(" ")(el.name)("\n");
    }
    
}

in this file return different data

module read;

import tango.io.Stdout;
import tango.io.stream.FileStream;
import tango.io.stream.DataFileStream;
import tango.io.stream.TypedStream;
import tango.io.Buffer;

struct Bata{
    int id;
    wchar[] name;
}



void main(){
    auto file2 = new FileInput ("1.dat");
    auto data2 = new DataFileInput(file2);
    auto type2 = new TypedInput!(Bata) (data2);
    
    Bata el;
    while(type2.read (el)){
	Stdout(el.id)(" ")(el.name)("\n");
    }
}

why the same code return different?

I have done something wrong, where it?

Posted: 09/20/08 03:29:25

I suspect the problem lies in the use of your 'dynamic' wchar[], since TypedStream deals only with the raw structure data. In other words, the wchar[] is likely written as the length and pointer combination, rather than chasing down the pointer and writing the contents held there (If you know C, then this would be akin to serializing a pointer value).

Using a fixed-length array would perhaps do what you expect?

struct Bata{
    int id;
    wchar[20] name;
}

Alternatively, DataStream might be what you're looking for?

Posted: 09/22/08 12:00:03

I've now got a fairly simple file IO serializer which will handle strings (although not generic pointers or classes), sub-structs and the basic types. Besides tango, dependencies are quite light (4 modules from my project): http://dsource.org/projects/mde/browser/mde/file/ssi.d

Usage is quite simple (from the unittest):

struct A {
    float x;
    dchar y;
    long z;
}
A a;
a.x = 0.0f;
a.y = '搀';
a.z = (cast(long) uint.max) + 1;

FilePath path = FilePath ("SSIUnitTest.ssi");
write (path, a);
assert (a == read!(A)(path));

Posted: 09/22/08 18:29:31

There's been a variety of attempts to do something like this via meta-programming (lots of templates), and I think the most successful I've seen originated with h3r3tic. Problem is that they're not necessarily the best way to go about things, and they're a bitch for the user to comprehend/debug when something goes wrong or doesn't compile properly :(