Download Reference Manual
The Developer's Library for D
About Wiki Forums Source Search Contact

Detecting position and EOF when using Reader?

Moderators: larsivi kris

Posted: 10/06/08 11:50:56

I have a file-loading routine that's basically like this:

auto file = new FileConduit(infilename);
scope(exit) file.close();
auto reader = new Reader(file.input);

// Read a null-terminated string
// (about 50 bytes in my test file)

// Debug output
Stdout.formatln("pos {}, len {}",
                file.position(),
                file.length());

// Main read loop
while(file.position() < file.length())
{
	// Read a somewhat complex field,
	// the structure of which varies.
}

On my test file (about 6k), the while condition is never true. The debugging output shows that file.length() is correct, but the file.position() is at the EOF, even though I've only read ~50 out of ~6k bytes.

This suggests to me that Tango is doing some buffering, which is good, I want that buffering. The problem though is that when I try to get the position, I seem to be getting the position of the buffering mechanism (which doesn't help me) instead of the position of my Reader. This isn't too surprising, of course, since AIUI the buffering is being done inside Reader and is therefore outside the control of the FileConduit?. But since Reader doesn't expose a position property of its own, this makes detecting EOF (and seeking relative to the current position, which I'll also need) a bit of a problem when using Reader and buffering (The nature of the rest of the file-loading code rules out waiting for any sort of EOF exception).

So is there a way to get the Reader's version of position? I'm hoping I won't have to keep track of it manually.

Author Message

Posted: 10/06/08 15:41:40

You can check to see how many bytes are available for reading in the buffer as an additional check:

...
while(reader.buffer.readable > 0 || file.position < file.length)
{
...
}

You can also calculate the 'buffer' file position based on that:

uint bufPos = file.position - reader.buffer.readable;

Unfortunately, Reader cannot have positioning info builtin because the source may not support it.

-Steve

Posted: 03/27/09 03:16:34

(Resurrecting an old thread here...)

Soon after the above two posts, I changed the code above to use DataInput? instead of Reader because I found out that DataInput? was apparently the preferred method. But now in 0.99.8, DataInput? no longer has a buffer exposed (or a "io.stream.Buffered"), or, as far as I can tell, any other way of accessing the number of unread bytes remaining in the buffer/io.stream.Buffered/etc. But it looks like Reader still has this exposed.

First of all, is this whole sort of workaround still needed in 0.99.8, or is there now some way to check the current position with this issue of "some of the data may already have been read but just buffered up" all abstracted away? Do I need to switch back to using Reader, or is that heading towards deprecation?

Posted: 03/27/09 07:32:39 -- Modified: 03/27/09 07:41:39 by
Abscissa -- Modified 4 Times

Ok, after scouring the new tango.io.* source, I see what I need to do. Apparently, DataInput?'s ctor is designed to add a buffer, *but only* if it's being passed a stream that doesn't already have a buffer attached upstream (which it does by invoking BufferedInput?.create(...)). So my new working 0.99.8 code (adjusted from the sample above) is basically like this:

auto file = new File(infilename);
scope(exit) file.close();
auto readBuffer = new BufferedInput(file.input);
auto reader = new DataInput(readBuffer);

// Read a null-terminated string
// (about 50 bytes in my test file)

// Debug output
Stdout.formatln("pos {}, len {}",
                file.position(),
                file.length());

// Main read loop
while(file.position() - readBuffer.readable() < file.length())
{
	// Read a somewhat complex field,
	// the structure of which varies.
}

Also, it seems that using Reader is apparently still discouraged since merely importing it triggers the warning about io.Buffer being "semi-deprecated" in favor of io.stream.Buffered. Not a problem for me though since I'm still doing fine without it.