Download Reference Manual
The Developer's Library for D
About Wiki Forums Source Search Contact

stream parsing

Moderators: larsivi kris

Posted: 01/05/09 15:39:23

I am writing a stream parser, this is quite similar to the iterators that inherit tango.io.stream.Iterator.

the source.next method is quite nice, but I miss some more control, basically knowing if the slice that I get from the buffer is a (possibly) partial slice, a slice to EOF, or a maximal slice (due to buffer capacity).

I want to know this because I want to parse things (like numbers) that are not explicitly terminated, so if I parse a number that ends at the end of the slice I do not know if I should ask to load more data or not (returning Eof if the file ends there would fail the match, even if it is actually ok).

The basic interface is flexible enough, so at the moment I have solved this by writing my next method that takes a scanner that has an extra argument (Partial,Maximal,End) and creating a delegate without that argument to pass on, and using some extra logic.

Still I wonder if something like that should not be part of the default interface (or I have missed something and there is a better way to do this).

Author Message

Posted: 01/05/09 15:52:49

There's usually a tradeoff between efficiency, simplicity, and functionality, and we just need to find the right balance (or the flexibility to enable extensions). The (current) generic interface is derived from a desire for simplicity and efficiency, and perhaps those can be retained while providing additional functionality?

Posted: 01/05/09 17:10:23

This is my workaround...

    enum SliceExtent{ Partial, Maximal, ToEnd }
    
    final bool next (size_t delegate (void[],SliceExtent) scan)
    {
        SliceExtent sliceE=SliceExtent.Partial;
        if (source.position == 0 && source.writable == 0) sliceE=SliceExtent.Maximal;
        while (source.read(delegate size_t(void[] s){ return scan(s,sliceE);}) is Eof)
        {
            if (sliceE!=SliceExtent.Partial) return false;
            if (source.position != 0){
                source.compress;
            }
            // read another chunk of data
            if (write(&source.read) is Eof) {
                sliceE=SliceExtent.ToEnd;
            } else if (source.writable == 0) {
                sliceE=SliceExtent.Maximal;
            }
        }
        return true;
    }

to me it still looks reasonably simple... but never trust the writer... ;)

Posted: 01/09/09 16:46:14

well the issues were a little bit more complex, I discuss them in ticket #1428