stream parsing

fawzi Joined: 02/20/08 Posts: 23

Posted: 01/05/09 15:39:23 I am writing a stream parser, this is quite similar to the iterators that inherit tango.io.stream.Iterator. the source.next method is quite nice, but I miss some more control, basically knowing if the slice that I get from the buffer is a (possibly) partial slice, a slice to EOF, or a maximal slice (due to buffer capacity). I want to know this because I want to parse things (like numbers) that are not explicitly terminated, so if I parse a number that ends at the end of the slice I do not know if I should ask to load more data or not (returning Eof if the file ends there would fail the match, even if it is actually ok). The basic interface is flexible enough, so at the moment I have solved this by writing my next method that takes a scanner that has an extra argument (Partial,Maximal,End) and creating a delegate without that argument to pass on, and using some extra logic. Still I wonder if something like that should not be part of the default interface (or I have missed something and there is a better way to do this).

Author	Message
kris Joined: 03/27/04 Posts: 581	Posted: 01/05/09 15:52:49 There's usually a tradeoff between efficiency, simplicity, and functionality, and we just need to find the right balance (or the flexibility to enable extensions). The (current) generic interface is derived from a desire for simplicity and efficiency, and perhaps those can be retained while providing additional functionality?
fawzi Joined: 02/20/08 Posts: 23	Posted: 01/05/09 17:10:23 This is my workaround... enum SliceExtent{ Partial, Maximal, ToEnd } final bool next (size_t delegate (void[],SliceExtent) scan) { SliceExtent sliceE=SliceExtent.Partial; if (source.position == 0 && source.writable == 0) sliceE=SliceExtent.Maximal; while (source.read(delegate size_t(void[] s){ return scan(s,sliceE);}) is Eof) { if (sliceE!=SliceExtent.Partial) return false; if (source.position != 0){ source.compress; } // read another chunk of data if (write(&source.read) is Eof) { sliceE=SliceExtent.ToEnd; } else if (source.writable == 0) { sliceE=SliceExtent.Maximal; } } return true; } to me it still looks reasonably simple... but never trust the writer... ;)
fawzi Joined: 02/20/08 Posts: 23	Posted: 01/09/09 16:46:14 well the issues were a little bit more complex, I discuss them in ticket #1428