tango.text.stream.StreamIterator

License:
BSD style: see license.txt

Version:
Initial release: December 2005

author:
Kris

class StreamIterator (T): InputStream, Buffered;

The base class for a set of stream iterators. These operate upon a buffered input stream, and are designed to deal with partial content. That is, stream iterators go to work the moment any data becomes available in the buffer. Contrast this behaviour with the tango.text.Util iterators, which operate upon the extent of an array.

There are two types of iterators supported; exclusive and inclusive. The former are the more common kind, where a token is delimited by elements that are considered foreign. Examples include space, comma, and end-of-line delineation. Inclusive tokens are just the opposite: they look for patterns in the text that should be part of the token itself - everything else is considered foreign. Currently tango.text.stream includes the exclusive variety only.

Each pattern is exposed to the client as a slice of the original content, where the slice is transient. If you need to retain the exposed content, then you should .dup it appropriately.

The content provided to these iterators is intended to be fully read-only. All current tokenizers abide by this rule, but it is possible a user could mutate the content through a token slice. To enforce the desired read-only aspect, the code would have to introduce redundant copying or the compiler would have to support read-only arrays.

See LineIterator, CharIterator, RegexIterator, QuotedIterator, and SimpleIterator

uint scan (void[] data);

The pattern scanner, implemented via subclasses

this(InputStream stream = null);

Instantiate with a buffer

StreamIterator set (InputStream stream);

Set the provided stream as the scanning source

T[] get ();

Return the current token as a slice of the content

StreamIterator push (T[] token);

Push one token back into the stream, to be returned by a subsequent call to next()

Push null to cancel a prior assignment

int opApply (int delegate(ref T[]) dg);

Iterate over the set of tokens. This should really provide read-only access to the tokens, but D does not support that at this time

int opApply (int delegate(ref int, ref T[]) dg);

Iterate over a set of tokens, exposing a token count starting at zero

int opApply (int delegate(ref int, ref T[], ref T[]) dg);

Iterate over a set of tokens and delimiters, exposing a token count starting at zero

T[] next ();

Locate the next token. Returns the token if found, null otherwise. Null indicates an end of stream condition. To sweep a conduit for lines using method next ():

                auto lines = new LineIterator!(char) (new FileConduit("myfile"));
                while (lines.next)
                       Cout (lines.get).newline;

Alternatively, we can extract one line from a conduit:

                auto line = (new LineIterator!(char) (new FileConduit("myfile"))).next;

The difference between next () and foreach() is that the latter processes all tokens in one go, whereas the former processes in a piecemeal fashion. To wit:

                foreach (line; new LineIterator!(char) (new FileConduit("myfile")))
                         Cout(line).newline;

uint set (T* content, uint start, uint end);

Set the content of the current slice to the provided start and end points

uint set (T* content, uint start, uint end, uint next);

Set the content of the current slice to the provided start and end points, and delimiter to the segment between end & next (inclusive)

uint notFound ();

Called when a scanner fails to find a matching pattern. This may cause more content to be loaded, and a rescan initiated

uint found (uint i);

Invoked when a scanner matches a pattern. The provided value should be the index of the last element of the matching pattern, which is converted back to a void[] index.

bool has (T[] set, T match);

See if set of characters holds a particular instance

bool consume ();

Consume the next token and place it in 'slice'. Returns true when there are potentially more tokens

IBuffer buffer ();

Buffered Interface

Return the associated buffer

IConduit conduit ();

InputStream Interface

Return the host conduit

uint read (void[] dst);

Read from conduit into a target array. The provided dst will be populated with content from the conduit.

Returns the number of bytes read , which may be less than requested in dst

void[] load (void[] dst = null);

Load the bits from a stream, and return them all in an array. The dst array can be provided as an option, which will be expanded as necessary to consume the input.

Returns an array representing the content, and throws IOException on error

InputStream clear ();

Clear any buffered content

void close ();

Close the input

anchorFromTitle('tango.text.stream.StreamIterator', 'source/', 'html');