Version 6 (modified by JJR, 17 years ago)
More fixes

Leave Comments, Critiques, and Suggestions Here?

Tango Input and Output

Conduits

Tango IO is stream oriented; each stream end-point is described by a Tango construct called a conduit. For example, there are conduits for communicating with files, sockets, the console, and so on. Some conduits require explicit connections to the end-point, whereas others connect implicitly – socket conduits are of the explicit variety, while file and console conduits are usually implicit.

Conduits are bi-directional; supporting contiguous reads and writes. There are conduit extensions in a related IO package to enable a variety of concurrent read and write operations, enabled via various OS-specific facilities (see tango.io.selector). Regardless of how the conduit operates, an application interacts via a pair of read/write methods, each accepting a void array and returning an integer representing the quantity populated or consumed respectively. Note that the return value may, in some cases, be less than the array-length provided. In particular, the reserved constant IConduit.Eos will be returned for end-of-stream conditions:

uint read (void[] dst);
uint write (void[] src);

Additional methods are provided to fully populate a provided array and to fully flush an array. These two methods may stall whilst completing, depending upon the conduit end-point, and are provided for the sake of convenience. There is also a method to copy another conduit content:

uint fill (void[] dst);
bool flush (void[] src);
IConduit copy (IConduit src);

Conduits support the notion of stream-filters; one or more filters may be attached to the conduit and each may intercept and potentially mutate data as it flows through them. Filters are connected in reverse order via this method:

void attach (IConduitFilter filter);

Each conduit should be explicitly closed by the application in order to perform a conduit-specific set of cleanup actions and to release any attached filters. Conduit Exceptions

As a rule, conduits do not throw exceptions because some error conditions are often considered valid. For example, IConduit.Eos is generally returned to the application intact.

Protocols

Conduit provides typeless data streaming only. Tango provides support for typed streaming via a reader and writer pair; readers extracts data-types from a stream whilst writers injects data types into a stream. Each of the native D types is supported, along with their one-dimensional array variants. Readers and writers have variants to converse in text or binary format, handle endian conversion and so on. Together, a reader & writer pair defines a transfer protocol – data written/encoded by one particular protocol can be read/decoded correctly only by the same protocol.

Protocols are often attached to a conduit, where the intent is to stream native D types to and/or from a file or socket; a conduit may be attached at construction time. In such cases, interacting with the protocol will access the attached conduit accordingly.

Some protocols treat arrays in a specific manner: they prefix each with the number of elements contained (this is not a byte count, and thus the stream is not self-navigable). When reading, each protocol normally performs array allocation on behalf of the application, utilizing said element count. While the default allocation strategy is to assign via the heap, the strategy itself is configurable and a handful of implementations are provided to choose from. One to note is the null strategy, where arrays are instead managed by the application and the length prefix is assumed to be missing from the stream. This allows for those cases where, for example, an array length might be declared in a header.

In contrast to the vararg style popularized by printf/readf et al, protocol methods are discrete for each element read or written, and support call-chaining. Thus, basic protocol usage can be illustrated as shown below where input & output represent a protocol reader & writer pair, and where opCall is leveraged in a symmetrical style known as whisper:

int i;  
double d;  
char[] text;  

output  (i) (d) (text);  
input    (i) (d) (text);

Tango protocols support a simple object serialization strategy, where compatible objects implement a specific interface. Such objects are capable of reading and/or writing their own content via the relevant interface method:

void read (IReader input);
void write (IWriter output);

The interfaces involved are IReadable and IWritable, each of which declares one method relating to their behavior. Compatible objects are handled in a manner similar to native data types. Following the prior example:

class  Wumpus : IReadable, IWritable   
{  
    void read (IReader input) {}  
    void write (IWriter output) {}  
}  

auto wumpus = new Wumpus;  
output (wumpus);  
input   (wumpus);

Protocol Exceptions

Protocols generally expect data to be available when an application asks for it. Thus, exceptions are thrown on unexpected end-of-stream conditions. This is quite unlike conduit IO, which is stream oriented rather than element oriented.

Buffers

Protocols require IO buffering to operate efficiently, and the Buffer class fills this role. Protocol buffering enables an underlying conduit to read and write large chunks of data instead of discrete data elements, provide a temporary space for specific protocols to mutate content efficiently, and can support efficient mapping of data-record content. Buffers are full duplex, and maintain distinct read and write locations; hence it is possible to be writing near the end of a buffer whilst reading from the beginning. Buffer provides the necessary function to handle many types of buffering activity, from simple append operations to the more prosaic producer/consumer balancing between conduits of differing bandwidths. Buffer also supports multiple clients. For example, more than one reader and/or writer may be attached and the buffer will maintain a common state across them. This can be handy where, for example, an application must deal with a mixed or interleaved protocol. These clients must operate contiguously as opposed to concurrently. When operating without a protocol client, buffer is data-type agnostic. It operates as a smart array, flushing and or refilling itself to a conduit as necessary.

Whilst a buffer is often attached to a conduit, it can be used standalone instead as purely a memory-based accumulator. In this case, the buffer cannot be automatically flushed when it overflows, so an exception will be raised where this occurs. A growing buffer can be used to handle this scenario where appropriate. Another buffer variety wraps OS facilities to expose memory-mapped files; buffer memory is mapped directly onto a (usually) large file, which can then be treated as just another buffer or indeed as a void[].

Buffer Exceptions

When not attached to a conduit, a buffer underflow and overflow condition may occur causing an exception to be thrown. Where appropriate, use a growing buffer to manage an overflow condition.

Tokens

Tango has a set of classes to split text into chunks matching a specific pattern. These classes operate upon a conduit, a buffer, or a discrete string, and are templated for char, wchar, and dchar data types. For example, there's a token class for producing lines of text based upon finding embedded end-of-line markers. Token classes are clients of buffer, and can therefore be mixed with reader and/or writer clients also.

Tokens are usually aliased directly from the containing buffer, thus avoiding heap activity where an application doesn't need it.

Token Exceptions

An overflow exception is thrown where the size of a single token is larger than the containing buffer. Either the containing buffer is too small, or the token is overly large (a typical IO buffer is eight or sixteen kilobytes in size). Increase the buffer size to accommodate huge tokens.