Download Reference Manual
The Developer's Library for D
About Wiki Forums Source Search Contact

Memory Leak in ZlibStream?

Moderators: kris

Posted: 07/16/09 18:27:47 Modified: 07/21/09 05:56:18

Hey,

I've tried to use the tango.io.compress.ZlibStream? class together with the tango.io.Path in
order to decompress all files in a directory. In that directory there are around 8'000 files,
which i need to decompress. Unfortunately the small program eats up to 800MB of memory and
therefore slows down if it reaches 4'000 files. As it occurred randomly I tried to figure out
what happens. I used valgrind to look for memory leaks. These are some of the output lines from
valgrind, which reoccur alot: Valgrind output

Below the small test program, which produced the above output. Is there anything I
need to set before using ZlibStream?. From the error message I would suspect that
I need to initialize some variables, but I cant find to see anything. Any help is
very much appreciated.

module zlibtest;

private import tango.io.device.File;
private import Path = tango.io.Path;
private import tango.io.Stdout;
private import tango.io.compress.ZlibStream;
private import tango.io.stream.Buffered;

void main (char[][] args)
{       
    auto zt= new ZlibTest();    
    zt.run();    
}


class ZlibTest
{
    public this () {}
    
    void run ()
    {   
        Path.FS.list("bin/queue/", &uncompress_files);
    }
    
    int uncompress_files ( inout Path.FS.FileInfo file )
    {
        // Stdout.formatln("{} - {}", file.path, file.name);        
        auto fp = new File (file.path ~ file.name);
        auto content = this.decode(fp.input);        
        if (content.length)
            // Stdout.formatln("{}", content[0..30]);   
            
        fp.close();        
        return 0;
    }
    
    char[] decode ( InputStream stream_in )
    { 
        char[]      content, final_content;
        int         size;
        ZlibInput   zlib_input;
        
        auto decomp = new char[1024];
        
        try 
        {
            zlib_input = new ZlibInput(stream_in);            
            while ((size = zlib_input.read(decomp)) > 0)
            {
                content ~= decomp[0..size];
            }
            
            return content;
        }
        catch (Exception e)
        {
            Stdout.formatln("ZlibError: " ~ e.msg);
        }
        return null;
    }
}

/lars

Author Message

Posted: 07/16/09 21:28:02

First; valgrind isn't very useful in finding out about memory leaks in D, since the GC technically is entitled to keep it along for as long as it want. Also, the stuff pasted here from Valgrind isn't memory leaks - in fact they are a necessity of the garbage collectors operation.

Now, looking at your example, it does allocate a huge amount of memory, so it is no wonder if you get problems.

I suggest using

scope f = new File

instead of auto, but that is minor compared with the

content ~= decomp[0..size];

Here you continously allocate (and potentially reallocate) memory (for the content array) for all the content in all the files you decompress. I don't know what you are using the decompressed data for, but you should definately rewrite such that you have one large buffer to decode all files into without the need to reallocate every time. Try to reuse memory as often as possible.

Posted: 07/17/09 07:51:47 -- Modified: 07/17/09 14:24:32 by
lars_kirchhoff -- Modified 4 Times

Thanks larsivi,

I think this is old habit from other programming languages, because I thought
that with every call of the decode function content would be new initialized and
therefore be empty. I'm still have some problems with the scope of variables in D.

But what still keeps me curious is that the memory grows large only sometimes, while
on other times it just runs fine.

/lars

PS: I'm not sure about valgrind, but this post on digitalmars let me try valgrind: http://www.digitalmars.com/d/archives/digitalmars/D/announce/3508.html

PPS: What tool can I use to find memory leaks?

Posted: 07/17/09 13:41:30

Only static arrays are put on the stack (even for arrays that are local variables), whereas you create a dynamic array (content). In addition you return a reference to it, so your program would crash if the array had been allocated on the stack. Then, every time you append information to content, the content array (a new one for each decode run), the GC must extend the memory (in some cases reallocate in a different place), and it is this process that is very slow, in addition to more and more memory being allocated. The GC may be abel to reclaim some of it, but when is a different question.

Posted: 07/17/09 17:06:28 -- Modified: 07/21/09 05:56:42 by
lars_kirchhoff -- Modified 2 Times

Thank you very much again, for the small D lesson :).. The rest is now not
related to ZlibStream? anymore.

The output are html and xml files that need to be parsed. I'm now very much in
trouble, because a lot of code that I've written is passing char[]'s to methods
in classes and return modified versions of it. As I understand your explanations
these are all potential memory leaks http://www.digitalmars.com/d/1.0/memory.html.

For example, I have a method that replaces some char[]'s in the text:

public char[] text_parse ( char[] s )
{   
    foreach (entity; entities)   
    {
        s = Util.substitute(s.dup, entity.search, entity.replacement);
    }   
   
    return s;
}

This will leaks also. How would I need to rewrite that?

I tried the following, but unfortunatley without luck.

public char[] text_parse ( char[] s )
{   
    char[] r = new char[s.length];
    r = s;
    s = r;
    foreach (entity; entities)   
    {
        s = Util.substitute(s.dup, entity.search, entity.replacement);
    }   
   
    return s;
}

I'm glueless because this makes D very complicated for any text manipulation operations.

/lars

Posted: 07/18/09 13:05:59

The idea of a garbage collector is that it frees any memory not freed elsewhere - so I don't think you can have a memory leak (without avoiding use of the garbage collector, which you're not). What you can have is lots of reallocation: you allocate one block, then another and another, and the garbage collector should free any you no longer need, but unfortunately it won't usually do that immediately and each new allocation still takes time.

lars_kirchhoff wrote:
public char[] text_parse ( char[] s )
{   
    char[] r = new char[s.length];
    r = s;
    s = r;
    foreach (entity; entities)   
    {
        s = Util.substitute(s.dup, entity.search, entity.replacement);
    }   
   
    return s;
}

This code doesn't make much sense. r = new ... allocates a block of memory, but then r = s; sets the pointer r to point to s's memory (thus leaving the newly allocated block to be freed). You could use r[] = s; to copy the contents of r to s, but it won't help speed things up.

I'm not really sure how Util.substitute works or could be improved, but if you want to speed up the .dup operation you could do something like this (but note this is not thread safe because it uses static memory):

public char[] text_parse ( char[] s )
{   
    static char[] r;
    foreach (entity; entities)   
    {
        if (r.length < s.length)
            r.length = s.length;  // reallocate r if it's too small, otherwise use existing memory. You might be able to reduce the number of times allocations are needed by choosing the length more smartly.
        r[] = s;  // copy
        s = Util.substitute(r, entity.search, entity.replacement);
    }
    
    return s;
}

This will save some of the allocations used by .dup, but I don't know how Util.substitue works. You might be able to do better, for instance if it doesn't need a copy of s at all.

What larsivi was pointing out is that instead of using content ~= ... with each decoded block, you could either write that to file immediately or try not to reallocate each time you're copying (use content[n..n+x.length] = x).

Posted: 07/18/09 15:18:07

So, that algorithm is quadratic space. Want to do it in linear space? Even better, want to remove almost all memory allocations? Okay.

module test;

import tango.io.Stdout;
import tango.text.Util;

// I'm assuming you already have something like this
struct Entity
{
	char[] search;
	char[] replacement;
}

const Entity[] entities =
[
	{ "&amp;", "&" },
	{ "&lt;" , "<" },
	{ "&gt;" , ">" },
];

// Set up an AA that will let us look up the entities quickly
char[][char[]] entityMap;

static this()
{
	foreach(ent; entities)
		entityMap[ent.search] = ent.replacement;

	// It's good to do this after creating lookup tables that will never change
	entityMap.rehash;
}

// Optional buffer to hold the data.  It's 'ref' so we can resize it and it'll be reused by the caller.
// Returns a slice of 'buffer', and allocates its own buffer if buffer is null.
char[] replaceEntities(char[] input, ref char[] buffer = null)
{
	// So, since every replacement is shorter than the entity name, we can assume the output buffer
	// only has to be at most as long as the input string.
	if(buffer.length < input.length)
		buffer.length = input.length;

	// This is our index into the output buffer.
	size_t outIdx = 0;

	for(size_t inIdx = 0; inIdx < input.length; )
	{
		// First, look for an opening &
		auto entBegin = input.locate('&', inIdx);

		// Output any text that came before the &.
		if(entBegin > inIdx)
		{
			auto len = entBegin - inIdx;
			buffer[outIdx .. outIdx + len] = input[inIdx .. inIdx + len];
			outIdx += len;
			inIdx = entBegin;
		}

		// Did we run out of string?
		if(entBegin == input.length)
			break;

		// Find the terminating ;
		auto entEnd = input.locate(';', entBegin + 1);

		// See if we've got a malformed entity (opening & but no closing ;)
		if(entEnd == input.length)
			throw new Exception("OH NOOOO, unclosed entity");

		// Grab the entity text and look it up
		// 'in' will give null if it doesn't exist, or an char[]* if it does
		auto pRepl = input[entBegin .. entEnd + 1] in entityMap;

		// Unrecognized entity?
		if(pRepl is null)
			throw new Exception("CRAP, don't know the entity '" ~ input[entBegin .. entEnd + 1] ~ "'");

		// Otherwise, replace!
		auto repl = *pRepl;
		buffer[outIdx .. outIdx + repl.length] = repl[];
		outIdx += repl.length;

		// Update the index
		inIdx = entEnd + 1;
	}

	// Return the slice of the buffer that's valid
	return buffer[0 .. outIdx];
}

void main()
{
	// Some test strings
	char[][] strings =
	[
		"This is a test.  No entities here",
		"I &lt;3 mudkips",
		"if(x &lt; 0 &amp;&amp; y &gt; 0)"
	];

	// Evil strings
	char[][] evilStrings =
	[
		"I have an unclosed entity&",
		"How about &odash;?"
	];

	// We don't have to allocate anything; replaceEntities will do it for us, and as long
	// as we reuse it, no garbage will be created!
	char[] buf;

	// Test it out
	foreach(str; strings)
		Stdout.formatln("'{}' => '{}'", str, replaceEntities(str, buf));

	// Test the evil strings
	foreach(str; evilStrings)
	{
		try
			replaceEntities(str, buf);
		catch(Exception e)
			Stdout.formatln("When converting '{}', {}", str, e);
	}
}

The output:

'This is a test.  No entities here' => 'This is a test.  No entities here'
'I &lt;3 mudkips' => 'I <3 mudkips'
'if(x &lt; 0 &amp;&amp; y &gt; 0)' => 'if(x < 0 && y > 0)'
When converting 'I have an unclosed entity&', OH NOOOO, unclosed entity
When converting 'How about &odash;?', CRAP, don't know the entity '&odash;'

Caveat: the string returned by replaceEntities is a slice of buf. If you keep that string around and then reuse buf, the string will be overwritten. If you need to keep the string returned by replaceEntities around, be sure to do a .dup on the return value from replaceEntities. In this example, we only use the string once until we reuse buf, so we don't have to do that.

I don't know why I'd make a new scripting language. I mean, I might as well just draw some lines in the sand with a stick.

Posted: 07/18/09 20:05:40

or consider using tango.text.xml.DocEntity

Posted: 07/20/09 08:43:31 -- Modified: 07/20/09 10:34:40 by
lars_kirchhoff -- Modified 4 Times

Thanks a lot for all the useful comments and suggestions. This helps a lot to understand
the whole issue with memory allocations and references used when working with char arrays.

Cyborg16 wrote:

What larsivi was pointing out is that instead of using content ~= ... with each decoded block,
you could either write that to file immediately or try not to reallocate each time you're
copying (use content[n..n+x.length] = x).

This is exactly what I did after larsivi's comment:

while ( (size = decomp.read(output)) > 0 )
{   
    end = start + size;
    content[start..end] = output[0..size];                
    start += size;               
}                        
return content[0..end]; 

After reading Jarrett's post I was thinking if it would be useful to dup the return,
because if I understand his code and comment.

JarrettBillingsley wrote:

Caveat: the string returned by replaceEntities is a slice of buf. If you keep that
string around and then reuse buf, the string will be overwritten. If you need to keep
the string returned by replaceEntities around, be sure to do a .dup on the return value
from replaceEntities. In this example, we only use the string once until we reuse buf,
so we don't have to do that.

Would it be correct to directly dup the return value?

return content[0..end].dup;

Or should the dup be made after the result has been retrieved from the method?

    char[] r = doSomething(s);
    s = r.dup;

But now I'm seeing this problem in lots of places in my code. After some more code inspection
I'm not sure anymore if the posted method with Util.substitute is the real problem. It might be
that the problem is elsewhere in the code.

/lars

Posted: 07/20/09 13:16:12 -- Modified: 07/20/09 13:17:17 by
JarrettBillingsley

lars_kirchhoff wrote:
JarrettBillingsley wrote:

Caveat: the string returned by replaceEntities is a slice of buf. If you keep that
string around and then reuse buf, the string will be overwritten. If you need to keep
the string returned by replaceEntities around, be sure to do a .dup on the return value
from replaceEntities. In this example, we only use the string once until we reuse buf,
so we don't have to do that.

Would it be correct to directly dup the return value?

return content[0..end].dup;

Or should the dup be made after the result has been retrieved from the method?

    char[] r = doSomething(s);
    s = r.dup;

Personally I'd never dup inside replaceEntities and leave that decision up to the caller. But that's just my (and Tango's) preference ;)

I don't know why I'd make a new scripting language. I mean, I might as well just draw some lines in the sand with a stick.

Posted: 07/20/09 17:14:10 -- Modified: 07/21/09 05:56:01 by
lars_kirchhoff -- Modified 2 Times

I had a look in tango.text.Util substitute function. There seems to be a problem if
I understood the comment from larsivi correctly:

foreach (s; patterns (source, match, replacement))
    output ~= s;

I tried to write my own substitute (with some inspiration from Jarrett ;)) with the goal
to replace html encodings. The problem with the code with Jarretts code is that I have inputs,
in which the encodings are correct and some in which they are not. So it needs to work with input
in which & is just a regular character.

I'm not sure if this working correctly all time, but it did in my case work quite well (I'm pretty sure
it can be optimized). Particularly I'm not sure if the grow option works..
Does anybody see any problems with the code?

module test;

private import Util = tango.text.Util;
private import tango.io.Stdout;

struct entity 
{
    char[] replacement;
    char[] search;
}

static entity[] html_entities =
[
    { "\"" , "&quot;"  },
    { "'"  , "&#039;"  },
    { "'"  , "&#39;"   },
    { "'"  , "&#8217;" },
    { "<"  , "&lt;"    },
    { ">"  , "&gt;"    },
    { "&"  , "&amp;"   },
    { " "  , "&;nbsp;" },
    { " "  , "&nbsp;"  }
];


public static char[] html_entity_decode ( char[] string, ref char[] buffer = null )
{
    foreach (html_entity; html_entities)
        string = replaceEntity(string, html_entity.search, html_entity.replacement, buffer);
    
    return string;
}


public static char[] replaceEntity ( char[] input, char[] search, char[] replacement, ref char[] buffer = null )
{   
    uint    out_idx         = 0, 
            in_idx          = 0,
            pattern_idx     = 0,
            add_len         = 0,            
            input_len       = input.length,    
            search_len      = search.length,
            replacement_len = replacement.length; 
    
    bool    grow = false;
    
    if (buffer.length < input_len)
        buffer.length = input_len;
   
    // check if the replacement string is larger than the search string
    // get the length difference in order to grow the buffer on each replacement  
    if (search_len < replacement_len)
    {
        add_len = replacement_len - search_len;
        grow = true;        
    }
    
    if ((pattern_idx = Util.locatePattern(input, search, pattern_idx)) < input_len)
    {                  
        do
        {
            // check if there is something to copy between search strings and 
            // and if yes copy it
            if (in_idx != pattern_idx)
                buffer[out_idx..out_idx+input[in_idx..pattern_idx].length] = input[in_idx..pattern_idx];            
                        
            // set new positions
            out_idx = out_idx + (pattern_idx - in_idx);
            in_idx = in_idx + (pattern_idx - in_idx);
            
            // copy the replacement string into buffer            
            buffer[out_idx..out_idx+replacement_len] = replacement[0..$];
            
            // set new positions 
            out_idx += replacement_len;
            in_idx += search_len;     
            
            // resize buffer if replacement is larger then search string
            if (grow)
                buffer.length = buffer.length + add_len;            
        }
        while ((pattern_idx = Util.locatePattern(input, search, pattern_idx + search_len)) < input_len);
        
        // copy the rest
        buffer[out_idx..out_idx+input[in_idx..$].length] = input[in_idx..$];
        out_idx += input[in_idx..$].length;
        
        return buffer[0..out_idx].dup;
    }
    
    return input;
}


void main (char[][] args)
{
    char[] buffer;
    
    // Some test strings
    char[][] strings =
    [
        "This is a test.  No entities here",
        "this is a dirty&;nbsp;Test",        
        "I &lt;3 mudkips",
        "if(x &lt; 0 &amp;&amp; y &gt; 0)"
    ];

    // Evil strings
    char[][] evilStrings =
    [
        "I have an unclosed entity&",
        "How about &odash;?"
    ];
    
    // Test it out
    foreach(str; strings)
        Stdout.formatln("'{}' => '{}'", str, html_entity_decode(str, buffer));

    // Test the evil strings
    foreach(str; evilStrings)
        Stdout.formatln("'{}' => '{}'", str, html_entity_decode(str, buffer));
}

Posted: 07/21/09 06:25:11

lars_kirchhoff wrote:
    foreach (html_entity; html_entities)
        string = replaceEntity(string, html_entity.search, html_entity.replacement, buffer);

Now you've gone and made it quadratic time again ;) You're doing:

foreach entity
    foreach character in the string
        is this the entity we're looking for?  if so, replace it

But there's no reason to have the outer loop. That's the whole reason I had the lookup table (associative array) in my code, so it'd just be:

foreach character in the string
    is this an entity?  if so, look it up (trivial!) and replace it

Also, you've used the buffer nicely, but then completely destroyed any savings by using .dup when returning the string!

Have a look at the full table of HTML entities. There are over 250 of them. Do you really want to be doing

for(int i = 0; i < 250; i++)
    replace one type of entity, potentially allocating a new string

on every single input string? You're generating as many as 249 intermediate strings that are just garbage. This is what kills GC performance ;)

I don't know why I'd make a new scripting language. I mean, I might as well just draw some lines in the sand with a stick.

Posted: 07/21/09 08:35:03 -- Modified: 07/21/09 08:36:17 by
lars_kirchhoff

JarrettBillingsley wrote:

Now you've gone and made it quadratic time again ;) You're doing:

Time is not that much important here, but rather that the replacement
of any entity is possible, regardless if it is in the syntax of html
encodings (e.g. bad encodings like nbsp; or &amp need to be decoded as well).
This is why I need to look for the whole entity string, although I
would prefer your method.

JarrettBillingsley wrote:

You're generating as many as 249 intermediate strings that are just garbage. This is what kills GC performance ;)

Would this really create 249 intermediate strings?

if ((pattern_idx = Util.locatePattern(input, search, pattern_idx)) < input_len)

I thought I would prevent this by checking if I find the pattern within
the input and only if I found it begin to fill the buffer. Otherwise just return the input.

/lars

Posted: 07/21/09 15:43:10

lars_kirchhoff wrote:
JarrettBillingsley wrote:

You're generating as many as 249 intermediate strings that are just garbage. This is what kills GC performance ;)

Would this really create 249 intermediate strings?

"As many as 249 intermediate strings" means "in the worst case." Yes, it probably usually wouldn't be anywhere near that, but if a string contained every possible entity - or something close to it - it would. A single-pass replacement algorithm would never create more than one intermediate string no matter how many entities the input contained, and if you didn't .dup it, you'd create virtually no garbage.

lars_kirchhoff wrote:
if ((pattern_idx = Util.locatePattern(input, search, pattern_idx)) < input_len)

I thought I would prevent this by checking if I find the pattern within
the input and only if I found it begin to fill the buffer. Otherwise just return the input.

Yes, that's right. But if you're expecting more of your strings to have entities than to not have them, you're still creating temporaries on most inputs, so it's not that much of a savings. You don't need to .dup before returning, anyway - leave that decision up to the caller.

I don't know why I'd make a new scripting language. I mean, I might as well just draw some lines in the sand with a stick.

Posted: 08/07/09 19:18:43

After the excursion to the substitute problem I would like to return
to the original
ZlibStream? problem. Although the suggestion from larsivi
improved the situation I now run in the same problem again, because I built
up a rather big associative array before running the uncompress. Here is a
small test program, that produces the memory problem:

module      zlibtest;

private     import      tango.io.Stdout;
private     import      tango.io.device.File, 
                        tango.io.stream.Lines, 
                        Path = tango.io.Path;
private     import      tango.io.compress.ZlibStream;


void main ( char[][] args )
{   
    ZlibTest zl = new ZlibTest();    
}

class ZlibTest
{   
    private     char[]          names_file  = "list_file.txt";   
    private     char[]          file        = "compressed_file.gz";
    private     char[]          content;
    private     char[][char[]]  global_list;
    public      ubyte[]         chunk;
        
    public this ()
    { 
        this.global_list = this.buildList(this.names_file);        
        
        for (uint k=0; k<10_000; k++)
        {   
            try 
            {
                Stdout.formatln("{}", this.file).flush();
                this.content = this.readCompressedFile(this.file);
            }
            catch (Exception e)
                Stdout.formatln("Read Error: " ~ e.msg);
        }
    }    
    
    char[] readCompressedFile ( char[] file_name )
    {  
        int start = 0, end = 0, size = 0;
        char[1024] read_buffer;
        char[2*1024*1024] c;
        
        try 
        {               
            scope stream_in = new File (this.file, File.ReadExisting);
            scope (exit)
                stream_in.close();
            
            scope decomp = new ZlibInput(stream_in);
            scope (exit)
                delete decomp;
            
            while ((size = decomp.read(read_buffer)) != stream_in.Eof)
            {  
                end = start + size;
                c[start..end] = read_buffer[0..size];                
                start += size;    
            }
            return c[0..end];
        }
        catch (Exception e)
            Stdout.formatln("Uncompress Error: " ~ e.msg);
        
        return null;
    }    
    
    char[][char[]] buildList ( char[] path )
    {
        char[][char[]] list;
        scope map = new File(path, File.ReadExisting);
        scope (exit)    
            map.close();    
        
        Lines!(char) lines = new Lines!(char)(map);
                
        foreach (line; lines)
            if (line.length)
                list[line.dup] = line.dup;
        
        return list;
    }
}

The memory starts growing if I declare the associative array as a class
property. If I declare the variable within the constructor everything
is fine. Without initializing the associative array everything works fine
also.

I had a look in the ZlibStream? class again and found that this line

in_chunk = new ubyte[CHUNKSIZE];

caused the growing of memory just by calling the constructor, without even
reading the compress file. I modified the class to add an input buffer (chunk)
to the constructor and modified the line above as follows:

if (input_chunk.length == 0)
{   
    in_chunk = new ubyte[CHUNKSIZE];
}
else
{         
    in_chunk = input_chunk;
}

I than created a buffer in the calling class and passed it in the
ZlibStream? constructor. This helped keeping memory in control (at least in my case).

module      zlibtest;

private     import      tango.io.Stdout;
private     import      tango.io.device.File, 
                        tango.io.stream.Lines, 
                        Path = tango.io.Path;
private     import      tango.io.compress.ZlibStream;


void main ( char[][] args )
{   
    ZlibTest zl = new ZlibTest();    
}

class ZlibTest
{   
    private     char[]          names_file  = "list_file.txt";   
    private     char[]          file        = "compressed_file.gz";
    private     char[]          content;
    private     char[][char[]]  global_list;
    public      ubyte[]         chunk;
        
    public this ()
    { 
        this.global_list = this.buildList(this.names_file);
// -->  this.chunk = new ubyte[256*1024];
        
        for (uint k=0; k<10_000; k++)
        {   
            try 
            {
                Stdout.formatln("{}", this.file).flush();
                this.content = this.readCompressedFile(this.file);
            }
            catch (Exception e)
                Stdout.formatln("Read Error: " ~ e.msg);
        }
    }    
    
    char[] readCompressedFile ( char[] file_name )
    {  
        int start = 0, end = 0, size = 0;
        char[1024] read_buffer;
        char[2*1024*1024] c;
        
        try 
        {               
            scope stream_in = new File (this.file, File.ReadExisting);
            scope (exit)
                stream_in.close();
            
// -->      scope decomp = new ZlibInput(stream_in, this.chunk);
            scope (exit)
                delete decomp;
            
            while ((size = decomp.read(read_buffer)) != stream_in.Eof)
            {  
                end = start + size;
                c[start..end] = read_buffer[0..size];                
                start += size;    
            }
            return c[0..end];
        }
        catch (Exception e)
            Stdout.formatln("Uncompress Error: " ~ e.msg);
        
        return null;
    }    
    
    char[][char[]] buildList ( char[] path )
    {
        char[][char[]] list;
        scope map = new File(path, File.ReadExisting);
        scope (exit)    
            map.close();    
        
        Lines!(char) lines = new Lines!(char)(map);
                
        foreach (line; lines)
            if (line.length)
                list[line.dup] = line.dup;
        
        return list;
    }
}

I would like to understand why a global class property can cause such a memory problem.
Could somebody explain this? I need the associative array to be a class property, because
it will be used in different methods. Is there something else wrong in the code that is wrong
and may cause this error?

thanks for any help.

/lars

Posted: 08/08/09 09:39:44

I had another look at ZlibStream? and was wondering why in_chunk is declared
as dynamic array and then set to a static size later in the constructor.

line 96: ubyte[] in_chunk;
line 179: in_chunk = new ubyte[CHUNKSIZE];

Is it because of the variable scope? Wouldn't it make sense to initialize
it as static right away?

line 96: ubyte[CHUNKSIZE] in_chunk;

/lars

Posted: 08/08/09 15:06:02

lars_kirchhoff wrote:

I had another look at ZlibStream? and was wondering why in_chunk is declared
as dynamic array and then set to a static size later in the constructor.

line 96: ubyte[] in_chunk;
line 179: in_chunk = new ubyte[CHUNKSIZE];

Is it because of the variable scope? Wouldn't it make sense to initialize
it as static right away?

line 96: ubyte[CHUNKSIZE] in_chunk;

/lars

"new ubyte[CHUNKSIZE]" is not a static array, it's a dynamic array. Confusing, I know.

I don't know why I'd make a new scripting language. I mean, I might as well just draw some lines in the sand with a stick.

Posted: 08/08/09 15:23:12

Exactly. that's is why I asked if it would make sense to declare the in_chunk as static array.

/lars

Posted: 08/08/09 23:33:40

*shrug*

Slap Daniel ;)

I don't know why I'd make a new scripting language. I mean, I might as well just draw some lines in the sand with a stick.