Forum Navigation

Digester not working?

Moderators: kris

Abscissa

Joined: 02/23/05

Posts: 39

Posted: 04/12/09 05:27:13 Modified: 04/12/09 05:28:31

I'm not sure if I'm doing something wrong here, or what, but io.stream.Digester isn't working for me (DMD 1.041 / Tango 0.99.8):

// File: main.d
module digester;

import tango.io.Stdout;
import tango.io.device.File;
import tango.io.digest.Crc32;
import tango.io.digest.Digest;
import tango.io.stream.Buffered;
import tango.io.stream.Data;
import tango.io.stream.Digester;
import tango.math.Math;

char[] digestOfFile_Digester(Digest digest, char[] path)
{
	auto file = new File(path);
	scope(exit) file.close();
	
	auto fileBuffer   = new BufferedInput(file.input);
	auto fileDigester = new DigestInput(fileBuffer, digest);
	auto fileInput    = new DataInput(fileDigester);
	
	while(true)
	{
		auto actualPosition = file.position() - fileBuffer.readable();
		auto amountToRead = min(file.bufferSize(), file.length() - actualPosition);
		
		if(amountToRead > 0)
			fileInput.load(amountToRead);
		
		if(amountToRead < file.length())
			break;
	}

	return fileDigester.digest.hexDigest();
}

char[] digestOfFile_Manual(Digest digest, char[] path)
{
	auto file = new File(path);
	scope(exit) file.close();
	
	auto fileBuffer   = new BufferedInput(file.input);
	auto fileInput    = new DataInput(fileBuffer);
	
	while(true)
	{
		auto actualPosition = file.position() - fileBuffer.readable();
		uint amountToRead = min(file.bufferSize(), cast(uint)(file.length()-actualPosition));
		
		if(amountToRead > 0)
		{
			auto data = fileInput.load(amountToRead);
			digest.update(data);
		}
		
		if(amountToRead < file.length())
			break;
	}
	
	return digest.hexDigest();
}

void main(char[][] args)
{
	auto checksum1 = digestOfFile_Digester (new Crc32(), "main.d");
	auto checksum2 = digestOfFile_Manual   (new Crc32(), "main.d");
	Stdout.formatln("Digester: {}", checksum1);
	Stdout.formatln("Manual:   {}", checksum2);
}

Output for me:

Digester: 00000000
Manual:   ec5628cf

Obviously that "Manual" checksum is going to change with tiny changes in the source, but on the "Digester" one I'm getting "00000000" no matter what.

Author Message
kris

Joined: 03/27/04

Posts: 581

Posted: 04/12/09 05:38:47

Thanks ... I'll look into it right away
kris

Joined: 03/27/04

Posts: 581
Posted: 04/12/09 06:12:08 -- Modified: 04/12/09 08:24:53 by kris -- Modified 4 Times

Fixed a bug in DigestInput and DigestOutput.

I'm sure you had good reason for the file-offset calculations, but there should be no need for such effort ... that sort of thing usually indicates a problem with the API, and we'd prefer to resolve it. FWIW, a more concise version of the original looks like this:
char[] digestOfFile_Digester (Digest digest, char[] path)
{
        auto input = new DigestInput(new File(path), digest);
        input.slurp.close;
	return digest.hexDigest;
}
Thanks for bringing this up :)

[edit: removed redundant DataInput wrapper, and added slurp)
Abscissa

Joined: 02/23/05

Posts: 39

Posted: 04/12/09 06:46:11 -- Modified: 04/12/09 06:48:00 by Abscissa
kris wrote:
I'm sure you had good reason for the file-offset calculations, but there should be no need for such effort ... that sort of thing usually indicates a problem with the API, and we'd prefer to resolve it. FWIW, a more concise version of the original looks like this (with the redundant DataInput intact):

Thanks :). I suppose I didn't need that fancy stuff in this example, but I came across the problem while working on something that I wanted to ensure scaled well to large files. So I didn't want a gigantic temp var to potentially be created. (Also the docs on the streams/conduits/io/etc seem kind of confusing, I wasn't quite sure I really knew what I was doing, so there may still have been a better way anyway...).

kris

Joined: 03/27/04

Posts: 581

Posted: 04/12/09 08:19:42 -- Modified: 04/12/09 08:20:15 by kris

added a cheap slurp() method to DigestInput, for when there's a need to merely consume remaining stream content.