Download Reference Manual
The Developer's Library for D
About Wiki Forums Source Search Contact

HttpClient weird behaviour

Moderators: kris

Posted: 08/12/09 19:32:36

I'm using the HttpClient? to fetch a lot of websites from different domains (approx 130'000 in the test case). The problem happens when using HttpClient? in combination with threads (approx. 50). Between 10'000 and 20'000 requests I get a "unable to resolve <domain name>" for the rest of the domains. I tried to figure out where the error comes from. So far I could track it down to the openFinish method within HttpClient? class. I wrote a little test program that creates the threads and opens the URL, without reading the content:

...
auto client = new HttpClient(HttpClient.Get, addr);
auto response = client.open();              
response.close();
client.close();
...

This code worked fine so far when all the functionality from openFinish was removed. Does anybody have an idea, where I should look? I'm not sure, but it seems to be a problem with the line iterator.

ciao Lars

Author Message

Posted: 08/13/09 18:10:04 -- Modified: 08/13/09 18:11:20 by
lars_kirchhoff

The problem seems to be, that too many connections stay open after the program is running. I'm guessing that some sockets are not properly closed in case of an exception. I will look further I try to pin down the problem..

/lars

Posted: 08/14/09 11:13:04

I just issued a ticket..

/lars