FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

HttpClient problem
Goto page 1, 2  Next
 
Post new topic   Reply to topic     Forum Index -> Mango
View previous topic :: View next topic  
Author Message
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Thu May 25, 2006 2:03 pm    Post subject: HttpClient problem Reply with quote

The following is a simulation of an http server:
Code:

module test.server;

import mango.io.ServerSocket,
       mango.io.SocketConduit,
       mango.io.Stdout,
       mango.io.Writer,
       mango.text.LineIterator;

int main (char [][] args)
{
   auto server = new TextServerSocket (new InternetAddress (50800));
   auto client = server.accept ();
   foreach (char [] ln; new LineIterator (client))
      Stdout (ln)(CR);
   auto writer = new Writer (client);
// server response here ------------------------
   writer (
`HTTP/1.0 200 OK
Date: Wed, 24 May 2006 20:21:01 GMT
Server: worklist
Content-type: text/xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<session id="1148502061456" />







`c)   (CR) ();

   client.close ();
   server.close ();

   return 0;
}

And this is a test client for such server:
Code:

module test.buggyclient;

import mango.http.client.HttpClient,
       mango.http.server.HttpHeaders;

int main (char [][] args)
{
   char [] uri = "http://localhost:50800/worklist";
   auto client = new HttpClient (HttpClient.Get, uri);
   auto reqHeaders = client.getRequestHeaders ();
   reqHeaders.add (HttpHeader.Authorization, " basic YWRtaW46YWRtaW4=");
   
   client.open ();
   
   return 0;
}

The request the client is sending is Ok. This is the output from the server:
Code:

GET /worklist HTTP/1.0
Authorization: basic YWRtaW46YWRtaW4=
Host:localhost
Connection:close


However, HttpClient is not parsing the response correctly, as I get "Error: truncated response", but as you can see in the above code, the response is fine, and testing it with another client (one that doesn't use HttpClient but rather opens directly a socket) proves that there's nothing wrong with it. The error is produced in mango/http/client/HttpClient.d:502.
Can somebody explain what's wrong here? I can just use sockets, but I was hoping Mango could help me here.
BTW, this has nothing to do with the socket not being textual.


Last edited by Carlos on Thu May 25, 2006 3:03 pm; edited 1 time in total
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Thu May 25, 2006 2:56 pm    Post subject: Reply with quote

At first glance I suspect the headers are not being returned correctly. With the HTTP protocol, there's supposed to be a CR following the response line, and then following each header. There's also supposed to be an additional blank line after all the headers (to indicate end of headers).

I'm guessing that your response string, although it looks OK, does not have the appropriate EOL terminators embedded within it? I say this because HttpClient has been operational with web-sites and the Mango servlet engine for a couple of years now. That doesn't mean it isn't broken, but it does indicate the potential for something else being amiss Smile
Back to top
View user's profile Send private message
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Thu May 25, 2006 3:06 pm    Post subject: Reply with quote

That's not the actual response, it's just what I got in another client (without HttpClient) against the real server. However, using that response with that other client, it worked, so the response is Ok. And that server is also working as it is (it's actually in Java), so I don't know. I posted the code because I think this should be analyzed fully and not by parts.
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Thu May 25, 2006 3:16 pm    Post subject: Reply with quote

Ach ... you're also inadvertantly sending back some 'garbage' at the start of the server response ~ remember that binary output of arrays will prefix the content with the number of bytes, as a four byte prefix. Use a different type of Writer, or bypass the writer altogether and write to either the underlying buffer or conduit.

There's a misnomer that Writer should be used for everything. Instead, what it does is to dictate a protocol suitable for the applicable Reader to parse correctly. In other words, Writer & Reader are data converters. They handle virtually all of D native types, and are application-friendly via IReadable and IWritable interfaces. I see people making the same mistake with Phobos.stream.write() vs Phobos.stream.writeText(), so something needs to be done about it.

The Reader & Writer classes were originally called BinaryReader & BinaryWriter, in a vague attempt to make the distinction clear Smile
Back to top
View user's profile Send private message
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Thu May 25, 2006 9:14 pm    Post subject: Reply with quote

Mmm... Ok, in this simulation, yes, but in the real server, I can assure you that doesn't happen, so it's back to square one again.
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Fri May 26, 2006 6:51 am    Post subject: Reply with quote

Just to prove the problem is in HttpClient, here's another version of that fake server that doesn't use Mango, thus can't possibly have the text/binary problem:
Code:

import std.cstream,
       std.socket,
       std.socketstream;

int main (char [][] args)
{
   auto server = new TcpSocket ();
   server.bind (new InternetAddress (50800));
   server.listen (10);
   auto client = server.accept ();
   auto str = new SocketStream (client);
   dout.copyFrom (str);
   str.writeLine (
`HTTP/1.0 200 OK
Date: Wed, 24 May 2006 20:21:01 GMT
Server: worklist
Content-type: text/xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<session id="1148502061456" />







`c);

   str.close ();
   server.close ();

   return 0;
}

I keep getting "truncated response" in the client side. In the server side, besides the correct client request, I'm also getting "Broken pipe", which happens because the client exits before the server sent the entire string (if I trim all those blank lines, it doesn't happen).
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri May 26, 2006 11:40 am    Post subject: Reply with quote

The thing with HttpClient is that is simply reads the headers, affording you the opportunity to read the (optional) following content. Invoking HttpClient.open() makes a request ~ it open the socket, sends whatever headers are configured, and waits for the headers to return. The body of the response is entirely for you to handle. HttpClient.open() returns a conduit, which is yours to deal with as you please. IIRC, this is how it's described in the documentation?

Thus, if your program simply calls open(), and then exits (or otherwise causes the open socket to be closed), the other end will almost certainly get a "broken pipe" indication.

If you wish the response body to be handled for you also, there's a wrapper called HttpGet, along with a corresponding example of how to use it.

Onto the "truncated response" error: this is generated when an HTTP response line is seen to be missing, for one reason or another. The check was added when it was noted certain web-sites would fail to return said response-line (including the IBM website). This is in flagrant violation of HTTP protocol, so HttpClient refuses to accept it. Now, for whatever reason, it appears your server is managing to trip that part of HttpClient. I suspect it's the EOL terminators, as noted before.

Perhaps you might try the httpget.d example?
Back to top
View user's profile Send private message
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Fri May 26, 2006 11:57 am    Post subject: Reply with quote

kris wrote:
The thing with HttpClient is that is simply reads the headers, affording you the opportunity to read the (optional) following content. Invoking HttpClient.open() makes a request ~ it open the socket, sends whatever headers are configured, and waits for the headers to return. The body of the response is entirely for you to handle. HttpClient.open() returns a conduit, which is yours to deal with as you please. IIRC, this is how it's described in the documentation?

It's not returning from open: the error is triggered there.

kris wrote:
Thus, if your program simply calls open(), and then exits (or otherwise causes the open socket to be closed), the other end will almost certainly get a "broken pipe" indication.

That was a small try. What I understand from it is that the client (posted here) launches the exception before reading the whole server response, so the dies, and the server can't keep sending data. But it doesn't matter anyway, because that's just an example to prove my point.

kris wrote:
If you wish the response body to be handled for you also, there's a wrapper called HttpGet, along with a corresponding example of how to use it.

Onto the "truncated response" error: this is generated when an HTTP response line is seen to be missing, for one reason or another. The check was added when it was noted certain web-sites would fail to return said response-line (including the IBM website). This is in flagrant violation of HTTP protocol, so HttpClient refuses to accept it. Now, for whatever reason, it appears your server is managing to trip that part of HttpClient. I suspect it's the EOL terminators, as noted before.

I admit I don't know the HTTP protocol in detail, but what header is missing in the code I posted? The response isn't missing, obviously. Is there something wrong with it?
Now, if there're no headers missing, the response line isn't missing either and it's not wrong, then HttpClient must be flawed somewhere (or my logic is).

kris wrote:
Perhaps you might try the httpget.d example?

I'll take a look.

EDIT

I checked the httpget example. The only difference is that it uses HttpGet instead of HttpClient. I tried that, and I keep getting "truncated response".

/EDIT

-=-

Now that you talked about HTTP, is a space supposed to be after the colon in the headers?
Content-type: text/html
Or is it possible to not be?
Content-type:text/html
I ask because Mango doesn't add such space (in HttpClient, at least, don't know about HttpServer), and this particular server doesn't parse them correctly, so I have to prepend a space to the values.
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri May 26, 2006 12:48 pm    Post subject: Reply with quote

I compiled and ran your examples. Two things to note:

1) I couldn't get it to emit the error you had indicated, but it does timeout instead. That gave me a clear picture of what was happening, which is ...

2) Your example server never responds to a request. Take a look at the loop that displays the incoming headers; it doesn't have an exit strategy, and therefore the server never responds. Thus, the server and client are deadlocked.

If you want to display the incoming headers, you must test for the empty line indicating "end of headers" in HTTP protocol. There are alternate ways to control this, but they can become quite complex and have too many side-effects and race-conditions. You can assume if the incoming line is of zero length, all the headers have arrived. You can see this in the mango HttpServer code.

Another thing to be aware of is this: your server should try to return a Content-Length header. This way, the client will know when it has received the full response. The alternatives are (a) have the server gracefully terminate the socket to indicate end-of-transmission, (b) implement the chunking protocol, (c) have the client utilize a timeout, and assume it recieved everything. Option (c) is the default for HttpClient, with a timeout of three seconds.
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri May 26, 2006 1:13 pm    Post subject: Reply with quote

HTTP Headers. This is from the HTTP spec, available online:

Quote:
4.2 Message Headers

HTTP header fields, which include general-header (section 4.5),
request-header (section 5.3), response-header (section 6.2), and
entity-header (section 7.1) fields, follow the same generic format as
that given in Section 3.1 of RFC 822 [9]. Each header field consists
of a name followed by a colon (":") and the field value. Field names
are case-insensitive. The field value MAY be preceded by any amount
of LWS, though a single SP is preferred. Header fields can be
extended over multiple lines by preceding each extra line with at
least one SP or HT. Applications ought to follow "common form", where
one is known or indicated, when generating HTTP constructs, since
there might exist some implementations that fail to accept anything
beyond the common forms.

message-header = field-name ":" [ field-value ]
field-name = token
field-value = *( field-content | LWS )
field-content = <the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string>


The take-home message is that, as stated, whitespace preceeding the field-value is optional. Also noted is the "case-insensitive" aspect of field-name ~ the mango HTTP classes are currently case-sensitive; just something to be aware of.
Back to top
View user's profile Send private message
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Fri May 26, 2006 1:33 pm    Post subject: Reply with quote

kris wrote:
HTTP Headers. This is from the HTTP spec, available online:

Quote:
4.2 Message Headers

HTTP header fields, which include general-header (section 4.5),
request-header (section 5.3), response-header (section 6.2), and
entity-header (section 7.1) fields, follow the same generic format as
that given in Section 3.1 of RFC 822 [9]. Each header field consists
of a name followed by a colon (":") and the field value. Field names
are case-insensitive. The field value MAY be preceded by any amount
of LWS, though a single SP is preferred. Header fields can be
extended over multiple lines by preceding each extra line with at
least one SP or HT. Applications ought to follow "common form", where
one is known or indicated, when generating HTTP constructs, since
there might exist some implementations that fail to accept anything
beyond the common forms.

message-header = field-name ":" [ field-value ]
field-name = token
field-value = *( field-content | LWS )
field-content = <the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string>


The take-home message is that, as stated, whitespace preceeding the field-value is optional. Also noted is the "case-insensitive" aspect of field-name ~ the mango HTTP classes are currently case-sensitive; just something to be aware of.

Ok, thanks.
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Fri May 26, 2006 1:40 pm    Post subject: Reply with quote

kris wrote:
I compiled and ran your examples. Two things to note:

1) I couldn't get it to emit the error you had indicated, but it does timeout instead. That gave me a clear picture of what was happening, which is ...

2) Your example server never responds to a request. Take a look at the loop that displays the incoming headers; it doesn't have an exit strategy, and therefore the server never responds. Thus, the server and client are deadlocked.

I think we're deadlocked too.

kris wrote:
If you want to display the incoming headers, you must test for the empty line indicating "end of headers" in HTTP protocol. There are alternate ways to control this, but they can become quite complex and have too many side-effects and race-conditions. You can assume if the incoming line is of zero length, all the headers have arrived. You can see this in the mango HttpServer code.

Another thing to be aware of is this: your server should try to return a Content-Length header. This way, the client will know when it has received the full response. The alternatives are (a) have the server gracefully terminate the socket to indicate end-of-transmission, (b) implement the chunking protocol, (c) have the client utilize a timeout, and assume it recieved everything. Option (c) is the default for HttpClient, with a timeout of three seconds.

I have no control over the server, so I can't change it, only use it.

Oh, well, I'll keep my current work of not using HttpClient.
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri May 26, 2006 1:57 pm    Post subject: Reply with quote

Carlos wrote:
kris wrote:
I compiled and ran your examples. Two things to note:

1) I couldn't get it to emit the error you had indicated, but it does timeout instead. That gave me a clear picture of what was happening, which is ...

2) Your example server never responds to a request. Take a look at the loop that displays the incoming headers; it doesn't have an exit strategy, and therefore the server never responds. Thus, the server and client are deadlocked.

I think we're deadlocked too.

I'm trying to help, Carlos. And, as noted early on, the possibility of HttpClient being broken is quite real. However, I had made an assumption (apparently incorrectly) that you were using the example-server to show where things were broken. Yet said example could not possibly work, right?

If you'd like to help in return, you might post the exact response from this third-party server? Write a client, using sockets only, to send a request and capture the response. I'd very much like to see what that response is. Both in text and hex, if you can provide that?

Alternatively: start that server up on port 80, make it available through your firewall, and give me your IP address. I'll take a closer look Smile


Last edited by kris on Fri May 26, 2006 2:59 pm; edited 1 time in total
Back to top
View user's profile Send private message
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Fri May 26, 2006 2:58 pm    Post subject: Reply with quote

Same working client as before, another request (to another port, actually). Here's the output:
Code:

HTTP/1.1 200 OK
Date: Fri, 26 May 2006 20:33:54 GMT
Server: Jetty/5.1.10 (Mac OS X/10.4.6 ppc java/1.5.0_06
Content-Type: application/xml
Content-Length: 986
Last-Modified: Thu, 11 May 2006 00:56:24 GMT
Accept-Ranges: bytes
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
(XML follows)

Same buggy client as before: "Truncated response".

EDIT: BTW, I tried this with the browser (Camino) and it worked.

-

I tried to debug HtppClient.d and I found that the LineIterator in line 490 doesn't return anything, so it's like it's not receiving what the server is sending.


Last edited by Carlos on Fri May 26, 2006 3:08 pm; edited 1 time in total
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Fri May 26, 2006 3:06 pm    Post subject: Reply with quote

kris wrote:
I'm trying to help, Carlos.

I know, and I can't thank you enough for that, it's just that the show must continue Laughing
kris wrote:
And, as noted early on, the possibility of HttpClient being broken is quite real. However, I had made an assumption (apparently incorrectly) that you were using the example-server to show where things were broken. Yet said example could not possibly work, right?

But the client shows the same behavior as with the real server.
kris wrote:
If you'd like to help in return, you might post the exact response from this third-party server? Write a client, using sockets only, to send a request and capture the response. I'd very much like to see what that response is. Both in text and hex, if you can provide that?

I have posted that, not in hex but in text. Well, not explicitily, but what I'm sending in the test server is what the real server sends.
kris wrote:
Alternatively: start that server up on port 80, make it available through your firewall, and give me your IP address. I'll take a closer look Smile

If I only could: I'm on dial-up, so that's impossible.
What you could do if you want to take that closer look is try to use that sample client I posted before to start a session with OpenWFE. All you need is a JDK and then start the thing following the readme. The (default) port to connect is 5080. If you want.
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Mango All times are GMT - 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group