View previous topic :: View next topic |
Author |
Message |
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Fri May 26, 2006 3:30 pm Post subject: |
|
|
Carlos wrote: | kris wrote: | And, as noted early on, the possibility of HttpClient being broken is quite real. However, I had made an assumption (apparently incorrectly) that you were using the example-server to show where things were broken. Yet said example could not possibly work, right? |
But the client shows the same behavior as with the real server. |
Are you sure? When I tried your example, it gave me a "Timeout" error. You indicated (this morning) that your example gave you the same "truncated response" error. These have quite different meanings
Carlos wrote: | kris wrote: | If you'd like to help in return, you might post the exact response from this third-party server? Write a client, using sockets only, to send a request and capture the response. I'd very much like to see what that response is. Both in text and hex, if you can provide that? |
I have posted that, not in hex but in text. Well, not explicitily, but what I'm sending in the test server is what the real server sends. |
I'm afraid that doesn't help very much. You see, it doesn't show what the original EOL terminators are. These can easily get changed when you cut and paste from one place to another. That's why I asked for the hex version also.
Jetty is a good server. I'm very surprised to see there's any issue here at all |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Fri May 26, 2006 3:40 pm Post subject: |
|
|
Carlos wrote: | I tried to debug HtppClient.d and I found that the LineIterator in line 490 doesn't return anything, so it's like it's not receiving what the server is sending. |
Code: | // skip any blank lines
while (line.next && line.get.length is 0)
{}
|
If that code ends up waiting forever, then it will eventually timeout and you'll get an appropriate error message. But, you're saying you get a "truncated response" error instead? If so, then something must have been returned for that to occur (must have got past the above code). This is why I need to see the raw data being returned by the Jetty server. |
|
Back to top |
|
|
Carlos
Joined: 19 Mar 2004 Posts: 396 Location: Canyon, TX
|
Posted: Fri May 26, 2006 4:54 pm Post subject: |
|
|
(brain fart edited)
Here's a sample of what I got:
Code: |
48 H
54 T
54 T
50 P
2f /
31 1
2e .
31 1
20
32 2
30 0
30 0
20
4f O
4b K
0d
0a
44 D
61 a
74 t
65 e
3a :
20
46 F
72 r
69 i
2c ,
20
32 2
36 6
20
4d M
61 a
79 y
20
32 2
30 0
30 0
36 6
20
32 2
32 2
3a :
34 4
39 9
3a :
33 3
39 9
20
47 G
4d M
54 T
0d
0a
53 S
65 e
72 r
76 v
65 e
72 r
3a :
20
4a J
65 e
74 t
74 t
79 y
2f /
35 5
2e .
31 1
2e .
31 1
30 0
20
28 (
4d M
61 a
63 c
20
4f O
53 S
20
58 X
2f /
31 1
30 0
2e .
34 4
2e .
36 6
20
70 p
70 p
63 c
20
6a j
61 a
76 v
61 a
2f /
31 1
2e .
35 5
2e .
30 0
5f _
30 0
36 6
0d
0a
43 C
6f o
6e n
74 t
65 e
6e n
74 t
2d -
54 T
79 y
70 p
65 e
3a :
20
61 a
70 p
70 p
6c l
69 i
63 c
61 a
74 t
69 i
6f o
6e n
2f /
78 x
6d m
6c l
0d
0a
43 C
6f o
6e n
74 t
65 e
6e n
74 t
2d -
4c L
65 e
6e n
67 g
74 t
68 h
3a :
20
39 9
38 8
36 6
0d
0a
4c L
61 a
73 s
74 t
2d -
4d M
6f o
64 d
69 i
66 f
69 i
65 e
64 d
3a :
20
54 T
68 h
75 u
2c ,
20
31 1
31 1
20
4d M
61 a
79 y
20
32 2
30 0
30 0
36 6
20
30 0
30 0
3a :
35 5
36 6
3a :
32 2
34 4
20
47 G
4d M
54 T
0d
0a
41 A
63 c
63 c
65 e
70 p
74 t
2d -
52 R
61 a
6e n
67 g
65 e
73 s
3a :
20
62 b
79 y
74 t
65 e
73 s
0d
0a
43 C
6f o
6e n
6e n
65 e
63 c
74 t
69 i
6f o
6e n
3a :
20
63 c
6c l
6f o
73 s
65 e
0d
0a
0d
0a
3c <
3f ?
78 x
6d m
6c l
20
76 v
65 e
72 r
73 s
69 i
6f o
6e n
3d =
22 "
31 1
2e .
30 0
22 "
20
65 e
6e n
63 c
6f o
64 d
69 i
6e n
67 g
3d =
22 "
55 U
54 T
46 F
2d -
38 8
22 "
3f ?
3e >
20
0a
0a
3c <
21 !
2d -
2d -
0a
20
20
20
20
4f O
6e n
65 e
20
6f o
66 f
20
74 t
68 h
65 e
20
73 s
69 i
6d m
70 p
6c l
65 e
73 s
74 t
20
77 w
66 f
64 d
2e .
20
41 A
64 d
64 d
72 r
65 e
73 s
73 s
65 e
73 s
20
61 a
20
77 w
6f o
72 r
6b k
69 i
74 t
65 e
6d m
20
74 t
6f o
20
74 t
77 w
6f o
20
70 p
61 a
72 r
74 t
69 i
63 c
69 i
70 p
61 a
6e n
74 t
73 s
20
0a
20
20
20
20
73 s
65 e
71 q
75 u
65 e
6e n
74 t
69 i
61 a
6c l
6c l
79 y
2e .
0a
0a
20
20
20
20
24 $
49 I
64 d
3a :
20
66 f
6c l
6f o
77 w
5f _
5f _
31 1
2e .
30 0
2e .
78 x
6d m
6c l
20
32 2
31 1
37 7
31 1
20
32 2
30 0
30 0
35 5
2d -
31 1
30 0
2d -
31 1
36 6
20
31 1
39 9
3a :
30 0
34 4
3a :
31 1
30 0
5a Z
20
6a j
6d m
65 e
74 t
74 t
72 r
61 a
75 u
78 x
20
24 $
0a
2d -
2d -
3e >
0a
0a
3c <
70 p
72 r
6f o
63 c
65 e
73 s
73 s
2d -
64 d
65 e
66 f
69 i
6e n
69 i
74 t
69 i
6f o
6e n
20
0a
20
20
20
20
6e n
61 a
6d m
65 e
3d =
22 "
66 f
6c l
6f o
77 w
22 "
20
0a
20
20
20
20
72 r
65 e
76 v
69 i
73 s
69 i
6f o
6e n
3d =
22 "
31 1
2e .
30 0
22 "
0a
3e >
0a
0a
20
20
20
20
3c <
64 d
65 e
73 s
63 c
72 r
69 i
70 p
74 t
69 i
6f o
6e n
3e >
0a
54 T
68 h
69 i
73 s
20
66 f
6c l
6f o
77 w
20
77 w
69 i
6c l
6c l
20
73 s
65 e
6e n
64 d
20
74 t
68 h
65 e
20
77 w
6f o
72 r
6b k
69 i
74 t
65 e
6d m
20
74 t
6f o
20
72 r
6f o
6c l
65 e
2d -
61 a
6c l
70 p
68 h
61 a
2e .
20
57 W
68 h
65 e
6e n
20
72 r
6f o
6c l
65 e
2d -
61 a
6c l
70 p
68 h
61 a
20
72 r
65 e
70 p
6c l
69 i
65 e
73 s
2c ,
20
74 t
68 h
65 e
20
77 w
6f o
72 r
6b k
69 i
74 t
65 e
6d m
20
69 i
73 s
20
74 t
68 h
65 e
6e n
20
73 s
65 e
6e n
74 t
20
74 t
6f o
20
72 r
6f o
6c l
65 e
2d -
62 b
72 r
61 a
76 v
6f o
2c ,
20
61 a
6e n
64 d
20
74 t
68 h
65 e
6e n
20
74 t
6f o
20
72 r
6f o
6c l
65 e
2d -
63 c
68 h
61 a
72 r
6c l
79 y
0a
20
20
20
20
3c <
2f /
64 d
65 e
73 s
63 c
72 r
69 i
70 p
74 t
69 i
6f o
6e n
3e >
0a
0a
20
20
20
20
3c <
73 s
65 e
71 q
75 u
65 e
6e n
63 c
65 e
3e >
0a
0a
09
3c <
70 p
72 r
65 e
2d -
70 p
61 a
72 r
74 t
69 i
63 c
69 i
70 p
61 a
6e n
74 t
20
72 r
65 e
66 f
3d =
22 "
72 r
6f o
6c l
65 e
2d -
61 a
6c l
70 p
68 h
61 a
22 "
20
64 d
65 e
73 s
63 c
72 r
69 i
70 p
74 t
69 i
6f o
6e n
3d =
22 "
70 p
6c l
65 e
61 a
73 s
65 e
20
61 a
64 d
64 d
20
66 f
69 i
65 e
6c l
64 d
73 s
2e .
2e .
2e .
22 "
20
2f /
3e >
0a
09
3c <
21 !
2d -
2d -
3c <
73 s
65 e
74 t
20
66 f
69 i
65 e
6c l
64 d
3d =
22 "
6f o
70 p
65 e
6e n
77 w
66 f
65 e
5f _
76 v
65 e
72 r
73 s
69 i
6f o
6e n
22 "
20
76 v
61 a
6c l
75 u
65 e
3d =
22 "
24 $
7b {
63 c
6f o
6e n
73 s
74 t
3a :
6f o
70 p
65 e
6e n
77 w
66 f
65 e
2e .
6f o
72 r
67 g
2e .
65 e
6e n
67 g
69 i
6e n
65 e
2e .
44 D
65 e
66 f
69 i
6e n
69 i
74 t
69 i
6f o
6e n
73 s
2e .
4f O
50 P
45 E
4e N
57 W
46 F
45 E
5f _
56 V
45 E
52 R
53 S
49 I
4f O
4e N
7d }
22 "
20
2f /
3e >
2d -
2d -
3e >
0a
09
3c <
73 s
65 e
74 t
20
66 f
69 i
65 e
6c l
64 d
3d =
22 "
63 c
75 u
73 s
74 t
6f o
6d m
65 e
72 r
5f _
6e n
61 a
6d m
65 e
22 "
20
76 v
61 a
6c l
75 u
65 e
3d =
22 "
66 f
69 i
6c l
6c l
20
74 t
68 h
69 i
73 s
20
66 f
69 i
65 e
6c l
64 d
22 "
20
2f /
3e >
0a
09
3c <
73 s
65 e
74 t
20
66 f
69 i
65 e
6c l
64 d
3d =
22 "
66 f
6c l
61 a
67 g
22 "
20
76 v
61 a
6c l
75 u
65 e
3d =
22 "
74 t
72 r
75 u
65 e
22 "
20
74 t
79 y
70 p
65 e
3d =
22 "
62 b
6f o
6f o
6c l
65 e
61 a
6e n
22 "
20
2f /
3e >
0a
0a
20
20
20
20
20
20
20
20
3c <
21 !
2d -
2d -
0a
20
20
20
20
20
20
20
20
3c <
73 s
65 e
74 t
20
66 f
69 i
65 e
6c l
64 d
3d =
22 "
74 t
61 a
72 r
67 g
65 e
74 t
22 "
20
76 v
61 a
6c l
75 u
65 e
3d =
22 "
24 $
7b {
66 f
3a :
63 c
75 u
73 s
74 t
6f o
6d m
65 e
72 r
2e .
6e n
61 a
6d m
65 e
7d }
22 "
20
2f /
3e >
0a
20
20
20
20
20
20
20
20
3c <
73 s
65 e
74 t
20
66 f
69 i
65 e
6c l
64 d
3d =
22 "
74 t
61 a
72 r
67 g
65 e
74 t
32 2
22 "
20
76 v
61 a
6c l
75 u
65 e
3d =
22 "
24 $
7b {
66 f
3a :
63 c
75 u
73 s
74 t
6f o
6d m
65 e
72 r
73 s
2e .
31 1
7d }
22 "
20
2f /
3e >
0a
20
20
20
20
20
20
20
20
2d -
2d -
3e >
0a
0a
09
3c <
70 p
61 a
72 r
74 t
69 i
63 c
69 i
70 p
61 a
6e n
74 t
20
72 r
65 e
66 f
3d =
22 "
72 r
6f o
6c l
65 e
2d -
62 b
72 r
61 a
76 v
6f o
22 "
20
2f /
3e >
0a
09
3c <
70 p
61 a
72 r
74 t
69 i
63 c
69 i
70 p
61 a
6e n
74 t
20
72 r
65 e
66 f
3d =
22 "
72 r
6f o
6c l
65 e
2d -
63 c
68 h
61 a
72 r
6c l
79 y
22 "
20
2f /
3e >
0a
20
20
20
20
3c <
2f /
73 s
65 e
71 q
75 u
65 e
6e n
63 c
65 e
3e >
0a
0a
3c <
2f /
70 p
72 r
6f o
63 c
65 e
73 s
73 s
2d -
64 d
65 e
66 f
69 i
6e n
69 i
74 t
69 i
6f o
6e n
3e >
0a
|
BTW, I think LineIterator.scan is not parsing correctly Mac EOLs. |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Fri May 26, 2006 6:02 pm Post subject: |
|
|
Thanks; and that looks perfect. Guess I have no idea why it fails. Being able to replicate the problem is probably the only way I can figure this out now.
LineIterator.scan() parses either a '\n' or a '\r\n' combination. What does the Mac use? |
|
Back to top |
|
|
Carlos
Joined: 19 Mar 2004 Posts: 396 Location: Canyon, TX
|
Posted: Fri May 26, 2006 7:32 pm Post subject: |
|
|
Mac traditionally has used \r, but I think more and more applications are going with \n as Mac OS X is a Unix too.
Anyway, here's what I get from the real server (the first one):
Code: |
48 H
54 T
54 T
50 P
2f /
31 1
2e .
30 0
20
32 2
30 0
30 0
20
4f O
4b K
0d
0a
44 D
61 a
74 t
65 e
3a :
20
53 S
61 a
74 t
2c ,
20
32 2
37 7
20
4d M
61 a
79 y
20
32 2
30 0
30 0
36 6
20
30 0
30 0
3a :
34 4
31 1
3a :
30 0
30 0
20
47 G
4d M
54 T
0d
0a
53 S
65 e
72 r
76 v
65 e
72 r
3a :
20
77 w
6f o
72 r
6b k
6c l
69 i
73 s
74 t
0d
0a
43 C
6f o
6e n
74 t
65 e
6e n
74 t
2d -
74 t
79 y
70 p
65 e
3a :
20
74 t
65 e
78 x
74 t
2f /
78 x
6d m
6c l
0d
0a
0d
0a
3c <
3f ?
78 x
6d m
6c l
20
76 v
65 e
72 r
73 s
69 i
6f o
6e n
3d =
22 "
31 1
2e .
30 0
22 "
20
65 e
6e n
63 c
6f o
64 d
69 i
6e n
67 g
3d =
22 "
49 I
53 S
4f O
2d -
38 8
38 8
35 5
39 9
2d -
31 1
22 "
3f ?
3e >
0d
0a
3c <
73 s
65 e
73 s
73 s
69 i
6f o
6e n
20
69 i
64 d
3d =
22 "
31 1
31 1
34 4
38 8
36 6
39 9
30 0
34 4
35 5
39 9
39 9
32 2
38 8
22 "
20
2f /
3e >
0d
0a
0d
0a
0d
0a
0d
0a
0d
0a
0d
0a
|
This is the code for the test server which produces that same output:
Code: |
import std.cstream,
std.socket,
std.socketstream;
int main (char [][] args)
{
auto server = new TcpSocket ();
server.bind (new InternetAddress (50800));
server.listen (10);
auto client = server.accept ();
auto str = new SocketStream (client);
while(true)
{
char [] ln = str.readLine();
dout.writeLine(ln);
if (ln=="")
break;
}
str.writeString ("HTTP/1.0 200 OK\r\n"c);
str.writeString ("Date: Sat, 27 May 2006 00:41:00 GMT\r\n"c);
str.writeString ("Server: worklist\r\n"c);
str.writeString ("Content-type: text/xml\r\n"c);
str.writeString ("\r\n"c);
str.writeString ("<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>\r\n"c);
str.writeString ("<session id=\"1148690459928\" />\r\n"c);
str.writeString ("\r\n"c);
str.writeString ("\r\n"c);
str.writeString ("\r\n"c);
str.writeString ("\r\n"c);
str.writeString ("\r\n"c);
str.flush ();
str.close ();
server.close ();
return 0;
}
|
The client would be the same I posted in the first message. |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Fri May 26, 2006 8:00 pm Post subject: |
|
|
hehe ... now I'm hopelessly confused
How many different servers do you have? You have the Jetty server, an example server, and something else recently called the "real server" ?
I'm also thoroughly confused about the client you recently speak of ... the only client posted in this topic is the one that uses HttpClient on page1; yes? So, which client are you referring to immediately above? A socket-based client used to capture the hex listing?
Finally, the two hex listings you posted are presumed to be what was actually captured by some socket-based client ~ the one that hasn't been posted yet? I'm assuming that one was run against the Jetty server, and then against the other "real server", since the listed responses are different?
thanks, and sorry about the confusion |
|
Back to top |
|
|
Carlos
Joined: 19 Mar 2004 Posts: 396 Location: Canyon, TX
|
Posted: Fri May 26, 2006 9:38 pm Post subject: |
|
|
kris wrote: | hehe ... now I'm hopelessly confused |
Sorry about that... hehe...
kris wrote: | How many different servers do you have? You have the Jetty server, an example server, and something else recently called the "real server" ? |
Ok, this application opens lots of ports all (I think) listening HTTP. Some of those are on top of Jetty. However, the one I'm more interested in is not Jetty, but provides a simple REST interface to access the application. My sample server simulates this REST server, and before I also tried HttpClient against one of the Jetty servers to see how it went.
kris wrote: | I'm also thoroughly confused about the client you recently speak of ... the only client posted in this topic is the one that uses HttpClient on page1; yes? So, which client are you referring to immediately above? A socket-based client used to capture the hex listing? |
My idea was that you can test this server I just posted (which simulates precisely a response of the REST server) with the client I posted in the first message in page 1.
kris wrote: | Finally, the two hex listings you posted are presumed to be what was actually captured by some socket-based client ~ the one that hasn't been posted yet? I'm assuming that one was run against the Jetty server, and then against the other "real server", since the listed responses are different? |
Yes, but I don't think that client (hex-dumper) is important. Also, the output from the Jetty server was just an example. You said you wanted to reproduce the server I had used first, so I wrote a simulation of such server (REST) and you can see how that goes.
kris wrote: | thanks, and sorry about the confusion |
I hope the confusion is over now. Basically, if you want to test it, use my last server and my client. |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Sat May 27, 2006 12:10 am Post subject: |
|
|
Thanks;
I just ran your server in conjunction with the mango example "httpget.d" (which used HttpClient and accepts an URL as a command-line argument). I made three changes to your server example:
1) changed dout calls to printf instead, so I couldn actually see something happen. I guess dout needs to be flushed?
2) I added a while(true) loop around the body of the server, so that it does not exit immediately after sending a response.
3) I removed the str.close()
It works fine like that. Let's look at why that was needed: ignore #1, since it's purely diagnostic (at least, I hope it is!). Both #2 & #3 are required, since the server has to allow time for the response to be returned (over the wire) before closing the socket.
When a socket is closed, it typically goes into hibernation for a while to allow the last packet to complete its journey. This is usually handled by the OS. However, what happens in the case of #2 is that the socket is (apparently) closed hard when the program exits. In the case of #3, I don't know how Phobos is actually closing the socket, but it has the same effect as #2.
That effect results in the following exception raised by HttpClient, via the httpget.d example: "Error: during socket recieve: An existing connection was forcibly closed by the remote host".
In english, the TCP/IP-protocol was not given sufficient time to perform a controlled closure. Thus, this example server needs some changes to make it operate correctly at the socket level. Once those changes are made, HttpClient works fine with it.
I should also note that I've completely failed to trigger the "truncated response" error message you'd been seeing. I don't even know how to trigger it on purpose
To sum up: given that the example server works (once corrected), I'm at a loss of what to suggest next. Any ideas? What error do you get when running httpget.d against this server? |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
|
Back to top |
|
|
Carlos
Joined: 19 Mar 2004 Posts: 396 Location: Canyon, TX
|
Posted: Sat May 27, 2006 6:39 am Post subject: |
|
|
There're a number of things here.
First, wsock is not important here as I'm on Mac. Second, since I'm on Mac, I believe part of the different behavior we've seen might be due to the operating systems handling sockets in different ways.
Third, I ran the httpget example but I forgot to pass the argument (luckily, I was connected to the internet then). Then I tried again with the actual address and it was again "truncated response". The interesting part is that when I ran the httpget example alone, I also got that error.
Based on that, I thought the problem is related to another one I reported this week: I versioned out ServerSocket.d:76 "setAddressReuse (socketReuse);", but then I realized there should be no relation between HttpClient and ServerSocket. Then I thought about other changes I made to Mango, and none of them are related to sockets by any chance.
So I think this could be about how the OS does things, and maybe Mango doesn't play too well with it. I think the closest you can get to my environment could be with Linux. And even so, it's not that similar.
You know what's the weirdest thing? I also wrote a client which reads from the socket as HttpClient (with a LineIterator, skipping blanks, etc) and it didn't throw.
Another thing I could is take HttpClient and write a "CarlosHttpClient" based on that and see adding what is triggering the error.
I think that's about everything I have in my mind right now. |
|
Back to top |
|
|
Carlos
Joined: 19 Mar 2004 Posts: 396 Location: Canyon, TX
|
Posted: Sat May 27, 2006 7:56 am Post subject: |
|
|
It's the timeout. When I versioned out socket.setTimeout in HttpClient.open, the error went away. |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Sat May 27, 2006 12:13 pm Post subject: |
|
|
OK; good.
That indicates the mac/bsd API for socket.select() is incompatible at some level (the timer values apparently mean different things than on Linux & Win32, for example). That's kinda' funny, since the socket layer originated on BSD |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|