View previous topic :: View next topic |
Author |
Message |
Cyborg16
Joined: 28 Apr 2007 Posts: 39
|
Posted: Thu May 17, 2007 7:13 am Post subject: API |
|
|
Hi,
I saw this project on the yage forum and here... and have some questions/comments. First I should say that I have no experience with networking, but some experience developing a game and some other software. Also I'm currently moving from C++ to D, so the code may not all make sense
Presumably the low-level API works by passing a ready-made data block and an ID (although the ID could just be part of the data block) to the library and saying send:
Code: | void dnet.send(uint ID, void* data); |
And how do you recieve data? A very inefficient way would be something like this:
Code: | void* dnet.wait(uint ID, uint timeout); /// wait up to timeout millisec for packet ID, return void* to data or return null | A more efficient method might be to run the library in a separate thread which constantly waits for packets and give it a function pointer or delegate to a function to run when a packet is received. OK, so that isn't really different from using the first method in a separate thread, but accepting packets with any ID and letting the programmer decide what to do with them...
About the high-level API example posted here,
all this API seems to do is create packets from a few variables, then synchronise them, which doesn't seem all that useful to me. Something like this, which isn't all that different functionally, seems like a slightly more useful approach to the end programmer, although it's not much different (this is just an easier way to set data IMO):
Code: | struct Data {...} // a container for data to be sent
data = new Data; // initialise some data
dnet.server s; // a server class (or client class for the client)
// init the server/client somehow...
s.addDataToSynchronise(cast(void*) data); // give the server some data to synchronise
while (running) { // main loop
...
s.synchronise(); // now synchronise everything
} |
This could be extended in a useful way (at least it would make things look neater):
Code: | // a base class in the dnet high-level library:
class SyncBase
{
void* getSyncData (out uint size); /// virtual func to creates and returns a data block of all vars in the deriving class which need to be syncronised
void setSyncData(void* data, uint size); /// virtual func to read a received data block and set vars
}
// Now in the program using the library:
dnet.Server/Client s; // create server/client
// init
class Data : SyncBase // example deriving the base class
{
int x;
void* getSyncData (out uint size)
{
size = sizeof(int); // note: size doesn't have to be constant for a given class
return cast(void*) x;
}
void setSyncData(void* data, uint size)
{
if (size != sizeof(int)) error;
x = cast(int) data;
}
}
Data data;
s.addSyncClass(data);
while (...) {
...
s.sync(); // at the server: call getSyncData() to get a data block, and send it
// at the client: call setSyncData() with the received data
} |
I may not be helping much, I'm really only trying to invent methods that look usable from the end programmers perspective with no experience on this...
Also, this high-level library only really works for a server-client model. However, any other model would require a fair bit of ingenuity on the part of the end programmer, so maybe it should only use the low-level lib. Maybe I'll try and work out some ideas for a synchronus model (i.e. both ends run a simulation and somehow merge the results) at some time.. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
bane
Joined: 01 May 2007 Posts: 41 Location: Pancevo, Serbia
|
Posted: Thu May 17, 2007 7:41 am Post subject: |
|
|
dnet is spawning 2 threads, one for listening on blocking socket and other doing send/decrypting packets. Blocking socket means you dont have to repeat loop indefinitelly
{{{
char[] buff = socket.read()
}}}
becouse socket.read will return value only when new packet arrives, untill then thread is frozen.
I suggest you read some basic info on sockets to get some info.
Well, server/client model seems to me most simplest at the moment. Shared calculations on more than one host lead to some issuess:
- connected clients need to get data from more than one host on connecting, allso to report back
- if one of "calculating" hosts drops out, what happens with whole picture? Who kicks back in, and with what data, when that on the server might be lost?
As far for "simplest api implementation" that really sometimes depends of persons taste. I'm trying to make simplest as "most obvious with least typing and options" couse I am lazy and in 2 weeks I am going to forget API i wrote myself so Ill have to learn it again. _________________ "I apologize only for my spelling" - a quote shamelessly stolen from some guy at unknown forum. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
Cyborg16
Joined: 28 Apr 2007 Posts: 39
|
Posted: Thu May 17, 2007 8:22 am Post subject: |
|
|
Thanks, I guess reading up on sockets before I do any net coding would make sense... likely I won't be doing any net coding for a long while though.
The alternative to the client/server model I was suggesting was that all applications do a full simulation and synchronise with each other, so there doesn't need to be a server. If one client drops out, all the rest are doing the same simulation, so it wouldn't make much difference. You have got a point that all clients need to communicate with each other though... probably resulting in a lot more network traffic with more than two players. This method would need a lot of working out obviously.
The advantages are: players can connect without a dedicated server, and without one server, stopping the game if it drops out.
OK, as I said before I don't know much about networking, and maybe this isn't the best idea. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
bane
Joined: 01 May 2007 Posts: 41 Location: Pancevo, Serbia
|
Posted: Thu May 17, 2007 1:34 pm Post subject: |
|
|
I know much only about server/client model. This one you are talking about is maybe peer to peer model (I dont know).
Another problem except much higher network usage is this one to (imagine FPS):
if host A calculates that you just got hit by a bullet and host B calculates bullet missed you because he has 5 ms lag (happened zillion times in counter strike, for example), which one will right? A or B?
With server clients server will always be right, even if he is wrong. Clients will obey and all of them will have the same reality.
with server and clients there is only one host to do prediction, lag time calculations and other stuff - with server. with your model there is nightmare of predictions on each host toward all other host.
If you have time to build such a complicated engine, then go for it ![Smile](images/smiles/icon_smile.gif) _________________ "I apologize only for my spelling" - a quote shamelessly stolen from some guy at unknown forum. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
bane
Joined: 01 May 2007 Posts: 41 Location: Pancevo, Serbia
|
Posted: Thu May 17, 2007 1:48 pm Post subject: |
|
|
I don't say i dislike model you suggest but, like in many other areas of production, simpler and easier to build solution is many times more convenient than correct but complicated (expensive) one.
I don't have enough resources (time & knowledge) to do that model. _________________ "I apologize only for my spelling" - a quote shamelessly stolen from some guy at unknown forum. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
Cyborg16
Joined: 28 Apr 2007 Posts: 39
|
Posted: Fri May 18, 2007 4:02 am Post subject: |
|
|
I'm certainly not asking you to implement a model like that - it looks like it would need to be heavily involved with the engine anyway, so couldn't really just be a network lib.
You've pointed out some more things I hadn't thought of. Hmm, this type of model seems horribly complicated with just two players; it could be much worse with more (although largely it would follow the same principles).
At any rate I should do a lot of reading up on networking methods, etc, before trying to do anything like this I guess. I won't be trying to implement any network code for quite a while anyway though. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
bane
Joined: 01 May 2007 Posts: 41 Location: Pancevo, Serbia
|
Posted: Fri May 18, 2007 12:44 pm Post subject: |
|
|
Only way to test the idea is to make it and put it in use.
And only then you see how horribly wrong you were when you first made idea
Happened with me X times implementing versions of dnet/dogslow in a game until I got to a point to know what would really work for me.
That was back in days playing with C++, enet & irrlicht engine. Now I am more of D-type of person... ![Smile](images/smiles/icon_smile.gif) _________________ "I apologize only for my spelling" - a quote shamelessly stolen from some guy at unknown forum. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
davidb
Joined: 03 Apr 2007 Posts: 15
|
Posted: Sat May 19, 2007 2:16 pm Post subject: |
|
|
@Cyborg16's API suggestion
Well, I find this very appealing because as I understand it you can leave out the intermediate step of transforming your data by
Code: | a.setString("model", "Ford");
a.setVector("position", vector(0,33,12));
a.setFloat("speed", 33.4); |
But I'm a bit concerned about synchronization. How could you decide which data to send (i.e. which data has to be sent because it was updated)?
Although this seems a minor concern at first, the larger the data you use becomes the more it hits the bandwidth. That said, I think Cyborg16's API suggestion would be very easy to use, but the concern mentioned above remains unsolved (at least I don't know how to implement it in this way), and so I would persue bane's ideas.
Just my 2 cents,
david |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
bane
Joined: 01 May 2007 Posts: 41 Location: Pancevo, Serbia
|
Posted: Mon May 21, 2007 8:17 am Post subject: |
|
|
davidb
I am not quite sure what are you refering to: dnet library has two parts. low level (dnet - same name) is for UDP comunicating only, while higher level (dogslow), which you quoted, is used for replication. it is up to you will you use this replication model or will you build your own on top of low level or will you use it at all
low level is basically complete, while dogslow is finished in few days. you can see most updated api in trunk/api generated bu ddoc _________________ "I apologize only for my spelling" - a quote shamelessly stolen from some guy at unknown forum. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
davidb
Joined: 03 Apr 2007 Posts: 15
|
Posted: Mon May 21, 2007 2:29 pm Post subject: Re: API |
|
|
@bane: As mentioned I was referring to Cyborg16's API suggestion in the first post:
Cyborg16 wrote: | Hi,
...
About the high-level API example posted here,
all this API seems to do is create packets from a few variables, then synchronise them, which doesn't seem all that useful to me. Something like this, which isn't all that different functionally, seems like a slightly more useful approach to the end programmer, although it's not much different (this is just an easier way to set data IMO):
Code: | struct Data {...} // a container for data to be sent
data = new Data; // initialise some data
dnet.server s; // a server class (or client class for the client)
// init the server/client somehow...
s.addDataToSynchronise(cast(void*) data); // give the server some data to synchronise
while (running) { // main loop
...
s.synchronise(); // now synchronise everything
} |
This could be extended in a useful way (at least it would make things look neater):
Code: | // a base class in the dnet high-level library:
class SyncBase
{
void* getSyncData (out uint size); /// virtual func to creates and returns a data block of all vars in the deriving class which need to be syncronised
void setSyncData(void* data, uint size); /// virtual func to read a received data block and set vars
}
// Now in the program using the library:
dnet.Server/Client s; // create server/client
// init
class Data : SyncBase // example deriving the base class
{
int x;
void* getSyncData (out uint size)
{
size = sizeof(int); // note: size doesn't have to be constant for a given class
return cast(void*) x;
}
void setSyncData(void* data, uint size)
{
if (size != sizeof(int)) error;
x = cast(int) data;
}
}
Data data;
s.addSyncClass(data);
while (...) {
...
s.sync(); // at the server: call getSyncData() to get a data block, and send it
// at the client: call setSyncData() with the received data
} |
|
... and voiced my opinion on that, see my last post.
That said, great work on dnet bane!
david |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
bane
Joined: 01 May 2007 Posts: 41 Location: Pancevo, Serbia
|
Posted: Tue May 22, 2007 12:59 am Post subject: |
|
|
thanx. still a long way to go ![Wink](images/smiles/icon_wink.gif) _________________ "I apologize only for my spelling" - a quote shamelessly stolen from some guy at unknown forum. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
Dima-san
Joined: 25 Mar 2007 Posts: 33 Location: Almaty, Kazakhstan
|
Posted: Fri May 25, 2007 11:00 am Post subject: |
|
|
Hi everyone,
bane
First of all, good work.
Why don't you make a lower level thing, something lower than dnet? I mean, why don't make a thing for building a single connection, like, just connecting two endpoints? Having such thing implemented, it'd be a little effort to build a client and server architecture, and it'll give a greater degree of freedom to the library users.
You see, I'm gonna do some research on distributed continous media, and I'd like to have an opportunity to build a peer-to-peer app with dnet.
Now on dnet. I had a glance at the code, and I suppose timeout delay should be increased a bit, and the one to ensure that packets are flowing must be the library user, not you.
There are some issues you should be warned of.
Network address translating routers sometimes remap packet destination ports, so, if you don't wanna get nice glitches during gameplay, you should think of a workaround.
The connection process should be little more complex, to avoid DoS attacks.
Still, I have a little spare time, so I can help only with my advices. I hope they will be somewhat useful.
Also, I've written the Replicator test program. It works . And the prediction works, too
Regards,
Dima-san |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
bane
Joined: 01 May 2007 Posts: 41 Location: Pancevo, Serbia
|
Posted: Fri May 25, 2007 3:03 pm Post subject: |
|
|
Thanx, Dima-San
I consider your suggestions and advices extremely helpful. And for sake of usefulness of dnet not only for you and me but for all users, I will give my best to incorporate them in this project.
So, to get down and dirty on the business:
1. You suggest something like NetConnection object for connecting two points, that creates like
NetConnection("other_end_host", other_end_port);
That object itself can have reliable/unreliable sending method and take care of other packet sending stuff. On this client/server can be built.
2. Connection process should have on server something like "don't spam me with more than x connection attempts per second or I'll put your ass on ignore blacklist" test? This way we can have some protection against DoS attacks?
3. I read something about spawning new listener on other port for incoming connection can have benefit (maybe that was only with TCP). For example, server listens on port 3000. There is connection request, and server answers with, "ok, you are in, join on port 3001" and binds new listener on that port only for that new client. Is this faster than all the clients using same port/only one listener?
4. Quote: | Network address translating routers sometimes remap packet destination ports, so, if you don't wanna get nice glitches during gameplay, you should think of a workaround. |
I don't quite get this part. Is it somehow related to item no 3. What do you suggest for solving problem?
Hope we can meet in the middle so replicator can use dnet to
Best regards _________________ "I apologize only for my spelling" - a quote shamelessly stolen from some guy at unknown forum. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
Dima-san
Joined: 25 Mar 2007 Posts: 33 Location: Almaty, Kazakhstan
|
Posted: Sat May 26, 2007 12:19 pm Post subject: |
|
|
bane
Can I expect to find my name at dnet credits?
Quote: | 1. You suggest something like NetConnection object for connecting two points, that creates like NetConnection("other_end_host", ther_end_port); |
Yes, that is what I suggest
Quote: | 2. Connection process should have on server something like "don't spam me with more than x connection attempts per second or I'll put your ass on ignore blacklist" test? |
That's it, bane! You could also add a ban list thingy to the dnet api. dnet listener would just ignore packets from IPs in the ban list.
Quote: | 3. I read something about spawning new listener on other port for incoming connection can have benefit (maybe that was only with TCP). |
I believe that approach may have benefit in case of large number of connections, so the processing can be distributed across multiple threads.
Quote: |
Quote: | Network address translating routers sometimes remap packet destination ports, so, if you don't wanna get nice glitches during gameplay, you should think of a workaround. |
I don't quite get this part. Is it somehow related to item no 3. What do you suggest for solving problem? |
An additional randomly generated port can be used to detect whether port number has been changed by a NAT router. When such change is detected, server should update the port number of client's address with the new one.
This also gives you an opportunity to distinguish between two clients with the same address.
Quote: | Hope we can meet in the middle so replicator can use dnet to Smile |
I'll do what I can for this to happen ![Wink](images/smiles/icon_wink.gif) |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
bane
Joined: 01 May 2007 Posts: 41 Location: Pancevo, Serbia
|
Posted: Sat May 26, 2007 6:00 pm Post subject: Step by step... |
|
|
Thanx to Dima-Sans suggestions, I rewrote dnet's logic to something more usable for both p2p and client/server. Goodies lie in the trunk. I removed dogslow and unrelated stuff for now. I want to focus on dnet only untill 0.2.
More than that, I tested dnet server with 20 clients connected. Server managed to keep flooding rate of 65 small packets per second toward each of the clients. No significant CPU usage is detected. That is a gread improvement from 0.1 _________________ "I apologize only for my spelling" - a quote shamelessly stolen from some guy at unknown forum. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|