FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Brainstorming - Sumac

 
Post new topic   Reply to topic     Forum Index -> DSP
View previous topic :: View next topic  
Author Message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Dec 23, 2005 12:32 pm    Post subject: Brainstorming - Sumac Reply with quote

(Thinking aloud and journaling my notes on the topic -- posts are welcome)

I've been kicking around the idea of a more universal and configurable HTTP server for a while now. It came about when it was obvious that DSP would need a more complete testbed, and somthing more configurable than what is offered in the mango "servlets" example.

I decided that an XML-config file driven approach, plus the dynamic load capabilities of DDL would fit the bill nicely.

Code:
<sumac>
   <ddl>
      <!-- layout would be different for ares, or these references
         could dissapear into the .exe completely -->
      <lib path="sumac.ddl"/>
      <lib path="phobos.ddl" version="0.141"/>
      <lib path="snn.ddl" version="0.141"/>
      <lib path="mango.ddl" version="1.7"/>
            
      <!-- auxillary libraries and plugins (DSP, DBBI, etc) -->
      <libpath path="plugins/"/>
   </ddl>
   
   <!-- example HTTP server - follows the equivalent code in mango/examples/servlets.d -->
   <server class="sumac.server.HTTPServer.HTTPServer"/>
      <port>80</port>
      <address>127.0.0.1</address>
      <threads>4</threads>
      
      <provider class="sumac.servlet.ServletProvider.ServletProvider">
         <context name="/example" class="sumac.servlet.ServletContext.ServletContext"/>
         <context name="/admin" class="sumac.log.Admin.AdminContext"/>
         
         <servlet mapping="/echo" name="echo" class="sumac.servlet.Example.Echo"/>
         <servlet mapping="/ping" name="ping" class="sumac.servlet.Example.Ping"/>
         
         <context name="" path="../doc/html" class="sumac.servlet.ServletContext.ServletContext"/>
         
         <servlet mapping="/" name="files" class="sumac.servlet.Example.FileServlet">
      </provider>
   </server>
   
   <!-- example HTTP server for DSP -->
   <server class="sumac.server.HTTPServer.HTTPServer"/>
      <port>8080</port>
      <address>127.0.0.1</address>
      <threads>4</threads>
      
      <provider class="dsp.servlet.ServletProvider.ServletProvider">
         <!-- possible admin module -->
         <context name="/admin" class="dsp.servlet.AdminContext.AdminContext"/>
         
         <!-- dsp test/conf suite -->
         <context name="/dspconf" class="dsp.servlet.ServletContext.ServletContext">
            <path>c:\dev\dsp\trunk\dspconf</path>
         </context>
                  
         <!-- dsp example application -->
         <context name="/" class="dsp.servlet.ServletContext.ServletContext"/>
            <path>c:\dev\dsp\trunk\testapp</path>
         </context>               
      </provider>
   </server>
   
   
   <!-- example FTP server -->
   <server class="sumac.server.FTPServer.FTPServer"/>
      <port>21</port>
      <address>127.0.0.1</address>
      <threads>4</threads>
      <ftp-root>c:\dev\sumac\trunk\ftproot</ftp-root>
      <user-file>c:\dev\sumac\trunk\ftpconf\users.txt</user-file>
      <pw-file>c:\dev\sumac\trunk\ftpconf\pw.txt</pw-file>
   </server>
</sumac>


Personally, I'd rather modify the above than your typical apache .conf file any day of the week.

The concept is simple: each node with a "class" attribute is instantiated and then configured with the XML enclosed within that node. As the config data is passed through the tree, the server is "built" at runtime via DDL. The process that kicks this all off would simply call configure() and then run() on the top-level "SumacServer" object.

Some pet-peeves I want to cover by creating Sumac Server are:

    - Easy configuration (see above)
    - Runtime manipulation, likely via an online admin suite
    - Restarting, reloading and reconfiguration at *any* level of configuration - servers, providers, contexts and servlets should all be in scope for this (allows for a whole slew of advantages and new behaviors).
    - Easy integration with additional technologies like database access or other protocol libraries. DDL maturity is a major factor here.
    - Side-channel support for debugging (get debug/log info as a web-chainsaw of sorts)
    - Transparent portability


Smile
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri Dec 23, 2005 12:59 pm    Post subject: Reply with quote

This is what the mango HTTP and Servlet engines will be doing as soon as DDL is available. I had played around with a Win32 version of the server that forked itself whenever a new binary was noticed ~ a nasty hack in lieu of DDL Smile

It's probably worth noting that mango.log already has full support for remote monitoring, and dynamic, remote, runtime control of said monitoring. If you'd be interested in combining efforts on this, that would be awesome!
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Dec 23, 2005 2:24 pm    Post subject: Reply with quote

kris wrote:
This is what the mango HTTP and Servlet engines will be doing as soon as DDL is available. I had played around with a Win32 version of the server that forked itself whenever a new binary was noticed ~ a nasty hack in lieu of DDL Smile

It's probably worth noting that mango.log already has full support for remote monitoring, and dynamic, remote, runtime control of said monitoring. If you'd be interested in combining efforts on this, that would be awesome!


That would be fantastic. How do you suggest we proceed?
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
AgentOrange



Joined: 21 Jul 2005
Posts: 61

PostPosted: Fri Dec 23, 2005 3:35 pm    Post subject: Reply with quote

I really like this idea. I have been planning to build a generic dynamic 'services' server with full http capabilities in D for a while now - it sounds similar to what you are describing here. I have a system written in C++ built out of a bit torrent tracker with a basic http server. I ported much of the code to D a long time ago which i was going to make a basic http library out of - so I could contribute to what you are doing here.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Dec 23, 2005 4:03 pm    Post subject: Reply with quote

AgentOrange wrote:
I really like this idea. I have been planning to build a generic dynamic 'services' server with full http capabilities in D for a while now - it sounds similar to what you are describing here. I have a system written in C++ built out of a bit torrent tracker with a basic http server. I ported much of the code to D a long time ago which i was going to make a basic http library out of - so I could contribute to what you are doing here.


Well, we're likely going to stick to using Mango for much of the I/O plumbing, but that torrent tracker of yours sounds neat. The more the merrier!

I probably won't get to it until the new-year, but I'm going to draft some interfaces for sumac and place them in the DSP SVN (after some major house-keeping). The three of us (or more if the community wants to contribute) can kick those around and see what is flexible enough for everyone's needs.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Sat Dec 24, 2005 2:34 am    Post subject: Reply with quote

pragma wrote:
kris wrote:
This is what the mango HTTP and Servlet engines will be doing as soon as DDL is available. I had played around with a Win32 version of the server that forked itself whenever a new binary was noticed ~ a nasty hack in lieu of DDL Smile

It's probably worth noting that mango.log already has full support for remote monitoring, and dynamic, remote, runtime control of said monitoring. If you'd be interested in combining efforts on this, that would be awesome!


That would be fantastic. How do you suggest we proceed?

My immediate thoughts:

If possible, I'd like to make it an upgrade to the existing server(s)? Would that be OK? If so, I see a need for four primary things:

1) the XML config spec, plus a parser. We should perhaps add logging configuration also? FWIW, mango.log follows the log4j configuration pattern, which is similar in approach to what you've outlined for the server
2) a contract between the loadable entities and the core server
3) an operational DDL
4) tweaks to the servers to take advantage of all the above

It seems that DDL is close enough right now to spike #4? If so, I could probably try a first pass of that part on Monday, to see how it goes? Unless you'd prefer to? There's a workable version of #2 in place for a spike, but it will probably need some refactoring (currently IProvider.d for the HTTP server specialization, and Servlet.d for the servlets).

What are your thoughts?
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sat Dec 24, 2005 8:05 am    Post subject: Reply with quote

kris wrote:

My immediate thoughts:

If possible, I'd like to make it an upgrade to the existing server(s)? Would that be OK? If so, I see a need for four primary things:

1) the XML config spec, plus a parser. We should perhaps add logging configuration also? FWIW, mango.log follows the log4j configuration pattern, which is similar in approach to what you've outlined for the server
2) a contract between the loadable entities and the core server
3) an operational DDL
4) tweaks to the servers to take advantage of all the above

It seems that DDL is close enough right now to spike #4? If so, I could probably try a first pass of that part on Monday, to see how it goes? Unless you'd prefer to? There's a workable version of #2 in place for a spike, but it will probably need some refactoring (currently IProvider.d for the HTTP server specialization, and Servlet.d for the servlets).

What are your thoughts?


That sounds about right. There are a couple of things that I see that may complicate all that.

Most notably, this effort is going to drag DDL by the nose into completion. That having been said, I find that efforts like this are almost necessary since I'd be developing DDL-side components in a vaccuum - its nice to get some actual use-based requirements for stuff like that. The same can be said for parts of Mango too. Smile

With respect to Mango's server components, the problem comes in that the Linker will need to be made thread-safe and will likely be passed down to the aggregate servers, providers, contexts and servlets so that those objects have that facility during runtime (take DSP for example). Combined with a config model that accepts the current XML config node, each component is simply passed what it needs to be set up.

Code:
interface SumacNode{
  void config(ddl.Linker linker, xml.dom.XMLNode node);
}


Developing the contracts for loadable modules is key. Were you thinking of a class-based model, or a free-function one? Also, we may want to hatch a path-based loading model similar to Java's classpath (the current model is explicit only); any thoughts about this?
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Sat Dec 24, 2005 12:52 pm    Post subject: Reply with quote

pragma wrote:
Most notably, this effort is going to drag DDL by the nose into completion. That having been said, I find that efforts like this are almost necessary since I'd be developing DDL-side components in a vaccuum - its nice to get some actual use-based requirements for stuff like that. The same can be said for parts of Mango too. Smile

Too true Smile

pragma wrote:
With respect to Mango's server components, the problem comes in that the Linker will need to be made thread-safe and will likely be passed down to the aggregate servers, providers, contexts and servlets so that those objects have that facility during runtime (take DSP for example). Combined with a config model that accepts the current XML config node, each component is simply passed what it needs to be set up.

Code:
interface SumacNode{
  void config(ddl.Linker linker, xml.dom.XMLNode node);
}

Yeah, I see what you mean. What you're suggesting seems reasonable though. I'll give it some thought and get back to you?

Regarding threads: can the binding be done one-shot, or would it be a series of requests? If the former, we could make that part synchronized? Failing that, we could impose a restricted set of linker functionality for the server(s) and synchronize at that level? I guess I'm asking if there's value in exposing a more limited abstraction of the linker (and, the config tree, for that matter) ...

pragma wrote:
Developing the contracts for loadable modules is key. Were you thinking of a class-based model, or a free-function one? Also, we may want to hatch a path-based loading model similar to Java's classpath (the current model is explicit only); any thoughts about this?

I suspect the factory-model might be appropriate here? Perhaps a known free-function that returns an interface instance? What do you think are the best options? Path-based loading sounds like a good idea (do you mean relative paths?), but might be a layer above DDL? Maybe? Maybe not?

What are the known technical limitations we'll run into?
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Dec 25, 2005 11:16 am    Post subject: Reply with quote

kris wrote:
Code:
interface SumacNode{
  void config(ddl.Linker linker, xml.dom.XMLNode node);
}

Yeah, I see what you mean. What you're suggesting seems reasonable though. I'll give it some thought and get back to you?[/quote]

Works for me. Smile

Quote:

Regarding threads: can the binding be done one-shot, or would it be a series of requests? If the former, we could make that part synchronized? Failing that, we could impose a restricted set of linker functionality for the server(s) and synchronize at that level? I guess I'm asking if there's value in exposing a more limited abstraction of the linker (and, the config tree, for that matter) ...


Well, we'd likely extend the base linker to get what we need. Again, the notion of maintianing lib paths dynamically comes to mind here. Aside from that, synchronization is pretty easy to accomplish via sub-classing thanks to the atomic nature of all of the operations on the Linker class:

http://ddl.dsource.org/module.ddl.Linker.html#class_Linker_

[quote]I suspect the factory-model might be appropriate here? Perhaps a known free-function that returns an interface instance? What do you think are the best options?[quote]

I think the most foolproof option would involve a factory free-function, declared as extern(C), so that the symbol name is easy to deduce. There are some sticky problems with class-name based loading right now, due to multiple namespaces existing in the same lib, and not knowing what the actual implemented class-name is.

Quote:
Path-based loading sounds like a good idea (do you mean relative paths?), but might be a layer above DDL? Maybe? Maybe not?


Any kind of path really. Heck, with Mango in scope they could be URL's for that matter.

Earlier drafts of DDL had a "Loader" class that sits on top of the linker. I think that model would easily apply here, where the loader provides the 'brain' for finding the needed binaries to pull into the linker.

If it proves to be generic enough, it could be folded into DDL. Otherwise, it could just live under sumac or wherever.

Quote:
What are the known technical limitations we'll run into?


Offhand, there are some memory consumption problems with DDL, due to overhead related to loading sizable .lib files. This can be mitigated with using multiple .objs instead.

With path-loading, what kind of policy do we want for binary storage? Should everything just get thrown into a given directory, or should the directories match the supported namespace for those binaries (much like the module statement and .d source files). The former has the loader scanning the directory for needed namespaces, and the latter has certain correctness constraints involved (much like with java).
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DSP All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group