FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Features

 
Post new topic   Reply to topic     Forum Index -> Sendero
View previous topic :: View next topic  
Author Message
rrichardson



Joined: 10 Sep 2007
Posts: 11

PostPosted: Mon Sep 10, 2007 9:49 am    Post subject: Features Reply with quote

I have been thinking about build such a framework myself, and I too have considerable experience with RoR. My goal for a Web framework in D would simply to accomplish what RoR (and others) cannot offer: massive scalability. The following features that I recommend reflect these values.

The obvious caveat here is that these are just my suggestions. I am more than willing to help impliment these if those involved feel that they would be beneficial.


*REST by default.*
It is somewhat of a hassle to build REST into rails apps because it wasn't designed for it. We have the ability to do just that. So that objects (and nested objects) are automatically referenced by a URL. One might be so bold as to make the controller layer exceptionally thin (ala Camping and other new frameworks) that take advantage of REST and intelligent models (this would be entirely a matter of taste, but developers should have this option)

*Prototype / Scriptaculous support, but not mandatory*
I really like these libraries, I think it should enable easy inclusion and use of the javascript libs, but if someone wants to use mochikit or whatever, they can.

*The web-framework is the server*
In the interest of massive scalability, we can have an asynchronous listener route jobs to a worker in a threadpool. From the looks of your docs so far, this is similar to what you have planned.

*Built-in load balancing*
The server can be easily configured to route requests to other servers in a session-sticky fashion. The amount of web-serving work would also would be a setting, so one could choose to have a front-end machine act primarily as a load balancer and a static-web page server, and any dynamic requests get delegated to one of the children in the cluster.

*Failover support*
works along side the above.

*REST Service/Resource consumption*
The data/model layer should be able to use services as if they were databases.

*Berkeley DB support*
This would be nice to see. BDB is extremely valueable in a web-service in a SOA architecture, because it allows a server to serve isolated data in a huge hurry. I have built a BDB wrapper for D using BCD (very basic at this point, but functional and fast)

It would be nice to see Sendero enable SOA in a way that few other frameworks can offer. A self contained, intelligent, small footprint server. which could even run as isolated services. i.e. on a single machine you could have 3 different web servers. One front-end with ajaxyness that does most of the view dirty work and inter-data relationships. A BDB based service that retrieves user preferences based upon a UID lookup (this would be the most frequently accessed data item so it's access is isolated and highly optimized). You could have a 3rd service that offers various utilities based on queries, it could use an RDBMS, or BDB or REST services, or all three.

I'm simply trying to start a dialog on what features are offered, planned, and see if my help would be appreciated in implementing these.
Back to top
View user's profile Send private message
aaronc542



Joined: 22 May 2007
Posts: 35
Location: NJ, USA

PostPosted: Mon Sep 10, 2007 10:57 am    Post subject: Reply with quote

Hi rrichardson,

Thank you for your suggestions and of course I would be interested your help on the project. My intention with the framework is to create software that helps myself and others get their work done - so whatever features will enable us do that are good.

Here are just a few thoughts in regards to your suggestions.

I have created a ThreadSafeQueue class which I think could serve as a building block for some of thread pooling features. It seems to work fine on Windows, but has a bug in Linux that causes a segfault. I haven't gotten around to inspecting this yet, so any help with that debugging and getting this class to be robust would be great. Then, of course, we could extend the pooling and load-balancing features....

How session-state is managed is actually something I've tried to consider a great deal. In a larger-scale scenario, we want to maintain session indepedent of which server is handling a given request. My current ideas include a mixture of binary serialization of encrypted state machine data to cookies or to a session-state database for more secure information. That way, the framework could at least have some support for statefulness (like Seaside) as well as load-balancing and rendundancy.

In terms of AJAX, I had been planning on using JQuery which seems a lot more lightweight than Prototype. Ext also seems to have a nice DOMQuery class. Ideally though I think it would be nice to have a uniform interface to AJAX features which would work with multiple backends.

Good support for REST and SOA would be great, so by all means...

Aaron
Back to top
View user's profile Send private message
rrichardson



Joined: 10 Sep 2007
Posts: 11

PostPosted: Mon Sep 10, 2007 11:12 am    Post subject: Reply with quote

Excellent. I use Linux pretty much exclusively so I'll get started on debugging that and getting a multithreaded worker pool and event loop going. (bcd has a wrapper for libevent which I'll probably use)

One thing I thought would also be cool: write maintenance scripts in dmdscript. http://www.digitalmars.com/dscript/index.html

Not sure if this is feasible.. but it seems like there will ultimately be some maintenance and deployment tasks, and since the dev will be using javascript anyways (most likely) we could keep the # of languages down to only 2. This is entirely a random thought and probably not something we'll have to deal with any time soon.

It would be nice to use a more interactive venue to discuss such things as architecture and designs for things such as REST. (irc maybe?)
Back to top
View user's profile Send private message
rrichardson



Joined: 10 Sep 2007
Posts: 11

PostPosted: Mon Sep 10, 2007 11:19 am    Post subject: Reply with quote

I forgot to mention. I have taken a look at JQuery, and I'm very impressed. If it benchmarks faster (which it seems like it would) I will definitely be sold.
Back to top
View user's profile Send private message
brad
Site Admin


Joined: 22 Feb 2004
Posts: 490
Location: Atlanta, GA USA

PostPosted: Mon Sep 10, 2007 11:32 am    Post subject: Re: Features Reply with quote

rrichardson wrote:
I have been thinking about build such a framework myself, and I too have considerable experience with RoR. My goal for a Web framework in D would simply to accomplish what RoR (and others) cannot offer: massive scalability. The following features that I recommend reflect these values.

The obvious caveat here is that these are just my suggestions. I am more than willing to help impliment these if those involved feel that they would be beneficial.


*REST by default.*
It is somewhat of a hassle to build REST into rails apps because it wasn't designed for it. We have the ability to do just that. So that objects (and nested objects) are automatically referenced by a URL. One might be so bold as to make the controller layer exceptionally thin (ala Camping and other new frameworks) that take advantage of REST and intelligent models (this would be entirely a matter of taste, but developers should have this option)

*Prototype / Scriptaculous support, but not mandatory*
I really like these libraries, I think it should enable easy inclusion and use of the javascript libs, but if someone wants to use mochikit or whatever, they can.

What do you have in mind for support here? Easy includes of individual modules of these libs? I would like to see server-side support for Dojo, but only the modules I need for that page. However, I haven't seen this work yet, at least not better than good ol' <script> stuffies.

rrichardson wrote:
*The web-framework is the server*
In the interest of massive scalability, we can have an asynchronous listener route jobs to a worker in a threadpool. From the looks of your docs so far, this is similar to what you have planned.

*Built-in load balancing*
The server can be easily configured to route requests to other servers in a session-sticky fashion. The amount of web-serving work would also would be a setting, so one could choose to have a front-end machine act primarily as a load balancer and a static-web page server, and any dynamic requests get delegated to one of the children in the cluster.

*Failover support*
works along side the above.

I think that in Tango/Mango, you'll find a lot of goodies on this front. Clustering is strong there, but might be a layer beyond the LB / Failover you mention. Still worth a look, though:

http://www.dsource.org/projects/tango/wiki/ChapterClustering

rrichardson wrote:
*REST Service/Resource consumption*
The data/model layer should be able to use services as if they were databases.

*Berkeley DB support*
This would be nice to see. BDB is extremely valueable in a web-service in a SOA architecture, because it allows a server to serve isolated data in a huge hurry. I have built a BDB wrapper for D using BCD (very basic at this point, but functional and fast)

It would be nice to see Sendero enable SOA in a way that few other frameworks can offer. A self contained, intelligent, small footprint server. which could even run as isolated services. i.e. on a single machine you could have 3 different web servers. One front-end with ajaxyness that does most of the view dirty work and inter-data relationships. A BDB based service that retrieves user preferences based upon a UID lookup (this would be the most frequently accessed data item so it's access is isolated and highly optimized). You could have a 3rd service that offers various utilities based on queries, it could use an RDBMS, or BDB or REST services, or all three.

I'm simply trying to start a dialog on what features are offered, planned, and see if my help would be appreciated in implementing these.


BA
_________________
I really like the vest!
Back to top
View user's profile Send private message
aaronc542



Joined: 22 May 2007
Posts: 35
Location: NJ, USA

PostPosted: Mon Sep 10, 2007 11:57 am    Post subject: Reply with quote

I think before we get heavily into AJAX support, we need to get a view design down.

Ultimately, I think the view design is the most sensitive aspect of the framework design. That is why I am intending on having a separate Data layer (which includes the ORM, XML, JSON, and serialization support). These features are more or less agnostic to view implementation and can be used across a wide range of applications that are web and non-web specific.

But, to implement AJAX support we need the view.

If you haven't seen it yet, check out this article:
http://www.cs.usfca.edu/~parrt/papers/mvc.templates.pdf

One view implentation I have works like this:

Code:

static class LoginView {
      mixin View!(LoginController);
      static void create(Div d, LoginController lc)
      {
         auto f = d.div("login").form(&lc.login, "login");
         auto sep = new Text(": ");
         f.text("Username");
         f ~= sep;
         f.textInput(&lc.username, &lc.username).br;
         f.text("Password");
         f ~= sep;
         f.passwordInput(&lc.password).br;
         f.submit("Submit");
      }
   }


It basically using HTML generators to generate the form and takes pointers to LoginController data members or getters/setters as binding parameters for data. The view is then compiled the first time it is used and the view ensures that only members of LoginController (or its children) are bound (it does this by mapping the pointers in LoginController). This ensures that there is a good level of model/view separation, but it is also somewhat complicated. I'm not really sure what I think... There are MANY other alternatives and its a little hard to find the best way...

Anway, I just wanted to bring up the whole idea of View design, because I think its a crucial issue in the framework design and any AJAX support is dependent on it.

Also btw, a nice thing about the view design above is that the Text class automatically takes all strings provided and compiles them into a Gettext like format so that localization support will be builtin. It also has builtin hooks to ICU so that integers are properly converted.

Here's example syntax:

Code:
auto t = new Text("There are {0} {1} here.").bind(&x.number).bind(&x.thing);


x.number will be automatically be outputted into the local number format by ICU.
Back to top
View user's profile Send private message
rrichardson



Joined: 10 Sep 2007
Posts: 11

PostPosted: Mon Sep 10, 2007 12:44 pm    Post subject: Reply with quote

The obvious addition to your example would be a specialized template system. A simple starting point might be:

http://code.google.com/p/google-ctemplate/

It would be rather easy to port to D (or completely reimplement) and supports simply inserting values from a dictionary into HTML. Any looping, helpers, or block cleverness would be in the View class in D
Back to top
View user's profile Send private message
brad
Site Admin


Joined: 22 Feb 2004
Posts: 490
Location: Atlanta, GA USA

PostPosted: Mon Sep 10, 2007 12:52 pm    Post subject: Reply with quote

aaronc542 wrote:

If you haven't seen it yet, check out this article:
http://www.cs.usfca.edu/~parrt/papers/mvc.templates.pdf


I read this paper, and really like it. However, I'm being swayed closer to the middle. Let me explain...

StringTemplate (in the paper) is one extreme of putting (.dup'ing?) all data into the Context and having a limited functionality set in the template engine, and JSP soup is the other extreme. Something in the middle is likely to give as much protection from shooting yourself in the foot, but still be efficient in getting data to the presentation layer w/o having call side-effects. When I get time, I'll be looking at Smarty and Tea, and possibly even Genshi, to see if they would make sense in any or all of the upcoming D web frameworks.

My $0.02
_________________
I really like the vest!
Back to top
View user's profile Send private message
rrichardson



Joined: 10 Sep 2007
Posts: 11

PostPosted: Fri Sep 14, 2007 9:42 am    Post subject: Reply with quote

To reinforce what I've mentioned above..

This little article about scaling Twitter is indicative of scalability problems a website owner will face.
http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster

There are already a bunch of frameworks out there (at least 1 per modern language?) and I'm a big fan of not building things unless they have something unique and beneficial to offer.

I think a diferentiating factor of Sendero can be that it offers all of these horizontal scaling options by default, and can be enabled by changing two words in a simple config file.

Now we need to pay close attention to ergonomics. Developers will certainly spend most of their time working on the view, so it should be as efficient as possible.

reiterated in a related blog:
http://smoothspan.wordpress.com/2007/09/14/twitter-scaling-story-mirrors-the-multicore-language-timetable/

My goal for the server end of Sendero would be to ensure that they'll never have to worry about a website's performance. As long as they have the machines, it just works, even if they're working on a 16 machine, load balanced cluster with intelligent caching and different DB's.
Back to top
View user's profile Send private message
aaronc542



Joined: 22 May 2007
Posts: 35
Location: NJ, USA

PostPosted: Fri Sep 14, 2007 10:15 am    Post subject: Reply with quote

Exactly what we want. I believe that scalability is one of the primary goals I've had in mind behind this framework and one of the reasons I chose D as opposed to another platform. Easier to program than C++, but inherently much more scalable than Ruby for instance...

rrichardson: Feel free to post any of the code you have been working on into the /sandbox/rrichardson directory. That way we can start collaborating. (I will be out of town next week btw).

Anybody care to chime in with their preference/ideas on view design? (In the long run maybe we'll have a few templating systems..., but if we can do 1-2 well... all the better.)

I was impressed by the Genshi design.

We could do something like this:

Code:

<span d:content="myText">This is replaced.</span>


as in Genshi, and then even use sendero's reflection to do variable bindings in templates, like this:

Code:

struct SomeData
{
char[] myText;
}

char[] applyTemplate()
{
SomeData d;
d.myText = "hello world";
auto tmpl = Template.get("myTemplate");
tmpl.use(d);
return tmpl.render;
}


This would then generate:
Code:

<span>hello world</span>


for the above template.

Thoughts?
Back to top
View user's profile Send private message
rrichardson



Joined: 10 Sep 2007
Posts: 11

PostPosted: Fri Sep 14, 2007 11:04 am    Post subject: Reply with quote

as long as that process is cacheable i'm happy Wink

I'm rovar when I'm trolling in the #d.* channels.

I'll make a post some time this week with proposed features.
I'm on vacation next week so I hope to add to subversion the clusterable web server that features libevent as well as my first (and probably awful) shot at ROUTING as well as converting (get|put|post|delete)) params into value hashes(objects?) for consumption.
Back to top
View user's profile Send private message
aaronc542



Joined: 22 May 2007
Posts: 35
Location: NJ, USA

PostPosted: Fri Sep 14, 2007 12:40 pm    Post subject: Reply with quote

I have done a fair amount of work on a potential routing framework and personally, I think it's fairly involved... I don't really think the way Rails does it by default is correct, because your object should have an action (i.e. object/action as opposed to :controller/:action/:id in rails)... I was thinking of something more like controller/object/action.

I also have basic code for parsing parameters and cookies into hashes and I will put that on SVN when I get a chance.

What were your ideas on routing?
Back to top
View user's profile Send private message
rrichardson



Joined: 10 Sep 2007
Posts: 11

PostPosted: Mon Sep 17, 2007 5:59 pm    Post subject: Reply with quote

(sorry for the disorganization of this post, I would like to formalize some of these ideas when I get a chance)

I like the idea of REST by default. This is, in a sense, very close to what you suggested, it depends on what your concept of a controller would be.

A controller is an artificial concept that is kind of being jammed in to existence as a loose collection of methods. What I envision is a url with OBJECT/METHOD/ID. Effectively dismissing the reference to the controller entirely. I find the presence of controller logic useful, but if you think about it, its existence really confuses things.

The case for controllers in the URL is as follows:

Suppose I have a social-network website that would, in a RESTful fashion, offer an interface of User/Profile/ID to look at a picture of a person. But in what context? I could have a social network that offers both friend-network exploration, and dating, so Dating/User/Profile/ID would use the same object, but offer different results from Friend/UserProfile/ID .

This treats the controller as some sort of "context" through which you can view specially prepared views of objects' data. But I argue that that context can simply be an object that stores collections of other objects. With our default support for Nested Resources we can offer controller type access and so much more.

For instance:
I have an auto dealership website, it features two objects that may contain vehicles, a dealer_lot and a marketplace. The instances of each type can contain collections of the same vehicle objects, but how they render pertinent information is up to them. i.e. DealerLot/Vehicle/Photo/view will produce different results than MarketPlace/Vehicle/Photo/view. One will show dealers what they want to see, the other will show to customers what they (the dealers) want them to see.

Having a controller that is incongruent from the model is, in most cases, a level of indirection that just adds confusion, in 90% of cases, a proper object model with each object having at minimum the restful view/new/del/edit is enough, and offers a default configuration that allows us to do a tremendous amount for the user.

"But where does the controller go?" you ask. It is in the same place, but given that important logic (and abstractions) happen in the URL routing and the ORM, the controller really isn't that important.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Sendero All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group