View previous topic :: View next topic |
Author |
Message |
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sun Mar 27, 2005 11:13 pm Post subject: Status Log |
|
|
Thought I'd use a sticky to keep status posts separate from the others.
I'm going through one of many rounds of refactoring on the code. Almost all of the old design is back up to working status from Mango Beta 7. Also, I've had to make some very minor changes in light of the current state of DMD.
It looks like the /misc branch is almost ready for release on its own, however it's still without documentation. I'm happy to say that all the library functionality works well, including the reloadable object proxies. However, that may need another round of refactoring as well, as there are some lingering issues to be resolved.
Sumac is taking shape as a configurable, multi-headed, mango-server framework. However, I think it may languish until I can get Apache support working first, as bringing DSP to existing server platforms is a must. That activity will also force DSP to come to creating bridges between Mango and existing server technologies; something that will have a deep impact for Mango devs.
The DSPCore server, for sumac, is coming along just dandy. It still needs some heavy refactoring to separate the DSP compilation and servlet management bits from the context and provider objects. It means cleaner code and easier to maintain parts ahead.
I'm also toying with the notion of using build as the servlet builder portion of the engine. The idea of a tool that's smart enough to find pragma() statements and follow dependencies, is enough to expand DSP's capabilities and security model.
The latest is presently in the SNV repos.
Building is also possible via Derek's Build tool. "dspcore.d" and "server.d" are the only two build targets at this time. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Mon Apr 04, 2005 3:49 pm Post subject: XML and disaster |
|
|
D has a complete lack of a good, native XML parser. The closest we have is expat. Also, Mango's XML parser is in limbo at the moment.
I genuinely want to improve this situation. I've tasked myself with constructing a non-validating XML parser in D, that takes advantage of slicing to boost performance. Also, DSP will benefit directly from this, as the current parser design is a maintenance nightmare.
I drafted some things earlier last week, working with the annotated XML spec on my monitor the entire time. Saturday, I coded my brains out and spent an amazing 12 hours at the keyboard putting my ideas into action. So far, it lexes, parses and provides XML bits to a custom-coded "XML Consumer". So far so good.
(BTW, the conformance suite from the W3C has proven *invaluable* in uint-testing this project)
Overall, the core design is loosely based on expat. The parser will expect a helper (IXMLConsumer) that contains all the callbacks for parsing events, including DTD handling. The consumer can be extended to create a parse tree, DOM or simply validate the parsed XML file; these will become standard building-blocks for the library over the long term.
Unlike expat, the parser also takes on a second helper: IXMLProvider. I plan on using the provider to separately resolve external entities in a manner separate from the IXMLConsumer. This way, consumer and provider tasks remain distinct, which will make for a more useful library.
Also, the consumer interface is unlike expat in that the method signatures are assumed to be against a mostly stateless implementation; something that expat doesn't let you do without compromising something.
The short term will see the Lexer, Parser, Consumer Adapter and Provider Adapter: your basic, non-validating XML kit. This will be followed shortly by a much-revised DSP.
The near term will add on a validating/testbed consumer. Adding a stock provider (possibly mango based) will also roll into this iteration. The W3C's namespace reccomendation may also be included here. The overall goal of this milestone is compliance, which is where the W3C XML test suite comes in real handy.
The long term may eventually see an XPath/XPointer implementation coupled with a DOM implementation. This will require an additional lexer and parser for the job. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Thu Apr 21, 2005 9:41 am Post subject: XML and recovery |
|
|
I'm drawing a line under the current DSP implementation. The old parser was an ugly, hard to maintain beast and will be hereafter called "Alpha2". Expect it to dissapear from the repos shortly. I will however, continue to use a variation of the old mango servlet classes, as they still have much utility. However, I think I'll have to refactor them significantly first; they're still too difficult to wield in their present configuration.
/trunk/dsp/ now contains the new, XML-parser-based design. "build dsptest" to check out the new code generator. While not much, the new design is highly optimizable blend of stateful and stateless generator parts.
So far, not much of the DSP vocabulary is supported, but this will come in time. As it is, I've had to revise parts of the DSP specification for XML conformance. I'm going to put aside the rest of the spec until I can get the servlet server working correctly. So for now, in/out/body/code blocks are all we have.
Meanwhile, the XML parser has grown by leaps and bounds. The conformance suite has been invaluable in pushing my work to the limits, as every new test case uncovers something else that I overlooked in the specification. I'm toying with the idea of adding SAX and expat-style consumer to the stack, as a test-case to ensure that the core parser is providing enough data for the future DOM, XPATH and XSL modules.
There is a XML namespace module out there for use as well: trunk/xml/xmlns. The current implementation is very new, and is likely full of bugs. It stands as more of an example of how I plan to use this library design to implment the rest of the XML capability stack.
DSP may get two parsing modes to work with, as I'm already seeing limitations in using a strict XML parser. An additional "relaxed" mode may come along once I add a relaxed XML parser to the XML lib. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sat Apr 30, 2005 1:34 pm Post subject: Updates |
|
|
DSP
The new design is moving along swimmingly. I have stuck to a more Mango-style implementation, by minimally extending classes as I go. The result is much more streamlined, if somewhat spare on the features.
The current DSP preprocessor/compiler only understands the code-style tags for now, and there are some strict limitations regarding the XML-nature of the language. For now the <?dsp ...?> style code tags can only be in the preamble of the document, anywhere else and the XML parser complains. Also, there is only one root element allowed, which forces the developer to write fully XML-style documents; there's no room for fragments, plain-text or HTML. I will eventually add a "relaxed" XML parser to the kit to ease these kinds of problems. DSP will then be able to run in one of two modes: "strict" or "relaxed".
Compiling "servertest" will run a single-thread http server much like the Mango example server. It will eventually serve up the files in the 'dspconf' directory, but for now it kicks back a test message for dsp files. Watch this space for more word on the progress of the server.
XML
The XML lib has made some small steps toward being a more useful library. The entire set of DOM level 3 core interfaces has been translated from IDL into D:
http://svn.dsource.org/svn/projects/dsp/trunk/xml/dom
The files contain just 'interface' definitions, which will be backed by a standard implementation. Developers wishing to implement DOM in D are *strongly* encouraged to use these interfaces and the DOMImplementationRegistry provided by 'Bootstrap.d'.
Also, contributions for other DOM reccomendations and standards are welcome.
Dexter
Dexter, the D-to-XML translator.
Many hours of hacking Ben Hinkle's repackaged GDC front-end (DMDFE) has yielded a fully-capable processor that generates XML files from d sources. The added benefit is that it will capture comments in the source file and attach them to the appropriate entities in the output.
The output xml is rich with everything that the typical D compiler could possibly know about the code processed: namespaces, mangled names, template types, function parameters... everything.
I wrote this tool with documentation generators in mind. Just process the xml files as needed (to build a cross-reference for instance) and run through an XSLT transform and presto: instant doc pages!
The code right now only builds for windows, as the win32.mak is the only modified makefile present. The codebase also does not have the latest GDC patches applied, so it may not be compatible with the most recent version of D. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Wed May 04, 2005 8:45 am Post subject: Re: Updates |
|
|
DSP
The servlet enigne gets a small tweak toward dynamic compilation. Simply build 'servertest', modify the path information in the file and in dspconf/config.xml, and try to access http://localhost/foo/bar/dsp01.dsp. The engine now compiles a servlet lib dll and runs it properly.
There is still a small issue with the runtime size of the dlls. I think some of this can be side-stepped by compiling in release mode and using release mode libs for linking. Also, the combination of mango and phobos together is bloating things some.
I've had some small issues using 'build' to compile servlets. Were this to work properly, I could forego .lib files altoegher and just rely on discrete .obj files for smaller libs. Then again, that would really bog down any given development cycle (longer compile times), so perhaps fat binaries are the best compromise for now.
Anyway, I plan on using mango's QueuedCache to solve the issue of having too many large dll's in memory at the same time. At least that way, the cache capacity is tunable, so a given administrator can make a DSP server memory or CPU heavy depending on the operating environment.
I've also put whole-application-dll's on the back-burner due to their complexity; I may just strike it from the platform altogether. There's a negligable benefit of this over having a everything in memory at once, and the mechanics for tracing dependencies (and their related file dates) is just too much overhead for medium to large applications. Just using the simple approach with a huge dll cache gets the job done nicely, even if it does repeat a lot of library overhead.
XML
Added some more classes for the upcomming DOM implementation.
The way forward is very well laid out in the W3C documentation, but there are some nasty 'holes' in the spec. The most irksome is that descendents of the Node interface can override the behavior of Node methods, yet the overrides are documented under Node (rather than on the overriding interface). I think we can attribute this to how the IDL for the spec is laid out, which has no convention for 'override'. Anyway, this causes a lot of flipping back-and-forth through the spec when implementing a Document class or Element class as you have to make sure that it's typename, name and value are being reported correctly.
I'm going to continue to focus on stubbing out the std DOM implementation so at least it compiles and implements the actual DOM interfaces. I've learned that the majority of these interfaces are interdependent, so this is pretty much the only approach available: it's all or nothing. Once the stubbed-out code compiles, I'll go back over and start implementing the Level 1 functionality, then 2 and finally 3.
Meanwhile, I'm keeping an eye on what data the DOM *really* needs, so when I write a DOMParser, the existing IXMLConsumer interface has all the hooks needed to make this work. The DTDParser will also be completed, so that the DOMParser can do its job correctly. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Thu May 05, 2005 10:15 pm Post subject: Re: Updates |
|
|
DSP
(trunk/ and trunk/dsp/ updated)
The Servlet Engine is practically working as designed. Servlet Libraries are now being loaded on-demand and are replaced if their dependency set (just the root .dsp file for now) is more recent than the .dll it represents. The engine also looks in the configured cache folder to see if the needed dll has been created already: this makes restarting the server cost a little less time since there's less to do in order to regenerate the in-memory cache.
There is a small (unconfirmed) issue regarding the use of Mango's QueuedCache, which put to work holding all the dll servlet references. When configured to a capacity of '2', inserting a 3rd new entity into the cache overwrites the oldest (first) entry. What is wrong is that the key for that entry doesn't go away and instead is now redirected to the third element.
Launching servertest and requesting dsp01.dsp, dsp02.dsp, dsp03.dsp and finally dsp01.dsp again, shows the correct servlets until you get back to the first one: it now shows the content for the third. My debug output confirms that the cache gets confused under these circumstances.
I'll be writing a bug report as soon as I cobble together a good test case and diagnosis of the problem.
On another note: the engine is now 100? XML configurable, with some minor enhancements due soon.
The ServletProvider now parses a 'server.xml' which contains configuration information for the server at hand. The ServletContext also parses a 'config.xml' that is located in the application root for that given context. The ServletContext gets all the settings from the ServletProvider, and are overwritten to allow for administrators to override settings for applications, on an application-wide basis; this may change.
XML
(no updates in CVS this time)
Did some work studying the XPath 1.0 and 2.0 specifications, and drafted out some code to handle functions. I was correct about the spec being catered to DOM. I was able to write a good portion of an XPath implementation w/o needing to use interfaces that weren't already provided in that spec. Everything else, is XPath specific.
The upshot of this approach is that custom DOM implementations can be used against the one-and-only XPath implementation. The downside is that you loose any advantage in speed you would get from coding against a custom class set. Personally, I think this is a win, since the need for versatility outweighs the perceived performance losses. Also, I've learned that the XPath functions and axis specifiers are really bound by the node set you give them. So document size and badly coded queries are the real performance killers here.
DOM is looking more and more like the sleeping giant that I feared it would be. As I go back over the specification with a fine-toothed comb, I find all these behaviors nestled safely in between the function call specs. Most of the Level1 stuff is straight-forward, but Level2 and Level3 have some really heavy routines and requirements. I may have to build and test in layers since the interfaces are so inter-related.
I've also taken to adding opApply() to the 'collection' style DOM interfaces: NodeList and NamedNodeList to name a few. It doesn't present that much more trouble for implementers and it makes coding much, much easier:
Code: | //before:
for(int i=0; i<node.getChildren.length; i++){
Node n = node.getChildren.item(i); // do something with this node
}
// after
foreach(Node n; node.getChildren){
// do something with n
}
|
I may also consider opIndex() while I'm at it. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sun May 15, 2005 6:29 pm Post subject: Re: Status Log |
|
|
DSP
Today I finally reached the first Beta milestone.
Announcement for Beta 1
It took an extremely long time to get here, and I owe most of that to learning how to use D, Mango, the complete XML specification the idosynchracies of dll's and those of web-server programming. The present mark is not just one of technical achivement, its actually one of personal growth in a multitude of areas.
The hardest part? Getting the servlet library cache to behave correctly. This looked easy enough on paper, but was actually quite hard to implement largely due to the time involved in troubleshooting it. (This will get a unittest to help cover this gap in the future).
Why? Servlets can exist as a dll, dll and source, or just source. Depending on the existance and/or file dates involved, the servlet can take on the states non-existant, pre-compilation, compiled, or out-of-date. Tracking all this has lead to some rather odd-looking code.
To make matters worse, one can only recompile a servlet if it's been completely unloaded from the runtime. So failures in the cache to relenquish a dll would block compilation testing. The opposite is also true: issues with the compilation unit would block proper testing of the cache.
For a while, it was a huge mess. To a large extent, it still is. At least now it works consistently.
XML
The XML parser recieved an overhaul due to the horrible mismatch between DOM and DSP in terms of character types required. Since DSP was written with utf8 in mind, and DOM requires utf16, something had to be done.
The baseline parser is now a templated class that expects a character type as a template parameter. The XMLAttributes collection and IXMLConsumer interfaces also follow this convention. For convienence, the utf8all, utf16all and utf32all files are provided for easy consumption of the library. Predefined aliases also exist for each flavor of these classes, so they may co-exist within the same compilation unit.
Furthermore, the behavior in the parser was too strict for DSP, and had too much overhead in the lexer to be considered efficent. I condensed the parser down to a single class that lazily uses a set of lexer rules/functions to gather what it needs from the input stream. The design suddenly fell in on itself, destroying the entire Lexer, Token and LexerStream classes completely.
At the same time, the consumer interface was fleshed out to handle basic DTD operations as well as enough metadata to support DOM as well as basic XML parsing tasks. Comparing the ConsumerAdapter class to the ConfigParser in DSP gives a very good illustration of how much range this library now has.
I did some work last week digesting the XSLT specification. The spec is a confusing mess composed mostly of the interactions between XSLT specific pseudotypes and XML Schema ones.
XPath has leaped forward some since I last updated it. I've jumped in at the level 2.0 spec, only because it had a complete BNF which was ideal for implementation. Also, 2.0 has all the side-effects and error states well defined, which is a far cry from what 1.0 gives you (read: practically nothing). The downside? There are a *lot* of functions in the spec, and I'm at a loss as to how to implement this most efficently. I think I'll just start with the basic expression tree parser and work my way down from there.
DOM is presently stagnant, but at least it doesn't make the compiler complain too much. The same goes for XMLNS, which hasn't been updated since the last XML parser update. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sun May 22, 2005 12:23 pm Post subject: Re: Status Log |
|
|
DSP
Spent the weekend working on documentation. The docs from the alpha were very stale and needed some improvement. I decided that html would be fine for most folks, so I whipped up a product website *and* a language reference web for all to see.
http://svn.dsource.org/projects/dsp/trunk/doc/index.html
http://svn.dsource.org/projects/dsp/trunk/doc/dsp-reference/index.html
Feedback is welcome; just PM me if you have any comments or suggestions.
I didn't just rehash the old .doc file I had lying around: I actually rewrote portions of the old specification to make for a more streamlined product. The end-result is that DSP will have fewer tags than before, which is a good thing since they remain mostly orthogonal to D itself.
I think that keeping DSP orthogonal to D is very important since it helps avoid the kind of tarpit that ColdFusion has become (ie. it's almost two langauges back-to-back). I've also tried to keep the tag attributes distinct in purpose so things are easier to code, which should keep the amount of checking the documentation to a minimum.
I'm holding off on the Server API documentation until that finally congeals, and I look deeper into solving some of the parsing issues. Also, I need to refine Dexter to make the job more manageable.
Dexter
Kicked around some stylesheets to see what I could do with the XML output. The results were encouraging, as I can now generate crude HTML documentation from single XML files; an aggregator will still be needed to compose more useful docs.
Much has to be done with how Dexter is emitting type names, function handles and call specs; virtually anything where the semantic pass obscures what was used in the original code. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Tue May 24, 2005 8:49 pm Post subject: Re: Status Log |
|
|
DSP
DSP now compiles with Mango 1.3 on DMD 1.25 using Build 2.07. Also the following bulletpoints on my todo were completed:
5/23/05
- Fixed minor documentation issues
- Completed refactoring of the basic XML library
- Added dsp:capture to documentation
- Added dsp:page tag
- Stubbed-out dsp:servlet tag
- Stubbed-out dsp:include tag
5/25/05
- upgrade to DMD 0.125, latest build and latest mango
- refactor using a ServletCompilationContext instead of so many parameters for servlet compilation
- DSPResponse needs the output stack feature
- DSPRequest needs the attributes and tag-forwarding features (use 'box' in DMD 0.125)
- Facility for latching custom tag references (servlets et al) at service time.
The biggest changes were made to DSPResponse which now holds the bulk of the servlet runtime in terms of hooks that allow the various DSP tags to work. A nice side-effect of this is that developers coding "below" dsp will be able to take advantage of the output stack and other goodies provided by that interface.
As I go forward, I plan on coding to my specification for the various dsp tags. The runtime, and server will be continuously refactoried as I go to support the new functionality. Also, I plan on writing a full conformance suite to make sure that the tags behave as they should.
XML
XML got a kick in the pants, as I was having issues meshing the XMLNS support with the original parser, by extending those classes. I had to move XMLAttributes out to struct so that XMLNSAttributes wouldn't conflict. The result is just as robust, if not moreso, than the original. In fact, the changes needed to DSP after the fact were minimal (change 'in' operator to 'contains()').
When time permits, I'll be tightening the implementation in the baseline parser up more and more as I push forward through XMLNS and DOM.
No word on when I'll implement validation just yet. The specifications have so many points to cover that it's an effort almost unto itself. For now, I'm desigining with validation in mind so others can validate what they must. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Wed May 25, 2005 10:14 pm Post subject: Re: Status Log |
|
|
DSP
The bleeding edge now contains dsp:include. Also the conformance suite has been touched up a little.
5/26/05
- started the beginnings of the dsp conformance suite
- completed dsp:include tag
- noted additional bugs, including potential multithreaded resource issue with the library cache
- refined additions to DSPServlet and DSPResponse that will be needed to support additional dsp tags.
- noted possibility of extending dsp:include and dsp:tag to incorporate http, https and ftp as targets. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Fri May 27, 2005 8:40 am Post subject: Re: Status Log |
|
|
No code changes in this update.
I finally acquired my new development system at home. I went from a PII/400Mhz to a Athlon 3200/64bit system. Although my copy of win2k isn't taking full advantage of the system's depth and breadth, the raw 5-fold increase in speed is *very* nice. I haven't seen the CPU utilization go over 20? yet (whereas the old system ran at *least* at that level of useage).
Sad part is: i'm still running off of my old graphics card. Seems that on paper, I mistook the systems' integrated perpherial 'NVidia nForce' chipset for an on-mainboard VGA 'NVidia gForce' chipset. When I got the thing out of the box and found no VGA port on the back, I knew I was in trouble.
Also my gimpy 7-year old DVD rom drive looks terrible in the new case, and is starting to show some signs of age and wear from the years gone by. It is a little slow, but it boots CD's just fine and reads DVD media well so I can't complain.
I'm considering dual-booting the thing into Debian/MEPIS so I can test out some linux builds with DSP and other things. Since it's a 64bit system, I can also play around with 64 bit environments later down the road; possibly investigating if D is ready for the next generation of software.
So it looks like my development schedule will be delayed somewhat as I shake this box down and work on getting a LAN going with the old PII system. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sat May 28, 2005 10:03 pm Post subject: Re: Status Log |
|
|
DSP
Mostly refactoring, which resulted in a streamlined code generator. While it hasn't been pushed to its limits yet, it does some pretty clever things with generating output. It's *much* closer to handwritten code than before.
5/28/05
- further enhanced the dsp:include tag
- need ability to insert arbitrary text into the XML stream at the current read position
- added XML events for start/end document
- Runtime needs a full type-conversion suite (supplied by box and std.string)
- Refactored code generator to concatenate string expressions for fewer calls to .put().
- Refactored code generator to use runtime 'Registers' for complex evaluation uses.
XML
Not much has changed here except for the inclusion of xmlStartDocument() and xmlEndDocument() to the XMLIConsumer interface. This was needed as DSP wouldn't terminate statements correctly at the end of the document. The additional events make sure that the consumer is well-behaved. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Wed Jun 01, 2005 10:15 pm Post subject: Re: Status Log |
|
|
[quote="pragma"]DSP
At this point, the architecture for DSP is beginning to congeal. As a result, implementing the standard tag set is only causing little changes here-and-there to the overall design. The more tags the runtime is forced to support, the stronger the engine will become.
I've resolved to try and get the entire tag set up and running in some form or another, then I'll go back and bugtest and refine things. I want to proceed in this fashion as I'm not 100? sure that the current tag design is useful as it currently stands.
Also, the docs will need some addtional annotations for the attributes, as many tags will have specific requirements for this. Generally speaking, some attributes must be constants, while some are allowed to be expressions. Also, some are subject to addtional validation, such as being valid D identifiers, namespaces and so-forth.
I won't leave source-line numbering support for last, but I have to wait as I do not have the best solution for this problem prepared just yet. The problem is that the dsp:include tag allows for direct-injection of dsp code, which is accomplished via the XML parser. Injecting code in this way basicaly makes the origional file look bigger than it really is, which would skew any straight forward approach for embedding #line pragmas; the parser will return the wrong line number and or file at some point.
Lastly, I added a rather sad attempt at line-number support for DSP parsing errors. Apparently the XML parser isn't tracking carriage-returns as accurately as I would like.
Overall, DSP is taking shape quite nicely; if I do say so myself.
6/01/05
- added reset() override to DSPResponse and DSPRequest (eliminated buggy behavior)
- refactored the expression parser to use less code (dumped split() function)
- refactored workspace slightly to use /lib instead of /runtime compiler now looks to dsp root for the 'runtime'
- fixed bugs with pushWriter and popWriter
- added dsp:import and dspconf page
- added dsp:capture and dspconf page
- added dsp:attribute and dspconf page _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Thu Jun 02, 2005 8:48 pm Post subject: Re: Status Log |
|
|
DSP
This update is a mixed bag. There are a host of small bugs that manifest when using any of the new tags. Consider this update to be dirty, broken, and not very usable. You have been warned.
Here's a look at my current buglist:
- Issues with paths on custom tags.. requires "./" to get local directory
- Lib cache 'ballooning' feature doesn't behave correctly. Causes aggregate .dlls to be permanently loaded.
- <?dsp:output?> and <dsp:output> both require a closing ';' to function
- attribute forwarding is broken (std.boxer won't convert char[] to char[]?!?)
Generally speaking, things work like they're supposed to. Custom tag aggregation works, but attribute forwarding fails due to bugs with std.boxer. Thanks to that, I have no good way to test out dsp:argument or dsp:default.
The cache 'ballooning' is a feature that has been hard to implement correctly. The problem is that there's an edge case where your cache may be full and 100? in use (locked), but you still need one or more libraries to complete a particular request (custom tags mapped to .dsp files). Rather than block until the other requests have unlocked their libs, the cache will 'balloon' to take on the additional libraries needed.
Since the read/write locking on the cache isn't implemented correctly anyway, I might as well fix this and make it fully threadsafe while I'm at it.
Some subtle DSP code-generation bugs were fixed in the existing tags. RuntimeTag is now a class, as the struct version clearly needed reference semantics to work correctly.
I'm looking into refactoring the tag support runtime to push more of it inside DSPResponse. This is a good move since it will help reduce the .dll sizes a tad, plus it will make code generation much easier. This may include the fromString and toString conversion routines.
I think I'll also move the more common codegen idioms into DSPGrammar, again, to further simplify things. Also, the codegen portions need some validation routines to enforce the types of the various parameters fed to DSP tags... those restrictions also needs to be added to the documentation. I would love to make every attribute support expressions ("#blah#") but its the price we pay for DSP and D not being interpreted languages.
6/02/05
- added dsp:output
- added dsp:tag
- added dsp:argument
- added dsp:default
- added custom tag support _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Fri Jun 03, 2005 12:40 pm Post subject: Re: Status Log |
|
|
DSP - Tech Note
After spending a good 15 minutes describing the DSP library cache mechanism to a co-worker, I realized that I should probably write a bit about it here.
The library cache is almost a misnomer, as it does a heck of a lot more than just buffer references to .dll files. The actual behavioral requirements look something like this:
- relibably buffer dll references
- reliably flag dll as 'idle' or 'in use'
- reliably unload dlls on demand, when needed*.
- read/write locking the entire cache (writer preference) for thread safety
- prioritize dll references according to popularity, so a full cache drops the least popular idle entry in exchange for adding a new cache entry.
- 'balloon' or expand the cache capacity in case all libraries are in use when attempting to add a new entry to a full cache
(* - easily the most important one of all since you cannot overwrite a dll that is loaded... one failure here and codegen outright fails to build a new dll in response to script changes. Once we loose track of a dll reference, restarting the server is the only way out.)
Out of all the pieces of the project, this one has been the hardest to get right. Implement it using stock classes, and you get a seven-layer-cake that is tough to debug and manage. Try to do it as an all-in-one class and you get a morass of methods 1200 lines long that is rediculously hard to manage. I have yet to get the balance just right.
So I'm looking into rebuilding it, this time for good.
The present implementation is a rendition of a similar Mango Cache class, that uses a Mango Hashtable for the read/write locking. This will have to be exchanged for something custom, since the code generator needs to execute *inside* the write lock due to the dll load/unload semantics.
Again like Mango's cache, the individual cache items also belong to a doubly-linked list which represents the popularity index. This feature may stay, as I have yet to find a better way to implement a relative popularity scheme.
Seeing as how the current implementation is a large source of bugs, I think this will be next on my list of things to fix. Wait and see. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|