FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Continuing the D.announce discussion

 
Post new topic   Reply to topic     Forum Index -> MiniD
View previous topic :: View next topic  
Author Message
bobef



Joined: 05 Jun 2005
Posts: 269

PostPosted: Wed Sep 26, 2007 2:42 am    Post subject: Continuing the D.announce discussion Reply with quote

As I posted on D.announce I tested MiniD again ( http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D.announce&article_id=10003 ). It works very fast with strings but when I add writefln(a,b) it goes slow. It goes from 7ms to 60ms with output redirected to file trough the console (">file"). With my script when I add println for the the variables in the loop it goes from 17ms to 20ms. So whats the difference? I use tango.io.Console.Cout for println.
Back to top
View user's profile Send private message
JarrettBillingsley



Joined: 20 Jun 2006
Posts: 457
Location: Pennsylvania!

PostPosted: Wed Sep 26, 2007 7:44 am    Post subject: Reply with quote

Hmm. Your example runs in about 5.6ms for me, and then changing it to use write() takes about 55ms when redirected to a file. This is indeed disturbing. I was using Stdout in the implementation of write(), but I changed it to Cout with no change in performance. I wonder what causes it..

However, redirecting to a file from the console seems to be rather slow in general. If I do it programmatically, by reassigning Stdout in the host program before running the code, it runs in about 20ms. Still not that fast, but better. Even better, and even closer to a web server situation, if I redirect Stdout to a GrowBuffer instead, it takes about 12ms.

Lastly, modifying your code a bit to use a StringBuffer and appending those strings to it, I can get the runtime down to about 8ms. Of course this isn't timing quite the same thing, but..
Back to top
View user's profile Send private message
bobef



Joined: 05 Jun 2005
Posts: 269

PostPosted: Wed Sep 26, 2007 11:09 am    Post subject: Reply with quote

I would like to do some more tests but this time in a real web app. To do so I will have to do some more development. And to do it properly I would like to have the language itself to know about <? ?> (i.e. embedded code in HTML). Would you advice me what modifications need to be done to make MiniD replace everything that is not inside <? ?> with, for example, write()?
Back to top
View user's profile Send private message
JarrettBillingsley



Joined: 20 Jun 2006
Posts: 457
Location: Pennsylvania!

PostPosted: Wed Sep 26, 2007 4:34 pm    Post subject: Reply with quote

There are two ways this can be done: the hard way, and the easy (but possibly ever-so-slightly-less-efficient-but-not-really) way.

The hard way would be to modify the lexer of the compiler to have two states: HTML mode and MiniD mode. It'd start in HTML mode, and would basically blindly skip stuff until it came across a <? tag, at which point it'd take all that text it skipped over, and turn it into a big string literal. Then you could inject a few tokens (a 'write' identifier, a '(', the string data, a ')', and then a ';'), and switch to MiniD mode. It goes until it comes to a ?> tag, where it switches back to HTML mode. And repeat.

The easy way would be to have a preprocessor which would look for <? and ?> tags in the page source. It'd output the HTML as it went, then when it came across a <? tag, it'd skip it, read until the ?> tag, then pass that slice of the input file to loadStatementString or so, directing MiniD output to the same output buffer. And then keep going.

The first requires a bit more work, but you end up with a MiniD function that represents the page, which you then run to generate the output. This might be useful if you wanted to i.e. cache a common page: just load the page once, and whenever the page is requested, just run that function to generate it. The second one is a bit simpler to implement (probably can be done with 30-40 lines of code, and done without modifying MiniD), but you have to do this processing every time the page is requested. (Well, maybe you could come up with some kind of caching scheme but it wouldn't be nearly as easy as the other way.)


Last edited by JarrettBillingsley on Thu Sep 27, 2007 7:24 am; edited 1 time in total
Back to top
View user's profile Send private message
bobef



Joined: 05 Jun 2005
Posts: 269

PostPosted: Thu Sep 27, 2007 1:30 am    Post subject: Reply with quote

I tried the second way but instead of printing the HTML chunks I wrap them in write(`HTML goes here`); and then pass the whole string to MiniD. This way I can cache the results, but it has one draw back. The blind replacement, without actually lexing the MiniD part may result errors if there is ?> inside a string. That's why my question was more about the first way. How to modify MiniD. I mean, please give me few tips where to start without having to learn all your sources.
Back to top
View user's profile Send private message
JarrettBillingsley



Joined: 20 Jun 2006
Posts: 457
Location: Pennsylvania!

PostPosted: Thu Sep 27, 2007 7:43 am    Post subject: Reply with quote

OK so the only changes you'd be making would be in minid.compiler. Specifically, the Lexer class, in the lex method (the very first method in the class). You're going to have to add another (static) member to the class, something like mIsHTML or something, as a flag to indicate whether you're currently skipping HTML or parsing MiniD code. Start off with that set to true. Notice right after the check for the "#!" on the first line, it goes into a loop, grabbing tokens and turning them into a linked list. This is where you'll be making the first change.

Change that loop so that it's something like "while(mPosition < mSource.length)". Inside the loop, you'll check to see whether you're consuming HTML or not. If you are, just keep incrementing position, checking for a '<' character. Come across one, go to the next, and check if it's a '?' character. If it is, go to the next character and turn off mIsHTML. Slice the HTML source and turn it into a string literal token (make a new token, set its type to Token.Type.StringLiteral, and its stringValue member to the slice). You'll also have to create a few more fake tokens, like a 'write' identifier, a '(', a ')', and a ';'. Link them all together in the correct order and add them to the token list. In that loop, if mIsHTML is not set, you just parse MiniD code as its written now, with one exception: check if the result of nextToken is null. If it is, don't add the new token to the list (since it's not a token, it's null).

The other change will be in the Lexer.nextToken method. You'll be modifying it to check for '?>' tags. Go to the "case '?':" case. Notice it moves to the next character, then checks if the character is '=' for the "?=" token. Add in another 'else' there to see if the current character is '>'. If so, go to the next character, set mIsHTML to true, and return null. This is why you had to add in that null check to lex().

That should be just about it.
Back to top
View user's profile Send private message
csauls



Joined: 27 Mar 2004
Posts: 278

PostPosted: Thu Sep 27, 2007 1:59 pm    Post subject: Reply with quote

I actually did some experimenting with this before, myself. One problem I ran into was MiniD's requirement that the module statement not only be present, but be at the top of modules. So if you intend to treat your MiniD-HTML files as modules, you'll have to overcome the possibility of a write(HTML); coming before the module statement.

What I did in my just-for-concepts-only implementation was to specifically check for the prefix pattern '...some-html... <? module *;' and shift the module statement to the top. Needless to say, this won't work for anything meant to see serious use.

There are various options. You could do a more slick shift of the module statement, or you could explicitly tokenize the module statement before entering the main lexing loop, or you could relax the rules in the case of a <?...?> block coming before the module statement. Probably more possibilities.
_________________
Chris Nicholson-Sauls
Back to top
View user's profile Send private message AIM Address Yahoo Messenger
JarrettBillingsley



Joined: 20 Jun 2006
Posts: 457
Location: Pennsylvania!

PostPosted: Thu Sep 27, 2007 2:16 pm    Post subject: Reply with quote

Or use the compileStatements function in minid.compiler instead of compileModule. That skips the module declaration entirely.
Back to top
View user's profile Send private message
csauls



Joined: 27 Mar 2004
Posts: 278

PostPosted: Thu Sep 27, 2007 2:20 pm    Post subject: Reply with quote

But what if I want to do creative things with import statements? Wink A MiniD equivelant to using PHP include()/require() for SSI-like behavior.
_________________
Chris Nicholson-Sauls
Back to top
View user's profile Send private message AIM Address Yahoo Messenger
JarrettBillingsley



Joined: 20 Jun 2006
Posts: 457
Location: Pennsylvania!

PostPosted: Thu Sep 27, 2007 9:38 pm    Post subject: Reply with quote

And you can't do those things if you use compileStatements? Import statements are normal statements.. the only thing compileStatements does is skip the module declaration.
Back to top
View user's profile Send private message
bobef



Joined: 05 Jun 2005
Posts: 269

PostPosted: Fri Sep 28, 2007 10:46 am    Post subject: Reply with quote

JarrettBillingsley wrote:
OK so the only changes you'd be making would be in minid.compiler. Specifically, the Lexer class, in the lex method (the very first method in the class). You're going to have to add another (static) member to the class, something like mIsHTML or something, as a flag to indicate whether you're currently skipping HTML or parsing MiniD code. Start off with that set to true. Notice right after the check for the "#!" on the first line, it goes into a loop, grabbing tokens and turning them into a linked list. This is where you'll be making the first change.

Change that loop so that it's something like "while(mPosition < mSource.length)". Inside the loop, you'll check to see whether you're consuming HTML or not. If you are, just keep incrementing position, checking for a '<' character. Come across one, go to the next, and check if it's a '?' character. If it is, go to the next character and turn off mIsHTML. Slice the HTML source and turn it into a string literal token (make a new token, set its type to Token.Type.StringLiteral, and its stringValue member to the slice). You'll also have to create a few more fake tokens, like a 'write' identifier, a '(', a ')', and a ';'. Link them all together in the correct order and add them to the token list. In that loop, if mIsHTML is not set, you just parse MiniD code as its written now, with one exception: check if the result of nextToken is null. If it is, don't add the new token to the list (since it's not a token, it's null).

The other change will be in the Lexer.nextToken method. You'll be modifying it to check for '?>' tags. Go to the "case '?':" case. Notice it moves to the next character, then checks if the character is '=' for the "?=" token. Add in another 'else' there to see if the current character is '>'. If so, go to the next character, set mIsHTML to true, and return null. This is why you had to add in that null check to lex().

That should be just about it.


Thanks for the good explanation. I will do this soon and give you feedback on success.
Back to top
View user's profile Send private message
bobef



Joined: 05 Jun 2005
Posts: 269

PostPosted: Mon Oct 01, 2007 10:48 am    Post subject: Reply with quote

Quote:
Thanks for the good explanation. I will do this soon and give you feedback on success.


I made it and it works. Nice Smile I also added a flag to loadModuleString and loadStatementString so the user can choose to parse the code as HTML or as pure MiniD. It is in MiniD mode by default so it is not breaking any code. Is anyone interested?
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Mon Oct 01, 2007 2:44 pm    Post subject: Reply with quote

bobef wrote:
Quote:
Thanks for the good explanation. I will do this soon and give you feedback on success.


I made it and it works. Nice Smile I also added a flag to loadModuleString and loadStatementString so the user can choose to parse the code as HTML or as pure MiniD. It is in MiniD mode by default so it is not breaking any code. Is anyone interested?


Absolutely, I'm very interested, it would be nice if you could put it online.
Back to top
View user's profile Send private message
JarrettBillingsley



Joined: 20 Jun 2006
Posts: 457
Location: Pennsylvania!

PostPosted: Tue Oct 02, 2007 7:46 am    Post subject: Reply with quote

Glad to see it worked out for you Smile You might be able to host it under Scrapple for the time being, I may set up some kind of MiniD.Scrapple for modifications that people make.
Back to top
View user's profile Send private message
bobef



Joined: 05 Jun 2005
Posts: 269

PostPosted: Tue Oct 02, 2007 8:58 am    Post subject: Reply with quote

This would be good. I was wondering where to upload it.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> MiniD All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group