FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

DDT Development contribution

 
Post new topic   Reply to topic     Forum Index -> DDT
View previous topic :: View next topic  
Author Message
progician



Joined: 22 Feb 2011
Posts: 5
Location: UK

PostPosted: Tue Feb 22, 2011 2:25 am    Post subject: DDT Development contribution Reply with quote

"Hi Bruno,

Hope you're well. I was playing around with the outline view (found it, sorry for the disturbance for it) and I came out some better more nicer view as it was. It is not ready by all means, but I am sending this patch because I would like to have your review before I move down this path. I just realized that you moved the project to mercurial, so I don't know how the patch could be applied, but I'm sure you still have the svn checkout directory on your box.

Basically what I did is that I reproduced (non-copypaste) some relevant parts of the JDT modified with the DDT's AST classes. It's not a big deal, but I found as a good entry exercise. Obviously I ran in to problems with the AST which is not that clear for me to add any code to it, so I tried to just came along with it. Frankly, I'm still confused about the d-tool project because I'm not sure how the old descent stuff is connected with all these other classes. But soon I will figure it out. The other issue was/is that I'm having difficulties finding proper documentation on the dltk, so i preferred to go with the JDT patterns, but I realised that this could cause some interference or redundancy. What's your view on this?

I couldn't push the binary files in to the patch, so I attached the new images from workspace root. I pull the DeeElementLabelProvider and co. classes to the ~.ui package root because logically it should be the replacement for the Script(UI)ElementLabelProvider at some point and all the views would use it later on such as the package view, script view, code completion, debug windows and so on. This was the point where I could not come to terms with the DLTK API so I went down the road wit the JDT version...

The BIG thing would be a debugger launcher with the proper views, but as far as I can see, there is no good option out there yet. The most promising the GDB which can be used across 3 compilers, but has poor support of D specific stuff (not sure how the patches applying here), so maybe I should take a look at that project also. Since the CDT working with gdb too, i guess it can be helpful to have a look at it.

Best Wishes"

Since this email I moved my changes to a mercurial repository clone, so ignore the parts about the patch. You can find all the changes I made and I will make at the http://code.google.com/a/eclipselabs.org/r/gyulagubacsi-sandbox/source/checkout.

http://img215.imageshack.us/img215/9861/ddtoutlineviewupdatedsc.jpg ((EDITED image to url link))
Back to top
View user's profile Send private message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Wed Feb 23, 2011 11:50 am    Post subject: Re: DDT Development contribution Reply with quote

Hi Gyula, again, thanks for the interest!

I haven't had the time yet to look to this with more detail. I want to finish and wrap the current task I'm working on (Auto Edits and stuff, relating to 3 open bugs). But there are a few things I can answer already:

progician wrote:
Frankly, I'm still confused about the d-tool project because I'm not sure how the old descent stuff is connected with all these other classes.

So, dtool uses Descent to parse D source (parsed into the Descent AST), and then that Descent AST is entirely converted in a new AST, the Dtool AST, which is an AST with a very different API and internal structure. After this, for all other DDT code, only the DTool AST is used and it has no knowledge of Descent code.


DTLK is often quite short on documentation, that is true. For most DTLK functionality, the best way to understand it is to download the source of one of the reference DLTK IDEs (like the Ruby or Tcl one), and see how they did things.

In the particular case of the ElementLabelProvider, I think this is specified using an extension point (not the most natural way one would think). But eventually this is what should be used, so that element icons and labels are available in all other views, not just the outline.
But this poses some problems, because although in the case of the outline you have an editor, full source and so you get the full AST for the source, however in the general ScriptElementLabelProvider case you will likely only be able to work with ModelElement's so these must be created with the necessary information (the attributes like private, protected, final, static etc., etc.) to create the icons and decorations.
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
progician



Joined: 22 Feb 2011
Posts: 5
Location: UK

PostPosted: Sat Feb 26, 2011 1:53 pm    Post subject: Tons and Millions of questions... Reply with quote

Can you tell me, how can I check out the source of the DLTK as it can be used in my workspace consistently with the ddt source? I was trying to use the Import Plugins/Fragments from PDE, and CVS but I never ended up a meaningful workspace where I the references are resolved properly. I'd like to stick some breakpoints to the DLTK classes as well to find out its behaviour...

I found the parts you were referring to, the ModelElement stuff is going through the DeeSourceElementProvider, which is kinda awkward, because 1. you convert the descent parse tree, 2. you produce the AST, then the DeeSourceElementProvider traverse this AST and produce the ModelElement hierarchy (even if it filtered). At this point the DLTK becomes partially incompatible with the D language structure, and I try to find out how can I extend it. As far as I can see, your DeeScriptUILabelProvider is never called by anything, though it's super class is responsible with the script explorer's labels. The provider is created, but never used.

The real difficulties come when I want to find any ways to replace the ModelElement hierarchy in the Script Explorer with DDT's ASTNeoNode's hierarchy, though it would make a lot of sense. You said, that the outline view is a different case, since their there is the full source code, and it's AST. But the D source parsing is called for each module, isn't it? I mean as far as I see, the ModelElement hierarchy is built from the original AST, isn't it? My idea was to somehow connect to the SourceModules (if i'm right) our own AST and make sure that the extension point is working (DeeNavigatorLabel/ContentProvider and is working with our new DeeElementLabelProvider?).
In my desperation I was playing with the idea that I create a brand new tree view based on package/project navigator classes. Basically my idea would be to make a view, either with the help of the ScriptExplorer or the PackagesView (the latter seems a bit more difficult, because it seems to tightly coupled with the JDT/JavaElement classes), that groups the modules into fully qualifiered packages (exactly the way how it looks in Java projects), and below the SourceModule level, you would see:

- Types/Aggregates (Classes/interfaces/structs(!Missing from the DLTK model!))
- Enums
- Variables
- Functions
- Templates (nested in the same way, as other aggregates) - I see this part is not implemented yet properly in the AST...
- Aliases
- Basically saying all the definitions, if i'm right(?)

We could apply the module naming rules of D here, because if there's no module declaration present, the module is named after the file. For experiment I forwarded this info from the Descent classes because the module's source file's name wasn't present in our AST.

By the way, as a different question: How the DGrammar files are related to the project. I couldn't get my head wrap around the whole project (it's impressive how you were able to keep up with extent of functionalities of your own project Smile ) yet, and I'm still unsure whether the DGrammar ANLTR files are generating the AST classes, or not. I read your diploma work, but I think I have received too much information at a time.
Back to top
View user's profile Send private message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Tue Mar 01, 2011 6:53 am    Post subject: Re: Tons and Millions of questions... Reply with quote

progician wrote:
Can you tell me, how can I check out the source of the DLTK as it can be used in my workspace consistently with the ddt source? I was trying to use the Import Plugins/Fragments from PDE, and CVS but I never ended up a meaningful workspace where I the references are resolved properly. I'd like to stick some breakpoints to the DLTK classes as well to find out its behaviour...

Both approaches work (Import Plugins/Fragments from PDE, or CVS), but best not to try do mix them unless one knows what one is doing. So, I think the simplest way is:
Go the DLTK 2.0 release downloads:
http://download.eclipse.org/technology/dltk/downloads/drops/R2.0/R-2.0-201006161315/
Then download the "SDK Downloads" archive for the "Core Frameworks", and also for the Ruby IDE, or TCL IDE (or both). Unzip these, and add them to the target platform. (Note, if you have another version of DLTK you should remove it, or just create a new platform from scratch. If it's the latter, you will need the EMF runtime as well)
Now you should already have the viewable sources for any DLTK class in Eclipse, however, the Ruby/TCL IDE classes will not show up in semantic Java search (like Open Type or Type Hierarchy) because they are not on the dependency tree. To setup this, open the "Plug-ins" view from DLTK, select all the ruby and TCL plugins as you wish, open context menu and choose "Add to Java search". (You could also import as a source project if you wanted to change the sources code for example).
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Tue Mar 01, 2011 7:18 am    Post subject: Re: Tons and Millions of questions... Reply with quote

progician wrote:

By the way, as a different question: How the DGrammar files are related to the project. I couldn't get my head wrap around the whole project (it's impressive how you were able to keep up with extent of functionalities of your own project Smile ) yet, and I'm still unsure whether the DGrammar ANLTR files are generating the AST classes, or not. I read your diploma work, but I think I have received too much information at a time.

Nothing in the DGrammar files is being used. Those are quite old files, before Descent was released, when I was beginning to write the parser myself. (if they were used, even if just for the AST classes, there would be a dependency on the the ANTLR runtime I think)
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Fri Mar 04, 2011 1:57 pm    Post subject: Re: Tons and Millions of questions... Reply with quote

progician wrote:

As far as I can see, your DeeScriptUILabelProvider is never called by anything, though it's super class is responsible with the script explorer's labels. The provider is created, but never used.

Yeah, that's what I see as well, it's instantiated but never used. This has confused me in the past, it may be just a deprecated DLTK API or something. I'm curious enough to ask about this in the DTLK newsgroup.
Note for example, that the org.eclipse.dltk.ui.AbstractDLTKUILanguageToolkit.createScriptUILabelProvider() of both Ruby and TCL IDEs return null.

progician wrote:

The real difficulties come when I want to find any ways to replace the ModelElement hierarchy in the Script Explorer with DDT's ASTNeoNode's hierarchy, though it would make a lot of sense. You said, that the outline view is a different case, since their there is the full source code, and it's AST. But the D source parsing is called for each module, isn't it? I mean as far as I see, the ModelElement hierarchy is built from the original AST, isn't it? My idea was to somehow connect to the SourceModules (if i'm right) our own AST and make sure that the extension point is working (DeeNavigatorLabel/ContentProvider and is working with our new DeeElementLabelProvider?).

Hold on, generally speaking, the ModelElement hierarchy is not meant to be a replacement for the AST, the ModelElement hierarchy is much more lightweight (has much less info, like, it doesn't have method bodies, or things like that). Furthermore ModelElements can be opened or closed (they only have child info if they are open). All of this is so memory usage can be reduced, and much more ModelElements can be in memory, as opposed to if the AST was used instead. (This was mentioned in the masters diploma document Smile )

Our UI label provider should work with ModelElement's and not the AST, because of the reasons above (and since that's what DLTK expects). It should still be possible to customized the icons a lot, even with different types (enums, classes, templates), because the created ModelElements has a modifiers field which we can extend with our own flags. (see mmrnmhrm.core.dltk.DeeSourceElementProvider and org.eclipse.dltk.compiler.IElementRequestor.ElementInfo)
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Fri Mar 04, 2011 2:11 pm    Post subject: Reply with quote

So, and as to how to actually extends DLTK to implement the UI label functionality, see this NG thread, it explains it: http://www.eclipse.org/forums/index.php?t=tree&goto=500932&S=9579579d236ecde18f896ed4c71ee538
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
progician



Joined: 22 Feb 2011
Posts: 5
Location: UK

PostPosted: Mon Mar 07, 2011 3:30 pm    Post subject: Element labels... Reply with quote

I check out this thread but honestly, I went through how does it work, and I think it's kind of a rigid thing all together.
To display different images is not a problem as long as I deal with the ModelElement objects. But I still don't get it, why do they need this ModelElement hierachy if there's a full blown AST implementation at hand.
The ModelElements are built from the information collected from the AST and this solution needs an extension not on the modelElementProvider, but the ModelElement itself. Even the subclassing doesn't work here, because the SourceElementParser and co. only provide the basic datas here (type/method/variable/field/function) with a limited number of modifier (my concern here is more about the D specific modifiers, such as immutable).

So much investigation and I will end up with a simple additional flag probably Smile
_________________
Gyula Gubacsi
"The only force stronger than the love for freedom, is the hatred towards they who kill it."
Back to top
View user's profile Send private message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Tue Mar 08, 2011 7:42 am    Post subject: Re: Element labels... Reply with quote

progician wrote:
To display different images is not a problem as long as I deal with the ModelElement objects. But I still don't get it, why do they need this ModelElement hierachy if there's a full blown AST implementation at hand.
The ModelElements are built from the information collected from the AST

Like I said, ModelElement hierarchy is used for performance reasons, mainly to use less memory. Yes, ModelElements are build from the AST, but after they are built the AST can be discarded. And also, (if I recall correctly) the ModelElements are persisted on disk, on an indexer, and are reloaded in a quick way when the IDE is restarted. They will only be recalculated from the AST if the source of the underlying file changes. (otherwise it would be crap to have to reparse all files every time the IDE started)


progician wrote:
ModelElement hierachy if there's a full blown AST implementation at hand.
The ModelElements are built from the information collected from the AST and this solution needs an extension not on the modelElementProvider, but the ModelElement itself. Even the subclassing doesn't work here, because the SourceElementParser and co. only provide the basic datas here (type/method/variable/field/function) with a limited number of modifier (my concern here is more about the D specific modifiers, such as immutable).

Maybe there is a misunderstanding here. When I said "the created ModelElements has a modifiers field which we can extend with our own flags" I did not mean "extend" in the sense of actually subclassing the ModelElements. Instead, I meant extend the functionality of DeeSourceElementProvider to create ModelElements with all information/flags we want for icon rendering (like immutable attributes). In other words, DeeSourceElementProvider will need to be modified to implement this functionality properly.
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
progician



Joined: 22 Feb 2011
Posts: 5
Location: UK

PostPosted: Tue Mar 08, 2011 10:23 am    Post subject: Re: Element labels... Reply with quote

phoenix wrote:
progician wrote:
To display different images is not a problem as long as I deal with the ModelElement objects. But I still don't get it, why do they need this ModelElement hierachy if there's a full blown AST implementation at hand.
The ModelElements are built from the information collected from the AST

Like I said, ModelElement hierarchy is used for performance reasons, mainly to use less memory. Yes, ModelElements are build from the AST, but after they are built the AST can be discarded. And also, (if I recall correctly) the ModelElements are persisted on disk, on an indexer, and are reloaded in a quick way when the IDE is restarted. They will only be recalculated from the AST if the source of the underlying file changes. (otherwise it would be crap to have to reparse all files every time the IDE started)


progician wrote:
ModelElement hierachy if there's a full blown AST implementation at hand.
The ModelElements are built from the information collected from the AST and this solution needs an extension not on the modelElementProvider, but the ModelElement itself. Even the subclassing doesn't work here, because the SourceElementParser and co. only provide the basic datas here (type/method/variable/field/function) with a limited number of modifier (my concern here is more about the D specific modifiers, such as immutable).

Maybe there is a misunderstanding here. When I said "the created ModelElements has a modifiers field which we can extend with our own flags" I did not mean "extend" in the sense of actually subclassing the ModelElements. Instead, I meant extend the functionality of DeeSourceElementProvider to create ModelElements with all information/flags we want for icon rendering (like immutable attributes). In other words, DeeSourceElementProvider will need to be modified to implement this functionality properly.


Oh, no worries, I did understand it and I already started to do so accordingly Smile. I thought I sent a new post... anyways, everything is clear to me on this one, so thank you for your answers, they were much appreciated.
_________________
Gyula Gubacsi
"The only force stronger than the love for freedom, is the hatred towards they who kill it."
Back to top
View user's profile Send private message
progician



Joined: 22 Feb 2011
Posts: 5
Location: UK

PostPosted: Sat Jun 11, 2011 9:13 am    Post subject: Reply with quote

As a theoretical question: If there would be any changes in the Descent project, how can we propagate that? The descent project looks pretty abadoned and if there would be any improvement on it, we need to make sure that it will end up in their repository.

As a restart I found the reason why the Package View doesn't show the constructor overlay and why we didn't have any parameter info on constructors definitions. Both has been fixed and I submitted to the progician-sandbox clone (as far as I know, that's the best way to get new features in to the main repository, you can just pick a revision and pull that changelist to yours if you happy with it).
_________________
Gyula Gubacsi
"The only force stronger than the love for freedom, is the hatred towards they who kill it."
Back to top
View user's profile Send private message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Tue Jun 14, 2011 11:46 am    Post subject: Reply with quote

progician wrote:
As a theoretical question: If there would be any changes in the Descent project, how can we propagate that? The descent project looks pretty abadoned and if there would be any improvement on it, we need to make sure that it will end up in their repository.

As a restart I found the reason why the Package View doesn't show the constructor overlay and why we didn't have any parameter info on constructors definitions. Both has been fixed and I submitted to the progician-sandbox clone (as far as I know, that's the best way to get new features in to the main repository, you can just pick a revision and pull that changelist to yours if you happy with it).


progician wrote:
As a theoretical question: If there would be any changes in the Descent project, how can we propagate that? The descent project looks pretty abadoned and if there would be any improvement on it, we need to make sure that it will end up in their repository.

I don't quite understand what you mean... The only common code (and thus, the only code susceptible to changes by DDT developers) is in the descent.compiler Eclipse project. But that code is already shared in terms of repository.

Anyways, before we continue, I wanna stop using the DSource forum, and move the DDT discussion to Google groups. See my reply here:
http://groups.google.com/group/ddt-ide/browse_thread/thread/ddb50a37b65f7947

(Sorry for yet another move ... this is not something one can just refactor instantly... Razz )
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DDT All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group