DWT Linux Browser Status
2008 November 2
Comment by John Reimer - JJR
Time for my first update. It's been over three months since I started working on porting the linux Browser package. About one month of that consituted a hiatus as things got busy in real life. Now, I am happy (and relieved) to say that a working alpha version of the Browser package is now available for Linux DWT in the repository. Within the next few days, I'll be including instructions on how to use it (installation and dependencies). Meanwhile, I'll cover some of the details of my porting adventure here.
Initially, I wasn't quite sure how to approach the whole thing since I had no practical experience with XPCOM programming, beyond some passing familiarity with COM programming from years before.
All I knew was this:
- The SWT Browser package for linux was based on Mozilla XPCOM for browser embedding
- The Browser package continued the SWT tradition of coding as many low level details within the Java modules as possible.
- The code looked far too complex for a XPCOM novice to play with
As is sometimes my inclination, I acted the fool and proceeded with the port, rather nervously hoping it would "just work" out in the end.
First, I started to do a little research and came across a few references online which proved helpful in introducing me to XPCOM concepts. Other than that, I found it to be excessively difficult to dig up material concerning XPCOM. Information on how some component works is often hidden inside source code samples in forum discussions. One needs some serious digging skills to uncover anything with even a smidgen of sufficient context. There were some API references, but these did little to describe how things worked together as whole. XPCOM seems to have become a moving target, even though there are a number of interfaces that it counts as "frozen".
In order to tackle this project, I ended up informally breaking down the port into several phases:
- Phase 1: Study up on XPCOM: get a basic understanding of what it is, what it does.
- Phase 2: Find an appropriate way to support XPCOM from D: specifically I was looking to adopt any work that had accomplished XPCOM interfacing for D already.
- Phase 3: Inject the D XPCOM interfaces into the Browser project and adopt modules that reflected the equivalents of the Java XPCOM components in use.
- Phase 4: Analyze the SWT Browser modules to see how classes could be integrated with the D XPCOM interfaces, or if they even could be.
- phase 5: Start converting the SWT Browser package modules with the above assumptions, all the while with crossed fingers.
- phase 6: Execute a cyclical compile period until all syntactic and semantic bugs are eliminated.
- phase 7: Link to a sample application and work out the remaining logic bugs
In between phases, of course, I would test certain theories to see how the D compiler handled code conversion. And because D has an interface type, I really hoped I could use that. It would allow me to circumvent some of the workarounds that the Java code required because of it's lack of the same (XPCOM compatible interface, that is). I further hoped that the port to D would significantly reduce the Java code complexity.
Phase 1 was ongoing too. But the initial research was enough to get me going. I found this online book quite useful: Creating XPCOM Components
Phase 2 consisted of reviewing a dxpcom project that had already been started and partially completed about a year prior. This project, by a Chinese D language group, was useful in moving the Browser port forward and saved me a whole lot of work. The project consisted of a modification of the C XPCOM generator (xpidl) to spit out D interfaces instead of C++. The project applied this ported xpidl tool to create the 1360+ XPCOM interfaces for D. Only about 145 or so of these were necessary for the Browser port. There were some problems that surfaced from using dxpcom, however:
- dxpcom was designed for phobos and thus had phobos imports
- dxpcom generated interfaces that did not match some of the Java SWT ones due to version differences
- Java SWT XPCOM interfaces were drawn from several different versions of XPCOM (mix and match, so to speak)
- dxpcom generated modules that had duplicated imports, redundant imports, and perhaps some yet undetected bugs
- dxpcom was built to work with windows only
I managed to workaround all these problems and take the best from dxpcom, nonetheless. The biggest problem with dxpcom, however, remained the last one. My project was for linux, and dxpcom used a "trick" to make D COM interfaces work with XPCOM on windows only. The same trick couldn't work on linux... well, not in the same way, at least.
Two problems surfaced with the D interfaces while I was working on this solution. Earlier I described that XPCOM and COM were very much alike. In fact, they are so much alike that their interfaces are binary compatible. COM's IUnknown can map directly to XPCOM's nsISupports. The provided contracts of QueryInterface?, AddRef?, and Release exist in the same vtable slots as each other (0,1,2). This is an important difference from regular D interfaces which push the first method to the slot 1 in the vtable. Slot 0 appears to become a special pointer for interconnecting with regular D classes. D interfaces further appear to have a different structure altogether since classinfo is also supported. It's easiest to see COM interface support in D as simply removing all the frills of the language interface in order to accommodate the COM compatibility. This feature of D becomes extremely convenient for working with COM and XPCOM.
That's where the first problem comes in, though. COM is windows only. The XPCOM "trick" is to use the D COM interface type (which is internally detected by the compiler as soon as you declare or inherit from an interface called IUnknown) by aliasing nsISupports to IUnknown. Voila, you can now connect to XPCOM interfaces. But my port is for Linux, not windows... so what to do?
COM methods are _stdcall, which means they are declared extern(Windows) internally, literally forcing _stdcall decoration on the contracts whether you see it or not. The only reason I discovered this is because I tried to fudge the rules a bit and force the same alias on Linux, just hoping that I could recourse to using a COM aliased interface to connect with XPCOM objects. It didn't compile. A dreaded covariance error spewed forth from the compiler. When I first saw it, I thought the compiler had a bug because the error complained about function return types not being equivalent: "voidWindows" not being covariant with "void" returns types. What in the world is "voidWindows"? Frank caught this one and pointed out that it was due to my using the "alias IUnknown nsISupports" (which I had adopted from dxpcom). After some investigation, I realized that this immediately caused the compiler (even on linux!) to assign _stdcall calling convention to the interface.
So what could be done for the linux port of Browser? Without access to the COM Interface, I'd have to resort to arcane workarounds that rivaled the Java version. I decided to push a little harder with the COM idea. What if it were possible to override the calling convention. What I needed on linux was extern(C) for all interface methods instead of extern(Windows). What if I could mix and match? I tried it and it worked. I forced extern(C) calling conventions on each method inside the interface, and this effectively overrode the internal compiler assignment of extern(Windows). Thus I obtained a COM interface with C calling convention, an appropriate interface for linux XPCOM.
But like many things, that wasn't the end of the story. You just can't talk behind a compiler's back and expect to get away with it. It's bad enough programming practice to begin with. The Browser classes that inherited from these interfaces were immediately assigned extern(Windows) by the compiler for all methods implementing the interface. I fought back determinedly and overrode the calling convention with extern(C) for those class methods. Then I also had to assign extern(D) to those non-interface methods in the class (that might inherit form another D class). What a mess... But this mess worked.
After a number of painful debugging sessions, a working Browser sample came to be (see Sample136.d and Sample128.d in the dwt-samples/snippets/browser package). XPCOM was proved to work on linux surprisingly effectively using D. There are yet more bugs to work out, but the hard work is done.
I remain somewhat uncomfortable relying on a compiler workaround as I have done. This may not be a compiler portable solution, and I would much rather see something like a pragma activated by the compiler to accomplish the same task. We'll see where this goes. I may post a request in the D newsgroup concerning this problem at some point. For now, I've got more work to do. :-)
Collection imports and Language files
2008 October 18
Comment by keinfarbton
The issue for having a all.d was discussed more than once before. Today I added all.d files for each main package in the SWT ports. And there is also a dwt/all.d that import everything (expect opengl - need external bindings) and there is also a dwt/std.d which import the most essential.
Also a new feature of DWT is now the support for the SWT language files. For example the texts on messagebox buttons should show up in your local language.
DWT Cocoa status
2008 October 10
Comment by Jacob Carlborg - doob
I've manged to compile a DWT Cocoa "Hello World" application. It is running without any runtime errors but there are too many things that are not implemented yet to be useful. For example, you cannot close, resize or minimize the window. In short, it doesn't respond to events. I guess the next step is to update and port the latest SWT revision, there are quite a lot of more things ported and working in the latest SWT revision. An image of the application: dwt-cocoa.png
2008 August 19
Comment by Jacob Carlborg - doob
I'm porting the swt-cocoa version and tough that I share my progress. I've uploaded the prepared source of the whole dwt tree to the mercurial repository. I've made the bindings for the necessary Objectiv-C functions and types and I'm quite sure that they all work. Though there is one function (so far) that doesn't compile because of the bug http://d.puremagic.com/issues/show_bug.cgi?id=1852 (I'm using the gdc bundle from the tango website - dmd version 1.025). I've ported all the NS classes in dwt.internal.cooca and for the few small tests I've made it's working. The tests (main.d) I've made are creating a NSStirng and print it and I've created a NSWindow (image) that displays.
Introduction to Journal
2008 July 28
Comment by John Reimer - JJR
This marks the beginning of the DWT Porting Journal where the developers will periodically comment, discuss, frustrate, and postulate all things pertaining to D, Java, and the SWT to DWT port in between. You, the reader, may think of it as a blog, if you like; but because that word sounds much too trendy and perhaps a little silly, we'll stick to the rather boring and self-serious word "Journal" instead... if it can survive the irony. With some luck, we'll be able to make this page more entertaining and educational than such a dry alias suggests and intrigue our readers into thinking that this page somehow consists of something substantially useful. Of course, the honest truth will be that this Journal helps tidy up vast amounts of horridly disordered D porting thoughts so that the developers can help themselves focus more carefully on systematic problem solving -- a self serving motive if there ever was one.
Well, okay, hopefully the sharing of these recently ordered disorderly musings will also help the reader escape similar porting and development woes and excite a better understanding of D internals and D programming in general.