FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Status Log
Goto page Previous  1, 2, 3 ... 8, 9, 10, 11, 12  Next
 
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries
View previous topic :: View next topic  
Author Message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Mar 28, 2006 9:24 am    Post subject: Reply with quote

Status

Let me preface this by saying that the code that exists on my main development machine is still available to me at this time. Development on DDL is not going to halt, but the current milestone will be further delayed. Read on, for all the gory details. Cool

Oh, the Insanity

The last weekend was nothing short of an adventure for my home computing environment. I recently acquired a used Dell laptop from a reseller friend of mine, and its a very nice machine. As luck would have it, he installed Win2k on it, at the SP2/3 patchlevel. Those of you that know what a bear it is to get such a system up to snuff probably know what happened next.

I plugged the running machine into the network and, in the very next instant, realized what a mistake I had made. Embarassed

Thanks to the high-bandwidth cesspool that I'm attached to (comcast.net), the poor laptop didn't stand a chance. Likely within milliseconds of Windows Networking acquiring an IP from my router, not one but two worms attacked this poor machine, and made it home. The now renegade laptop then attacked my main development machine (likely via file-sharing) and it all went downhill from there.

Yes, I should have taken better precautions. The router should have had a more agressive security policy, and my main desktop system should have had had the screws tightened down long before I tried any of this. Live and learn I guess.

The net result? A CHIR.B and Nimda.E virus/worm cocktail on my desktop machine, with every .exe and shred of html and php code infected. Thankfully, it doesnt attack .d code (or any other media for that matter), so I'll be on a rescue mission to get my latest code out of that machine (and code commit following a file-by-file audit).

Its Alive!

After ripping my apartment to pieces trying to find my Win2k CD, I resigned myself to alternative solutions. After years of battling barn-sized security holes, running bloatware antivirus apps, and bending over backwards to recover broken systems, I decided that I had enough. It was time to build a system that at least has a decent failure mode, and is better suited for my style of use and maintenance.

My old, trusty 400Mhz system was thankfully offline during this fiasco, so I booted it up and went about getting floppy images for a net install of Debian. Said machine will probably be the last Windows machine I run, barring any future project requirements.

I've flirted with Linux in the past, but never really made it very far beyond installing RedHat or Slackware and kicking it around some, only to blow it away for another Windows install. Part of my problem was building custom machines out of wildly unpopular (read: grossly unsupported) hardware, only to have Linux not understand what the fsck I was running. Using a older stock Dell laptop pretty much garanteed me that I was no longer walking my own path.

As it turns out, installing Sarge on this thing was a breeze. I kind of wish that the Debian folks had a separate distro for laptops since I had to dig deep to find power management modules and such. For the time being, it still eats my battery alive whenever it gets the chance, and drops my session mercilessly (turns off) once the battery gives up unannounced. So I'm still teaching this dog some new tricks.

Getting the D toolset to run on this was also very easy. I just downloaded the compiler from Digitalmars and apt-get'ed svn; Build, Mango and DDL were on board soon thereafter. However, I *did* have to muck with the install layout for the D compiler to make it multi-user friendly - Walter seriously needs to get a proper .rpm online for this.

So I'm now leveling my sights on my AMD 3400+ desktop machine to do the same. The problem? Its newer hardware, using the Nforce3 chipset which isn't that well supported - the stock installer for Sarge seems to only want to support Nforce2. All signs are pointing at using the AMD64 build of Debian which is on the V2.6 branch. So while I'm thrilled at the prospect of developing D on a 64 bit system, I'm very hesitant at what that'll do for being compatible with other software out there. If it works, then I'll have a *very* diverse development environment to work from.

So it seems I'm almost done making "linuxade". Just in time to start working the ELF/COFF DDL milestone too. Smile

So What About Windows?

I'm going to continue DDL Windows development under Linux/Wine where I can, and fall back to my old 400Mhz system for testing and validation. At some point in the future, I may spring for a copy of XP and do that step from VMWare on my desktop - until then, things are going to be pretty slow.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Tue Mar 28, 2006 1:06 pm    Post subject: Reply with quote

Hehe, evolution can be brutal Very Happy
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Mar 28, 2006 1:21 pm    Post subject: Reply with quote

larsivi wrote:
Hehe, evolution can be brutal Very Happy


Only in this case, it was really more of the 24hr comic-book style of "evolution", rather than the kind that takes eons to accomplish (like developing software for the government). Indeed I now have amazing new superpowers that were granted me in a short period of time, due to some freak accident that destroyed my lab. Thankfully, I'm still myself when exposed to kryptonite nor am I horribly disfigured. Smile

Also, last I checked, my LAN was not self-aware, so it looks like I'm good to go.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Tue Mar 28, 2006 1:28 pm    Post subject: Reply with quote

pragma wrote:
larsivi wrote:
Hehe, evolution can be brutal Very Happy


Only in this case, it was really more of the 24hr comic-book style of "evolution", rather than the kind that takes eons to accomplish (like developing software for the government). Indeed I now have amazing new superpowers that were granted me in a short period of time, due to some freak accident that destroyed my lab. Thankfully, I'm still myself when exposed to kryptonite nor am I horribly disfigured. Smile

Also, last I checked, my LAN was not self-aware, so it looks like I'm good to go.


I was thinking of that other OS, the one that is extremely reproducable, but have less luck with it's mutations Wink
Back to top
View user's profile Send private message
qbert



Joined: 30 Mar 2004
Posts: 209
Location: Dallas, Texas

PostPosted: Tue Mar 28, 2006 2:32 pm    Post subject: Reply with quote

pragma wrote:

Only in this case, it was really more of the 24hr comic-book style of "evolution", rather than the kind that takes eons to accomplish (like developing software for the government).


Lol , sounds like a nightmare all around Sad . I think soon I too am going to follow the road you've taken and abandon my Win32's for something more secure.

Unfortunately there's really not that many choices for mainstream development , win32 - linux - macintosh . I've been wanting to get a Mac a long time , this year might have to be the year.

I am surpised you went with Debian though, I remember it being the least 'new user' freindly - but that was 6 years ago. I tried a Debian deriviative called Knoppix once that was extremely freindly , and will probably use it again for the next machine.

Does DDL have a stand in developer for Win32 development ? I better check out svn before i ask anymore questions
Back to top
View user's profile Send private message MSN Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Mar 28, 2006 7:01 pm    Post subject: Reply with quote

qbert wrote:
I am surpised you went with Debian though, I remember it being the least 'new user' freindly - but that was 6 years ago. I tried a Debian deriviative called Knoppix once that was extremely freindly , and will probably use it again for the next machine.


Its not too bad. I've tried a few other *nixes in the past and I have to say, its the best of the bunch so far.

Quote:
Does DDL have a stand in developer for Win32 development ? I better check out svn before i ask anymore questions


As such, no. I wish I had more of the documentation complete so others can join in, but SVN is where it all is right now.

If you're looking to join the bug-hunt on the win32 side, you may want to familiarize yourself with the OMF.pdf spec in the repository. It'll give you an idea of what we're dealing with.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Wed Mar 29, 2006 10:21 am    Post subject: Reply with quote

pragma wrote:
qbert wrote:
I am surpised you went with Debian though, I remember it being the least 'new user' freindly - but that was 6 years ago. I tried a Debian deriviative called Knoppix once that was extremely freindly , and will probably use it again for the next machine.


Its not too bad. I've tried a few other *nixes in the past and I have to say, its the best of the bunch so far.
.


IMHO the best distros are the Debian derivatives. I use Kubuntu (Ubuntu with KDE) myself and it is much better at autodiscovering hardware for some reason. It also has much more recent support for hardware. I have installed the Dapper release candidates on both my computers and they are now working very well, connected to the net by WPA-encrypted wireless.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Apr 03, 2006 1:29 pm    Post subject: Reply with quote

I would like to post yet another big thanks to the D community for being so supportive with my recent lab mishap.

Status

I'm going to be asessing the codebase to see where things are in terms of the, now very skewed, project roadmap. If I think that OMF support is solid enough to call it 1.1 then I will, otherwise, I'll consider merging the 1.1 and 1.2 milestones into a continuation of the 1.1 refactoring step -- Lars has already made some very serious headway into ELF support so its not as much of a coding push as I envisioned so many months back.

I've recovered the latest batch of updates from the ashes of my old HD image, and I'll be committing these as soon as I get subversion installed on my desktop.

Lab
The lab is now shaping up. I'm now looking to dual-boot my desktop with XP and Debian (rather than going pure linux) for a few reasons:

1) Mrs. Pragma needs iTunes support.
2) I still need a good Windows development environment.
3) The hardware doesn't seem to be 100? exploited under Linux for a wide variety of reasons (Shuttle XPC SN85G4 V3 for those interested).
4) Wine doesn't do everything - but it comes pretty darn close.

For a few reasons, I've taken to naming the machines after the EVA's in Evangelion - and of course they're complete with the appropriate wallpaper for extra nerdy goodness.

EVA-00 - P2 233Mhz (downgraded some time ago due to mobo problems)
EVA-01 - P3 Laptop
EVA-02 - AMD-3000 Desktop

Those that have seen the series will appreciate the fact that EVA-00 seems to be 'off' most of the time until EVA-02 is in serious trouble, as was the fact last week. Also, the laptop seems to have only a 30 minute battery life - I think its a power management thing moreso than a bad battery.

Linux Land
(or "Pass 'Go', and do not recompile kernel.")

For reasons that will likely be forever beyond my comprehension, the Debian community is still pushing the 2.4 kernel even though *everything* seems to run better under 2.6. Virtually everything ACPI based, and then some, was causing wierd problems (read: "Fn" key = lock up my laptop) until I switched. Thankfully, aptitude/apt-get makes everything pretty brainless once you find the right tutorial online.

I've set up my dev environment, and I'm shopping for a good text editor (vi *and* emacs fans need not respond) to use while in gnome. Bluefish seems to work okay, so I may invest a little more time using it first before I decide on something else.

My only problem now is trying to use that laptop in a dark nightclub. As the screen brightness only goes down so far, the compact keyboard and its impossible layout is hidden in that inky blackness that is the entire world when that screen is aimed at your retna. I'm seriously considering a mod of some kind, like a LED lamp aimed at the board. Its either this, or lug a 144 key keyboard with me so I can touchtype more effectively.

Anyway, I'm starting to find this a very pleasurable experience. My only major gripe is that I have to be online if I'm going to figure out how to do anything new with this system. The pre-installed documentation is very, very slim and doesn't really point you in the direction of anything more substantial than 'man' or google. I'd happily apt-get a reference of some sort, if only there was enough documentation to tell me what such a project is called. Sad
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Apr 05, 2006 8:21 pm    Post subject: Reply with quote

Status

Today's pass marked a new step for improving the DDL workspace so I can continue to work in a cross-platform environment. The roadmap will be revised shortly to reflect all this - the ultimate release date be delayed by at most a month because of all this (sorry).

Anyway, the .bat files had to go. Ideally, I wanted to use a script format that would suit both windows and *nix environments with 100? of the same code. I found that even with the GNU utils for windows installed, there was still the issue of slashes standing in the way of full compatibility. So I decided to take a very different route.

Cross-Platform Scripting

Somewhere between shell scripts and makefiles, exists this wonderful little niche fufilled by D itself.

I was inspired by the talk out on the newsgroup surrounding the notion of using .d for scripting, and came to the conclusion above. So, I gave it a shot for DDL build scripting. After all, it couldn't be any worse than TCSH scripts, right?

Code:
#!/usr/bin/dmd -run

import utils.Script;
mixin Script!();

void main(){
   auto packageName = "../downloads/ddl.utils." ~ ddlBinVersion ~ ".zip";
   
   echo("* Building Utils");
   
   build("-full","utils/ddlinfo.d");
   build("-full","utils/bless.d");
   build("-full","utils/insitu.d");

   removeFile("ddlinfo.map");
   removeFile("bless.map");
   removeFile("insitu.map");
      
   echo("* Packaging Binaries");
      
   removeFile(packageName);
   zip(packageName,
      exeFile("utils/ddlinfo"),
      exeFile("utils/bless"),
      exeFile("utils/insitu")
   );
}


The above is the buildutils.d file that replaces the buildutils.bat file. It covers 100? of the same functionality and eliminates the need for an installed line-command zip utility (this should make everyone happy) and requires just vanilla DMD to run/create. I'll also add that this is the most complicated script out of the complete set for DDL.

In retrospect, I realize that 'echo' isn't really needed and should probably be writefln() instead -- Embarassed

You may have noticed the bang-line at the top: a suitably configured *nix system can execute this as a script directly. The rest of us simply need to type "dmd -run buildutils" and off it goes. You also don't need build to compile one of these scripts thanks to the use of the Script!() mixin - that way, everything is forced into this module so a single pass by DMD will do the trick.

FYI, I tried to do this without the mixin, by using implicit template instantiation, but that feature is pretty broken under some very needed conditions.

The functions in the Script.d file iron out the kinks between platforms like slashes in paths, what commands are needed on each system and details like executable-file extensions. Its all in there.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
neilbryant



Joined: 09 Apr 2006
Posts: 1
Location: Richmond, VA

PostPosted: Sun Apr 09, 2006 11:41 am    Post subject: Reply with quote

Quote:
I've set up my dev environment, and I'm shopping for a good text editor (vi *and* emacs fans need not respond) to use while in gnome.


I've been a fan of Ultraedit under Windows, for years, but recently I've been moving over to SciTE, partly for its' Linux portability (I haven't actually tried it on Linux yet, but I'm thinking ahead to my own lab meltdown ;)

Been pretty happy with SciTE - lots of syntax highlighting options, ongoing development, and LUA scripting.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Apr 12, 2006 10:12 am    Post subject: Reply with quote

Thanks to neilbryant for pointing me in the right direction on editors - its nice having the same environment everywhere you go.

Status

The current codebase is dirty. It compiles but is in dire need of testing for the Archive loader and ELF loader parts. COFF will be next for refactoring and testing after the ELF libs are vetted.

Proper documentation is still pending. I'm toying with the idea of side-stepping ddoc completely and using LM to generate documentation directly.

Yesterday saw a flurry of updates by yours truely. I've been in a pattern for the past few weeks of developing on the laptop (linux) while out at the pub on Sundays, then I swing back to Windows at home for verification and testing. The code only makes its transit from one workstation to another via SVN.

The result is that I can catch missing source files, bad commits, and non-portable code much faster than before. I also get the end-developer's point of view on using the codebase as everything I work with comes down from SVN before I start typing.

That said, the inital refactoring pass for ELF and AR is done. Now I have to verify the code against the specifications and make sure that I did it right, followed by a good old fashioned set of tests. Expect the /test directory to expand over the next few iterations.

Linux

Linux binaries are now available in the downloads area.

Apparently my template-import trick for running D scripts doesn't work under linux. I'm going to have to raise the bar a bit and require build for running and compiling D scripts, in favor of having a more succinct command line. The internal bang lines will be changed as well.
Code:
build -run buildutils.d


I noticed an odd bug with the GC under linux, using DMD 0.150. Trying to consume phobos.lib Segfaulted somewhere in gcx.d - presumably while trying to free a large block of memory. It doesn't look like its my code that's causing a problem, but then again, I've yet to ensure that collections wont' screw with DDL yet anyway (not at that stage yet).

I'll try this with 0.153, to see if there's any difference. I may wind up replacing libphobos.a with a debug version of phobos and use gdb to find the problem.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Apr 18, 2006 9:07 pm    Post subject: Reply with quote

Status

I'm finally homing in on the 1.1Beta milestone. The latest revision, 177, is full of bugfixes, updates and improvements.

The biggest, no-brainer improvment was to dump the demangler code in favor of Walter's implementation in Phobos. This way, any changes made to the reference implementation will automatically become incorporated into DDL. The move made me a little sad though, as the original was backed on James Dunne's parser from way back in the project, so he is no longer attributed in the code. He still gets credit as a secondary contributor, so he'll remain on the Contributor's page in the wiki. I wish him well. Smile

The build numbers for the utilities became skewed somewhere around 170 when I did an automated conflict resolution via SVN. The result was two different build-number versions becoming concatenated. So I just moved all three build numbers to 500+ and that was that.

Linux
Lars mentioned to me that he had some problems building /trunk under linux. Are there any other intrepid linux users out there that would be willing to try the latest DMD and Build against /trunk as it currently stands? I'm curious to see what the landscape looks like out there, and if there are any variables that I haven't accounted for in my own (minimal) Linux setup. Smile
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Apr 27, 2006 11:42 am    Post subject: Reply with quote

Kris, Lars, All
I moved that last conversation out to its own thread:

http://www.dsource.org/forums/viewtopic.php?t=1450

It also has a ticket for tracking the issue at hand:

http://www.dsource.org/projects/ddl/ticket/46#preview
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Apr 27, 2006 12:20 pm    Post subject: Milestone 1.0B Reply with quote

Status

The Milestone 1.0B is complete. There are still some outstanding bugs but the goals outlined for this step are done.

http://www.dsource.org/projects/ddl/milestone/Beta?201.1?20-?20Refactoring

This represents 19 tickets, two (minor) time overruns, a major redesign and a lot of API changes.

I would like to thank Lars and Kris for their support in testing and validation for helping make this possible. I would also like to thank the rest of the D community for their support for DDL both in concept and in deed.

What's Next?

http://www.dsource.org/projects/ddl/milestone/Beta?201.2?20-?20COFF?20and?20ELF?20support

The next milestone will be a two-part effort. Part one is to get the ELF loading portion of DDL online so we can meet the portability goals stated in the milestone. The second part, COFF loading, will be done in tandem, largely as a refactoring effort as it hasn't been touched since before the last milestone.

All the while, we will be accepting bug reports against 1.1B, and will be folding those fixes into the 1.1B milestone trunk. The downloadables will all freeze at the 1.0B mark while this is taking place.

Documentation

The docs are horribly stale, and will probably remain that way for a while longer. I'm looking into using a solution less reliant on external tools, such as Candydoc or a custom ddoc parser - my XML solution just seems too hackish to be very useful. Candydoc does a fine job, but it suffers from the same problems that standard ddoc does as it has no cross-referencing or diagrams.

As for custom parsing, I've been working on a parsring framework in the same vein as DGrammar called "Enki". I have a very crude prototype that digests EBNF (with annotations) and emits a parser frontend in D code (non table-based). I'm almost to the point of having the frontend 100? self-hosting.

Enki

I think that a tool like Enki may help with developing a custom ddoc processor, that can handle all the features we come to expect from toos like doxygen. I have no clue how much time it'll take to get working, but it should be cake once Enki matures a little bit more.

One oddity I did encounter while developing Enki, is my choice for a prototype code generator. Rather than use a table-based parser, I wanted each production to have its own method, which would in turn use other methods for the other BNF terms. I used a scheme of anonymous delegates and thrown exceptions (for failure) to tie the whole mess together. Despite how the code looks, it runs very fast - not at all what I expected. It also lends itself toward very readable code, which is a nice tradeoff for potential inefficencies if you ask me.

Anyway, I may stick with this scheme for a bit longer, and then move to a goto/return-bool (kind of like a stripped down if-else) style design to squeeze the last bit of performance out of the compiler. I haven't quite worked out the kinks for expressing the BNF parse tree as the latter, as its a more complex way of doing things.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon May 08, 2006 11:21 am    Post subject: Reply with quote

Status

I've pushed the coff code around a bit, to set the stage for refactoring. I've also been spending more time in Linux land lately, investigating the potentials of DDL in that environment as well as validating code. ELF bugfixes are still pending further work.

Linux

Its hard to say if its the nature of my machine's configuration, or DMD in general, but I'm still having the same segfault issues with ddlinfo when accessing phobos.lib. If its still the same old problem, then I may start looking into building a custom libphobos to help debug the problem.

Outside of that, I learned a few things about the drawbacks to to ld.so and the dynamic loading capabilities native to Linux. Basically, ld can be coerced into generating a proper map file that can be parsed as an in-situ module. In a perfect world, I'd like to interrogate ld.so at runtime to pull in that information, since its already a apart of the running binary - but that info *can* be removed via the "strip" command. Also, there seems to be no formal API to do this - and my timeline doesn't account for any kind of deep hacking like that. So this feature is looking like a post 1.0 thing at the earliest.

The other thing is that the map symbol info includes dependencies that are bound by ld.so at runtime. As these addresses are mapped in the executable's memory space, I'm pretty sure these are trampolines, but I have no way to identify them as such (yet, anyway). This could get very tricky.

The problem with getting the linker to generate a map file, is that DMD likes to use 'gcc' as the linker under Linux, and it doesn't handle the "-L" flag correctly for everything. Build 2.10 would seem like the right way to go on this, thanks to Derek breaking out the link step separately. However, last time I tried this version, it wouldn't behave as nicely as 2.09 did. I'll make another pass on this, so I can get things working correctly.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries All times are GMT - 6 Hours
Goto page Previous  1, 2, 3 ... 8, 9, 10, 11, 12  Next
Page 9 of 12

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group