FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

.a and.lib conflict
Goto page 1, 2  Next
 
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries
View previous topic :: View next topic  
Author Message
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Thu Dec 29, 2005 5:50 am    Post subject: .a and.lib conflict Reply with quote

It turns out that the COFFLoader use the same magic string as my ArLoader (.a) to identify the filetype, namely "!<arch>\n". So much for your 8 bytes of identification Wink Anyway, it seems like the format is similar, but not entirely the same. Since my ELFArchive is coded quite differently from COFFLibrary, I can't see the immediate differences.

I've used the specs linked to below when parsing the .a-files. Could Eric or AgentOrange look at or point me to the COFFSpec? Maybe the archive loader should be common, with a possibility to hold several types of objects? At least we need to think some more about where to differentiate. Currently my elfloading works because the ArLoader is checked first Smile

http://docs.hp.com/en/B9106-90011/ar.4.html

and

http://ou800doc.caldera.com/en/man/html.4/ar.4.html
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Dec 29, 2005 9:03 am    Post subject: Reply with quote

Yikes, I was afraid this might happen. I'll look into it. Smile

In the meantime, one can always use Bless and force the type to either "COFFLIB" or whatever signature you're using for .a files. At least that way, the resulting .ddl is "signed" with the correct type to load.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Dec 29, 2005 9:31 am    Post subject: Reply with quote

Its amazing how close these two file formats are. Is the .ar type technically a COFF format? Lars, take a look at the PECOFF spec, section 7. As far as I can tell, they're identical.

http://svn.dsource.org/projects/ddl/trunk/doc/pecoff.doc.

I think the COFFLIB loader could be revised to not assume the format of the internal .obj files, and feed the loading of the .obj back into the LoaderRegistry as to properly detect the type. Take a look at the DDLLoader for an example of how that could work.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Thu Dec 29, 2005 11:30 am    Post subject: Reply with quote

pragma wrote:
Its amazing how close these two file formats are. Is the .ar type technically a COFF format? Lars, take a look at the PECOFF spec, section 7. As far as I can tell, they're identical.

http://svn.dsource.org/projects/ddl/trunk/doc/pecoff.doc.

I think the COFFLIB loader could be revised to not assume the format of the internal .obj files, and feed the loading of the .obj back into the LoaderRegistry as to properly detect the type. Take a look at the DDLLoader for an example of how that could work.


(Warning: I realized something after writing a lot of the stuff below, so it might be a tad inconsistent)

Some further reading on how this hangs together, although noone mentions PECOFF using Ar.

http://en.wikipedia.org/wiki/COFF

http://en.wikipedia.org/wiki/Ar_(Unix)

http://en.wikipedia.org/wiki/Portable_Executable

So, when "creating" their specs, MS more or less grabbed what was available in the Unix world, the old COFF-format and Ar.

The Ar-format is just a general Archiving format, not imposing any sort of rules on the content. The binary object files collections in the Unix world, can optionally include two files that isn't object files, a symbol table and a table for those modules that has too long names. The PECOFF spec specifies that three such files are present (a deprecated(?) one, a symbol table, and a long names table). Because of these differences, it might make sense to have two different loaders, or one loader, ArchiveLoader, being able to distinguish between them. Calling it COFFLoader would be plain wrong Smile

It would therefore be most correct to have an ArchiveLoader that can load both COFF and ELF (and possibly others), along the lines I tried to suggest in my first ELF-post, using the existing Registry system. My ArchiveLoader isn't especially good looking at the moment, and is possibly somewhat tied to the ELF-format yet, but should be rather general as it handles the ELF feature of not putting the data in a directly readable sequence. (It's no good reading ELF from start to end). The COFFLoader seems clean, but very focused on it being COFF.

Anyway, a DynamicLibrary is only a collection of DynamicModule's, and thus the internal object formats shouldn't matter for DDL as a whole, if the DDL architecture is good enough.

Also, as you can see on the Wiki-page on Ar, there might be Ar-formats around that DON'T match the format we're discussing at the moment, and I think we might think about accomodating for this, although we probably have enough to accomodate Smile

My suggestion:

ArchiveLoader : Loader (loads .lib and .a, then, if needed, do something with the non-object sections, before using the object loaders to load the object themselves.)

COFFLoader : Loader (loads .obj files, either standalone or as a part of an archive)
ELFLoader : Loader (loads .o files, either standalone or as a part of an arcive)

ELFModule : DynamicModule
COFFModule : DynamicModule
OMFModule : DynamicModule

DynamicLibrary (collects ELFModules, COFFModules and OMFModules and abstracts them away from their internal formats, unless this impose some huge performance hit. Also, the information at the start of the archive files, might differ enough to warrant different implementations, although the only interesting info seems to be the symbol tables, which should be general enough for a single implementation...)

Anyway, I might try to make a common ArchiveLoader in the coming days, if you think the above suggestion sounds okay. The stuff in SVN still works for COFF, anyway.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Dec 29, 2005 12:07 pm    Post subject: Reply with quote

I was hoping that you'd say that MS used the unix ar format, as that certainly helps us out. This is very good news.

We're on the same page, especially with the architecture bit (although you left out the OMF and DDL loaders, but I get your point). Are you suggesting that the registry handle module and library loading as two separate tasks?

Quote:
Also, the information at the start of the archive files, might differ enough to warrant different implementations,


You may want to consider using generic attributes for any useful bits of information that are otherwise hard to express via the DynamicLibrary and DynamicModule formats. I've considered putting a getAttributes() method on DynamicModule, so maybe this is yet more of a reason to do so.

As you suggest, making DynamicLibrary a more responsible/authoritative base-class is certainly the direction I'm trying to move in. Right now, I'm refining some extensions to the interface (look in SVN) that should function as enhancements to linking and better leveraging the proposed path and .zip loaders.

larsivi wrote:
Anyway, I might try to make a common ArchiveLoader in the coming days, if you think the above suggestion sounds okay. The stuff in SVN still works for COFF, anyway.


That seems like the best approach. I'd reccomend that you put it off in its own world under ddl/ar/ARLoader.d and ddl/ar/ARLibrary.d or something along those lines. The tree under /ddl is expected to fan out like this, so it's cool. Smile

For now, I'd expect ARLoader to feedback into the registry and pull the discrete modules out of the returned DymamicLibraries so that they all fall into an ARLibrary instance. Yea, its messy and throws away a lot of objects but it'll get things moving for now.

Hrm.. maybe DynamicLibrary and DynamicModule should be made the same class: just make everything DynamicLibrary instead? That way, the non-module formats would just be assumed to contain nested libraries, and that would be that. (this solves a lot of other problems too)
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Fri Dec 30, 2005 4:24 am    Post subject: Reply with quote

So, what extension should getFileExtension return for ArLoader? Maybe char [][] getFileExtensions() would be more appropriate in any case?
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Dec 30, 2005 8:27 am    Post subject: Reply with quote

larsivi wrote:
So, what extension should getFileExtension return for ArLoader? Maybe char [][] getFileExtensions() would be more appropriate in any case?


I think the only place that is presently being used is under utils/bless.d, where it tries to put the filename back together on an extract operation. Having getFileExtension() return multiple values probably wouldn't do the trick since its still unresolved how the software will pick one extension over the other.

We could depricate the method all together, and add the original filename or extension into the DDL spec so extract operations will return something like the original filename. Since this is the only place its being used, this may be the way to go.

Another way to go is to make getFileExtension() return ".ar" on *nix and ".lib" on win32 -- or just make it always return ".lib". Bless users can force the proper filename using the -f switch.

Thoughts?
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Fri Dec 30, 2005 8:55 am    Post subject: Reply with quote

pragma wrote:
larsivi wrote:
So, what extension should getFileExtension return for ArLoader? Maybe char [][] getFileExtensions() would be more appropriate in any case?


I think the only place that is presently being used is under utils/bless.d, where it tries to put the filename back together on an extract operation. Having getFileExtension() return multiple values probably wouldn't do the trick since its still unresolved how the software will pick one extension over the other.

We could depricate the method all together, and add the original filename or extension into the DDL spec so extract operations will return something like the original filename. Since this is the only place its being used, this may be the way to go.

Another way to go is to make getFileExtension() return ".ar" on *nix and ".lib" on win32 -- or just make it always return ".lib". Bless users can force the proper filename using the -f switch.

Thoughts?


If it isn't used by the loader system for anything, I don't think it should do what it do now. As a library is loaded, maybe the extension should be stored then? Bless then can use getFileExtension() to get the last one used, or preferably just have it in the DynamicLibrary and drop it altogether from Loader.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Dec 30, 2005 9:37 am    Post subject: Reply with quote

larsivi wrote:
If it isn't used by the loader system for anything, I don't think it should do what it do now. As a library is loaded, maybe the extension should be stored then? Bless then can use getFileExtension() to get the last one used, or preferably just have it in the DynamicLibrary and drop it altogether from Loader.


Agreed: it doesn't belong on Loader. Come to think of it, that method doesn't really belong on DynamicLibrary either, so I'll just push it out to the DDL support area and be done with it. Smile

Since this is going to cause a (minor) arch change, I'll notify you when I put this change through.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Fri Dec 30, 2005 12:31 pm    Post subject: Reply with quote

So who the f*** gave MS the monopoly? When using the Ar format for PECOFF, they also broke it. Not that it is an big problem, but these small things annoy me. The Ar spec clearly say:

If the archive contains a symbol table, the file is named "/ ". If the archive contains a table of long names, the file is named "// ". Now those great guys in Redmond decided to do the same, only that they put a file at the beginning, also named "/ " and deprecated it. Now this misfeature can be used to identify the lib, but as it isn't needed for anything, it is just cruft...
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri Dec 30, 2005 12:40 pm    Post subject: Reply with quote

larsivi wrote:
So who the f*** gave MS the monopoly? When using the Ar format for PECOFF, they also broke it. Not that it is an big problem, but these small things annoy me.

Market-forces have little or no regard for technicalities, or intellectual cohesiveness Very Happy
Back to top
View user's profile Send private message
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Fri Dec 30, 2005 12:43 pm    Post subject: Reply with quote

kris wrote:
larsivi wrote:
So who the f*** gave MS the monopoly? When using the Ar format for PECOFF, they also broke it. Not that it is an big problem, but these small things annoy me.

Market-forces have little or no regard for technicalities, or intellectual cohesiveness Very Happy


Thanks for the clarification. I already had some suspicions Wink
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri Dec 30, 2005 1:07 pm    Post subject: Reply with quote

larsivi wrote:
kris wrote:
larsivi wrote:
So who the f*** gave MS the monopoly? When using the Ar format for PECOFF, they also broke it. Not that it is an big problem, but these small things annoy me.

Market-forces have little or no regard for technicalities, or intellectual cohesiveness Very Happy


Thanks for the clarification. I already had some suspicions Wink

Laughing
Back to top
View user's profile Send private message
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Sun Jan 01, 2006 6:36 am    Post subject: The next steps Reply with quote

Ok, I got much of this stuff collected now, but I wonder somewhat over how to organize the classes.

So far, the ArLoader creates a class called Archive that holds the logic for loading a file with the "!<arch>\n" magic string. But ArLoader wants to return a DynamicLibrary, so Archive is now a subclass of that, but the export stuff isn't implemented yet so currently my test bombs.

Anyway, currently the registry and DDLFile isn't optimally ready for taking a chunk of data from memory.

What i do, is read the data, and do this:

<code>
DDLFile ddlfile;
ddlfile.filename = fName;
ddlfile.buffer = cast(ubyte[])memberData;
DynamicLibrary dl = this.registry.load(ddlfile);
this.modules ~= dl.getModules();
</code>

A couple of things here:
* DDLFile could/should possibly have a load([]buffer),
* The registry.load call seems a bit kludgy as long as the return library really is a DynamicModule
* The above code don't currently grab the exports, but that isn't a problem, just mentioning it Smile

Other things to consider:
* I should probably call the class ArchiveLibrary or ArLibrary...
* As long as the archive contains a symboltable, we should use this for optimization. At least I think it might help us locate only the files in the library that is of relevance. It's just about finding a good algorithm. The problem as I see it (related to your lazy loading schemes), is that we then DON'T want to parse all of the lib now, just the symbol table (if the symbol table ain't present, which is quite possible with Ar the whole archive must be processed), and then parse the objects pointed to by the symbol table. As it is, I think we should think a bit on how we want this, just do the all encompassing loading for now (as it is more or less already there), then when we get some good use cases to test, we can start with some additional profiling/lazying/etc.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Jan 01, 2006 1:23 pm    Post subject: Re: The next steps Reply with quote

larsivi wrote:
Ok, I got much of this stuff collected now, but I wonder somewhat over how to organize the classes.

So far, the ArLoader creates a class called Archive that holds the logic for loading a file with the "!<arch>\n" magic string. But ArLoader wants to return a DynamicLibrary, so Archive is now a subclass of that, but the export stuff isn't implemented yet so currently my test bombs.

Anyway, currently the registry and DDLFile isn't optimally ready for taking a chunk of data from memory.


This is excellent news.

Quote:

A couple of things here:
* DDLFile could/should possibly have a load([]buffer),


No problem. I just added load(ubyte[] buffer) to my code so it'll go in with the next changeset.

Quote:

* The registry.load call seems a bit kludgy as long as the return library really is a DynamicModule


I totally understand the kludge factor there, as its bothered me too a little bit. It could be enhanced by two sets of loaders (one for libs and one for modules), but that might make things needlessly complicated. I've looked into unifying DynamicLibrary and DynamicModule, but there's no way to do it that won't make the Linker jump through hoops.

If it makes you feel any better, consider what gyrations the .zip loader will have to do. Shocked

Quote:

* The above code don't currently grab the exports, but that isn't a problem, just mentioning it Smile


For now, I guess you can just create an export dictionary like I have with the OMFLibrary. D's hashmaps seem to work pretty well performance wise.

Quote:

Other things to consider:
* I should probably call the class ArchiveLibrary or ArLibrary...


ArchiveLibrary seems best to me. Smile

Quote:

* As long as the archive contains a symboltable, we should use this for optimization. At least I think it might help us locate only the files in the library that is of relevance. It's just about finding a good algorithm. The problem as I see it (related to your lazy loading schemes), is that we then DON'T want to parse all of the lib now, just the symbol table (if the symbol table ain't present, which is quite possible with Ar the whole archive must be processed), and then parse the objects pointed to by the symbol table. As it is, I think we should think a bit on how we want this, just do the all encompassing loading for now (as it is more or less already there), then when we get some good use cases to test, we can start with some additional profiling/lazying/etc.


I'm with you 100? on this. These are some things I plan on making the OMFLibrary class do as well, at some point...

One architecture change that I'm putting into place, that will help all this, is the addition of these two methods:
Code:
public DynamicModule getModuleForExport(char[] name);
public ubyte[] getResource(char[] name);


Obviously, getResource() isn't useful to ArchiveLibrary. However, it can be mapped to raw reads of parts from the archive if you want. Wink

The former, getModuleForExport() is used almost exclusively by the linker -- that way, you don't need everything loaded until getModules() or getExports() is called. This will facilitate several lazy-parsing/loading scenarios as well as some lookup schemes. For example, cracking the symbol name down into a namespace (when possible) can help with looking up the correct module.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries All times are GMT - 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group