FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Ideas Wanted
Goto page 1, 2  Next
 
Post new topic   Reply to topic     Forum Index -> Titan
View previous topic :: View next topic  
Author Message
trevorparscal



Joined: 25 May 2005
Posts: 73
Location: Bay Area, CA (USA)

PostPosted: Thu Oct 13, 2005 6:16 pm    Post subject: Ideas Wanted Reply with quote

Welcome to the Titan project.

The project is a kernel written entirely in D. The designs and features for this kernel are still in planning, so the first thing you can do to help out, is to put in your 2 cents, and make suggestions for how the kernel should be written...

Some initial goals for the kernel include...

- Modular Design (perhaps an exokernel)
- Support multiple executable formats (OMF, COFF, ELF, PE, BIN)
- Support Modern Hardware Only

Things that should be done at the Operating System Level include...

- Various Garbage Collection Systems (copy, generational, incrimental, and perhaps a hybrid gc too)
- Transparent Memory (Orthogonal Transparency)
- Transparent Data (local and network resources act the same)

The target for this kernel, is for two or more operating systems to be written for it. The first being a User OS (Apollo), and the other being a Server OS (Zeus). But for now, we need to focus on a kernel...

Two main things to think about in all features...

Is it high-performance?
Is it innovative / intuitive

If yes, than it's worth thinking about, so post it here!

Thanks,
Trevor Parscal
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
trevorparscal



Joined: 25 May 2005
Posts: 73
Location: Bay Area, CA (USA)

PostPosted: Mon Oct 17, 2005 10:10 am    Post subject: File Format Reply with quote

I got this email from a veteran programmer last night... Wonderful ideas - read up!
- Trevor


Mr. Parscal:

I saw your post on DigitalMars about Titan.

I would like to help, if possible.

I am an old time programmer, with about thirteen years development experience, total.

Languages: C, C++, IBM System 360/390 Assembler, PDP-11 Assembler, Other dippy languages too numerous to mention.

I have some questions about your current goals.

Is Titan intended to work in the current Unix-like environment? In other words, if the user recompiles his Linux/BSD/What-have-you source code for Titan, will it run? Titan does not have to, or have to not, do this. I really don't care. Just curious. Might make adapting current Linux and BSD drivers and code a whole lot easier.

If you are really wild and wooly, and not necessarily following the GNU/Unix model, please let me make some unusual suggestions.

1. Re-examine the concept of "files". Make a file more than just raw data. Make the file consist of several sections, and do a much better job of data descripton. Somebody recommended something like this to me on a website somewhere, long, long ago.

I don't much like Microsoft, but they have a fairly new file type out that looks interesting: adtg files. (adtg = Advanced Data Table Gram). Each adtg file is really a whole collection of stuff, being a table description and the table itself, both in the same file. This is the same idea.

Right now, a "file" is usually considered to be a string of bits, starting at location "A" and ending at location "Z". Ask for the file, and it is fed to you, a character at a time.

I think this could be handled better. New approach: a file in the system consists of several parts. All files therefore have the same type: they are files, period.

The ME section keeps demographic information: file name, generation information (daily jobs create daily generations: which one is this one?), hash code (verification that the complete file is complete), date of creation (independent of system date for file). Other info might be appropriate here, as well, like intended target use (e.g. this is a .doc type of file), or number of records (if organised as records).

The COMMENTS section is pure text. This allows the author to add text commentary about the contents of this file. I find that I am always trying to figure out how to keep corresponding comments linked with files. Either I have too much, or not enough, or the file and comment are disconnected, and I cannot put them back together without looking at everything. With this file format, it is trivially easy for creating programs to add identifying information, and the information will not be easily lost.

XDD section: XML data definition, in SQL style. The data that comes later needs a description. Structures, field names, data formats, everything necessary for any general program to come along and use the information. This way, everything can be stored, and stored efficiently. Individual programs will not need to know anything about the data, just that within this file, there are records containing the field called "MyField", or "Morris", or whatever. If the program needs numerical information, and the field named "MyField" contains only the text "LeGarbage", then there is an error. Otherwise, there can be any number of different formats, and the OS will handle the conversions. Binary, Hex, and text digits can all be used in different files, and the program itself will never know the difference, nor care. If you can rationally describe it in XML, you can use it.

DATA section: the data itself, in the format described in the DD section.

Length test and hash check every section. This was unacceptably expensive on the PDP-11. These days it is unacceptably sloppy not to check.

2. DOOM as a GUI paradigm. Why just use a mouse? This is kind of like the two-index-finger approach to typing: there are better ways. Who needs MSWindows when you have DOOM?

Besides, using the BFG9000 to open a document just makes the day go better!

3. Kernel work will require Assembler. Anything you can do to implement some kind of Object Oriented Assembler will save you, and every Kernel developer, a ton of time. Anything that says "you can't branch here, this is a code block" will help. Anything that requires methods for particular data types will help. A small example: when is a number not a number? When it is a Social Security Number. 266-19-6363 is not a number. You may not add to it, multiply it, divide it, or take the average of two SSNs. It is a pure text identifier, but composed solely of digits. "C" will let you commit all sorts of sins, in the name of efficiency. This precise issue caused no end of problems to one of my former employers on a major telecommunications project. So define the thing as a special type, consisting of nine characters, all digits. In fact, I rather suspect that every single field should have its' own type. Are they all text strings? To be sure, Yes, they are. However, tight restrictions on when and where conversion may occur, and the precise conditions, will save you a ton of time in the long run. I prefer debugging compilation errors at my desk to debugging logical problems on the user's site, every time.

Please feel free to use or post the contents of this message anywhere, with or without attribution.

Andy Walker
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
trevorparscal



Joined: 25 May 2005
Posts: 73
Location: Bay Area, CA (USA)

PostPosted: Mon Oct 17, 2005 10:32 am    Post subject: Reply with quote

1. I love your file idea.. I think there is allot we can do to change the way users interact with information, and enhancing meta-data is certainly a big one. Another thing is that file systems are a single structured system.. There is only one way that all the files relate to each other.. What if you could have paralell structures? What if you could browse your files by date of creation, modification, etc. Not just searching, browsing. This kind of paralell structure by meta-data would be a big help in many circumstances. Also, the ability to have related files/folders anywhere, so if I were working with a music project file for instance, when I mixed the project down to an audio file, the project file's meta information should now include a relation to the mixdown, and if an os wanted to read those relations, it could easily get a list, and than a user could easily access the related files.. And the other direction, I am looking at the audio file, and I can now easily open the project from there...

2. New human user interface devices are going to be a very important step for a new gui to suceed.. Hence the keybord (command line) or the mouse (the 2.5 gui like xerox/macos/windows)... Now, what kind of device that is, how easy it is to use, how much it costs to make... These are factors we need to evaluate, and perhaps invent something, perhaps adapt something already invented.

3. Object orientation in ASM is currently beyond me, but I hope to learn. You have some ideas nonetheless, and I think supporting D features at a very low level would really be awesome, as we would end up making D itself run faster.

Thanks,
Trevor Parscal
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Oct 17, 2005 11:38 am    Post subject: Reply with quote

Some thoughts.

I've done some very light reading on the topic of OS design and how to get a project rolling. That having been said, please take these suggestions with a grain of salt. Smile

From what I've seen, the most sane approach is to start off with an ELF-formatted binary and a compatible bootloader like GRUB. This gets you mostly out of the way of having to mess with most of the low-level nastiness that is booting on a PC. This also feeds directly into the potential to use that same binary file directly, as a hosted OS.

All-in-all, this approach should make development easier. Especially since GRUB seems to be so well documented and favored by OS-hackers.

Beyond that, the only thing I can suggest is that the root-most kernel should provide the absolute minimum to get the core D-language support up and running (which is roughly where you're heading anyway with putting the GC first). I'd reccomend tearing apart Ares to see what that's about, as thats about as minimal as D gets.

Then, your only remaining hurdle is to replace all of the c-language and os-level hooks in order to make this happen. You'd then have a platform where the remainder of development can proceed in pure D. Every 'exokernel' concievable would merely be a set of D modules: what could be better?
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
AndyW



Joined: 17 Oct 2005
Posts: 9
Location: St. Louis, MO

PostPosted: Mon Oct 17, 2005 4:05 pm    Post subject: Boot loader and other things Reply with quote

I have to agree with Eric.

There may be reasons to reinvent the boot loader, but offhand, I cannot think of any that would justify the effort.

Grab GRUB, conform to its' requirements, and forget about it. GRUB will load your kernel. It just doesn't care what is in it.

I used to have a copy of NASM .... somewhere. Been a while. Might be on my development machine, the one I have not turned on in the last two years.

I also recall a High-Level Assembler out there, again, "somewhere". I will take a look-see.

Please consider baby-steps: do something, get it to work correcty, go to the next step. IIRC, Linus Torvalds started with Minix, and sequentially replaced functions until it was Minix no more. Overall, pretty successfull as a strategy, IMHO.

AndyW.
Back to top
View user's profile Send private message
trevorparscal



Joined: 25 May 2005
Posts: 73
Location: Bay Area, CA (USA)

PostPosted: Mon Oct 17, 2005 6:20 pm    Post subject: Reply with quote

I will agree - Grub is the way to go for now - so, I am going to work on getting something that grub can boot, and install grub on a vmachine to test it with...

Baby steps - this I agree with.. However, aside from some very basic kernel concepts that are in D so far, there isn't an existing kernel that I can replace bit by bit, so.. Its kind of a ground up effort.

Thanks for the input - its such a big project, sometimes its hard to see the program through the code Smile

Thanks,
Trevor Parscal
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
AndyW



Joined: 17 Oct 2005
Posts: 9
Location: St. Louis, MO

PostPosted: Mon Oct 17, 2005 9:56 pm    Post subject: Negativism Reply with quote

So far, I feel like I am being pretty negative here.

This should be fun for you, most importantly.

Is it big? Yup. At least you are not such a fool that you missed that part. Most of the time I try not to be a fool. Often, I succeed.

Start with a list. What do you want it to do? What do you want it not to do? This is the developer's lament: I can write the code to do anything the customer wants done, but they won't tell me what they want it to do!

Start with a short, imperfect list and add to it. A poor list beats no list, every time. Besides, all too soon, the short list will become a monster that can no longer be fed, because these kinds of projects catch fire, then everybody wants it and wants all kinds of things with it. Walter Bright and "D" are the perfect example.

A kernel is a program. It performs specific tasks. It is the resource allocator that really should be hard-wired as part of the machine. No one does this because it is just too complicated, and a bug fix would mean designing a new machine.

Get creative. Email Linus Torvalds, or Tso, or any of the Linux lieutenants. Do your due diligence research before the email, then ask them what they wished they had done differently. Torvalds might not answer, but one of the others will. That insight will be pure gold.

"Due diligence" research is being responsible enough to go to the Linux websites and spend a few hours browsing messages. Look for the organizational messages, the flamewars, the unpleasantness. You do not have to memorize the entire Linux kernel, but should be well founded on some of the things that took up time.

This is respect for them and their time, and they will respond in kind.

AndyW
Back to top
View user's profile Send private message
AndyW



Joined: 17 Oct 2005
Posts: 9
Location: St. Louis, MO

PostPosted: Mon Oct 17, 2005 10:22 pm    Post subject: Ratio of Assembler to higher-level code Reply with quote

Um.

Trevor, from your comments, I am not sure you are aware of some of the history. I am probably mistaken, so if I am, please ignore this.

Ritchie and Kernighan came up with the "C" language so that they could write Unix in it. Unix preceeded "C". They did not write Unix in Assembler because writing it in "C" made it much more portable. This is ancient history, from 1972. The same logic applies here. You asked if I know x86 Assembler, as well as PDP-11 Assembler. The answer is yes, not super ggod but yes, and your question is a strong reason why you do not want to write most of the kernel in assembler. Every time you go to a new machine, you will have to find a new team of assembler programmers.

You are better off writing the kernel in a language at a level above Assembler. Going back to Linus Torvalds: one of his first tasks was writing a "C" compiler, for a truly portable O/S, like Unix. This is where Stallman came in. Stallman had a "free" C compiler. Real clunky, to be sure, but it mostly worked. The world owes Stallman a huge debt of gratitude for this single contribution, alone.

"D" is just a dandy language. It will solve problems for you that you didn't even know you had. Not every problem, to be sure, but many of them.

"D" is pretty well optimised. This means, on average, the compiler will do a better job of creating optimum Assembler statements than you can. Once in a while, this will not be true, but for the most part, the compiler is better at it.

D is more fun than assembler, certainly more reliable. Write your kernel in D. Once in a while, this will not be enough. Certain blocks of code will be used time and time again. This is where a little "hand optimization" can make a big difference. IMHO, what really happens is that a lot of random attempts at machine optimization are made, and the successful ones are kept! Doesn't make it sound like such an art, does it?

So, as I asked above: what do you really want?

AndyW.
Back to top
View user's profile Send private message
trevorparscal



Joined: 25 May 2005
Posts: 73
Location: Bay Area, CA (USA)

PostPosted: Mon Oct 17, 2005 11:58 pm    Post subject: Reply with quote

  • I want a kernel that doesnt suck.
  • I want a foundation for a different kind of operating system. Not different just to be different. Just, from a new perspective.
  • I want an operating system from which applications that will make things possible that were not before may be written with ease.
  • I want to create applications that make things possible that were never possible before, easy or not...
  • I want to do this all, in D, because it's smarter, faster, and neater.


As far as the history - I think you might have filled some gaps, but I do love to read (well, so long as I am reading in a web-browser) and am farmiliar with much of the beginings of C, and Linux.. Not as much about Unix.

I think you are right about D's compiler being better at assembler in most cases than I will ever be. And also correct that certain things might be best done in assmbler, but can be written in D for now until optimization.

I guess as you might imagine, since this project is soooo big, even though I have been dreaming it up for years, I am still not really ready to just sit down and write it. I have allot of learning to do of my own... From documentation online, and from guys like you, and segfault.

I appriciate your advice, insight, and generosity of your time in helping me where you can. I know that I am better off getting advice from veteran programmers and doing it myself than trying to sub it all out...

But if I ever need and ASM question answered I know who to ask... (you or a book) And thats the most important part of learning - knowing where to get the answers..

Thanks,
Trevor Parscal
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
mbutscher



Joined: 29 Apr 2005
Posts: 5
Location: Hof, Germany

PostPosted: Mon Oct 24, 2005 1:26 pm    Post subject: Reply with quote

I have no experience with kernel programming, so it may be a naive question if I'm wondering how the boot loader should find the kernel code and additional libraries.

These libraries (and maybe the kernel) are normally stored in a file system, but initially the kernel (if I understand correctly) does not contain code to handle the file system but uses a library to do that.

Michael
Back to top
View user's profile Send private message Send e-mail Visit poster's website
trevorparscal



Joined: 25 May 2005
Posts: 73
Location: Bay Area, CA (USA)

PostPosted: Tue Oct 25, 2005 2:05 am    Post subject: Reply with quote

I think the bootloader has to understand the filesystem... Anybody know how this works?

Still researching just that.
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Oct 25, 2005 8:35 am    Post subject: Reply with quote

As for how bootloaders load up a filesystem, I found this page which seems to be quite informative: http://my.execpc.com/~geezer/osd/boot/

Just scroll down to "BIOS role in bootstrap" and read from there. Wink

Another tip, I just saw this on hackaday.com: http://www.hackaday.com/entry/1234000153064739/

This might help you out if you're only using one computer and don't want to fumble with floppy disks all afternoon. Basically, you can utilize the free 'vmware player' software, and outfit it with whatever bootable image you want. Honestly, I don't think it gets any better than this.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Jaymz031602



Joined: 18 Aug 2004
Posts: 14

PostPosted: Mon Oct 31, 2005 2:47 pm    Post subject: Reply with quote

grub understands a limited subset of filesystems available for Linux. I believe it can understand at least EXT2 and EXT3 (being an extension of 2). So, your bootloader would have to have a dedicated /boot partition (in Linux-speak) formatted as EXT2 or 3 which would contain your kernel binary along with other bootstrapping files you'd need, like a initial ramdisk image, config files, etc.

Seeing how great and wonderful GRUB is compared to LILO, unfortunately I cannot run it on my own machine because I made a partitioning mistake long long ago on my machine, and therefore have no free space to create a /boot partition with.
_________________
"Yeah, well you're hogging all the ... UGLY!" - Chris (Family Guy)
Back to top
View user's profile Send private message Visit poster's website AIM Address
Jaymz031602



Joined: 18 Aug 2004
Posts: 14

PostPosted: Mon Oct 31, 2005 2:56 pm    Post subject: Reply with quote

regarding compiling a D kernel... it makes a BIG difference what your development platform will be for compiling the kernel, as the Win32 version of the DMD compiler won't produce an ELF, whereas the Linux version will.

However, you still have to take into account how you're going to link all your code together, dynamically/statically, which binary linkable format you'll use (COFF, OMF, ELF, PE to name a few).

There's also GDC to consider - it has the entire GCC backend system to play with, so much more flexibility in compiling/linking options, and can also move much more freely into the 64-bit world. I believe the DMD compiler is doomed to be stuck in 32-bit world unless some serious development work will be performed by Walter on his backend code generator.

Also consider how you're going to get a D compiler hosted on your future OS, when the time comes. It seems like that will be very difficult to do, as the DMD backend is still proprietary code. Going the other way with GDC support would mean having to support a subset of POSIX and/or GNU/Linux just to get GCC's systems up and running.

And please, don't forget about multithreading support within the OS. =P

I don't mean to discourage, but there are some serious speed-bumps in the future of this project that need to be ironed out. Just a friendly warning Smile
_________________
"Yeah, well you're hogging all the ... UGLY!" - Chris (Family Guy)
Back to top
View user's profile Send private message Visit poster's website AIM Address
Jaymz031602



Joined: 18 Aug 2004
Posts: 14

PostPosted: Mon Oct 31, 2005 3:09 pm    Post subject: Reply with quote

Will this be a multi-user system?

If so, ditch the whole root user thing... a single point of failure for a distributed system just seems so... dumb. Have potentially multiple accounts responsible for different tasks.

Also, make life easier for applications to install themselves. It should be "one click"/"just hit enter"/"just blast this door" simplicity =P. (that last one was from the DOOM GUI suggestion).

A great feature I've always pondered about at the OS/FS level:
Each application gets his/her own application unique ID before installing.
Tag this unique application ID on to the meta data for each file that the application will install.
...and now the kicker... all files that the application creates dynamically FOR ITS OWN PURPOSE, NOT THE USER'S will also be marked with this application ID.

For example, an audio editing tool should mark its own disk cache files with its application ID, but should NOT mark the user's project files (and associated audio files) with that application ID. This will allow uninstallation of the application to be done by querying the filesystem for files which have the application's ID in their meta data. This distinction of application-owned vs. user-owned files should be made clear and explicit in the filesystem API and should perhaps even be present in the "physical" filesystem itself. Perhaps applications should gain their own "user account ID"s and let the permissions system handle things?

This leads naturally to my next point: association of files with their handling applications. This should be done with a standardized table of globally-unique application IDs (which should be easily accessible from an online internet server), and the file's meta data should record that application ID of the application that the file should be handled with.

Note that this globally-unique application ID is NOT the same as the installer application ID mentioned earlier. That installer application ID is unique only to the machine in which the applications are installed, and should only be used by the installer/uninstaller suite of tools.

Anyway, I've always thought installer/uninstaller suites have always been riddled with such "how to cleanly uninstall?" problems since the dawn of time. I see, finally, an opportunity in which to learn from the mistakes of the past.
_________________
"Yeah, well you're hogging all the ... UGLY!" - Chris (Family Guy)
Back to top
View user's profile Send private message Visit poster's website AIM Address
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Titan All times are GMT - 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group