FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Live feed

 
Post new topic   Reply to topic     Forum Index -> FragBots
View previous topic :: View next topic  
Author Message
h3r3tic



Joined: 30 Mar 2004
Posts: 261
Location: Torun, Poland

PostPosted: Sat Oct 29, 2005 12:45 pm    Post subject: Live feed Reply with quote

This project may seem dead to some, but don't worry - it's not. I'm putting as much time in its development as I can afford. The lack of an internet connection at home makes me work offline and not interact too much with the D community. My D newsgroup posts have almost disappeared.
But I'm being motivated by pragma's constant progress updates on the DDL/DSP forums and I think I'll be following his way to some extent.
Despite the fact that I'm still working on the rendering and scene management parts of the engine, it's quickly getting more and more complex. It won't be long before it gets more complex than Mango, which officially is the largest D project. On the other hand, anyone who has ever peeked at Mango's code will admit that it's very nicely structured and documented. My project completely lacks documentation and in some parts of it, the structure is far from perfect.
I hope that by exposing some of the engine's design here in plain English, I'll develop a documentation habit and maybe some dudes reading these forums will notice potential flaws in the design and hopefully let me know Wink /+ If that doesn't work then at least my writing skills will have a chance to improve Razz +/


I'd like to start by admitting something that may scare some of you to death. Not until recently did I realize how important careful design is. Not too long ago I used to just come up with an idea, think about it for a second and jump into code. The funny thing is that this approach seemed to work at first. But I realized it might've been the reason why I had 3 rewrites of my scene graph system in the previous engine and about 4 rewrites of a tool called a 'portalizer'. Eventually they worked, but at the expense of implementing the same functionality over and over again.
Currently as I approach coding something new for the engine, I leave my PC alone and the only really active process on it is Winamp. I grab a piece of paper and draw some UML-alike diagrams representing what I want to code + some descriptions, even questions to myself regarding the design.
Recently this approach has proved to be very effective in the development of my plugin system. The final structure was far superior than the very first design.
I'm probably saying things absolutely obvious here, but I'm curious if any of you had similar experiences with learning how to develop more complex applications.


Having experienced the design-shock after reading a bit about UML and proper engine design on gamedev's forums I decided to cut the engine into pieces. /+ I was willing to release a demo, but that will be when I'm done with the refactoring. +/ I decided I need an engine that's independent of the graphics API. Don't panic yet, I don't mean I'll implement OpenGL, D3D, Software, Glide, MentalRay and other exotic renderers. I'm currently sticking to OpenGL, but I don't want the engine to be completely dependent on just this API. When in some time a Foobar company releases a new graphics api for its GPU monster, I don't want the engine to be lagging behind just because it's using OpenGL. Switching renderers should be relatively easy. Thus I've developed a plugin system that can provide me all the necessary abstraction and dynamic loading capabilities. But before diving into it's design, I'd like to highlight the main components of my engine:


The 'Engine' class is the core of the system although it's functionality is minimal:
- registering, configuring, initializing, updating and destroying EngineSubsystem's
- executing and scheduling IJob's
- spawning Process'es
- registering MessageHandler's and sending Message's


The engine is currently one of two global objects in the program. It's initialized by a function that let's users get a reference to its only instance, the engine() function.


The 'GlobalObjectDatabase' (GOD) is the second global object. It serves as a registry for any object. These can be organized in folders and are accessed by textual identifiers. All engine subsystems are registered automatically in the GOD.
To get a reference to the RenderingSubsystem, one can write:
god().get!(RenderingSubsystem)("engine/RenderingSubsystem");
There's also a matching non-templated set function.
The GOD system works across DLL boundaries, effectively overcoming the cast problem through the usage of a dllCast template which compares classnames.
One might argue that it's not very efficient to access objects this way, but that operation doesn't have to happen too often. Objects that will be accessed thousands of times per second won't reside in the database. They are located in specialized structures, like the scene graph.
The GOD approach has one huge advantage to using normal references. When a plugin is loaded into the system, it's given only to objects that need to be initialized in its modules: the engine and the GOD. After doing that (behind the scenes), the plugin writer may override any more significant object in the engine by just setting the appropriate object in the global database. If these were not accessed from the GOD, a plugin would have to do a lot of clumsy work to achieve that functionality.


The next component to be described is naturally the PluginSubsystem.
In order to achieve maximum flexibility, a user-defined plugin implements the IPlugin interface whose only operation is to return an array of IPluginComponent's that define this plugin. Each DLL exports just one IPlugin but may define multiple IPluginComponent's, so writing plugins is both simple and flexible. Each plugin component must provide its name and a list of dependencies. More on that later.
At the initialization time of the PluginSubsystem, a list of plugins to load is obtained (e.g. by the PluginSubsystem by recursively traversing a given directory and looking for files with the proper extension (currently .hp - Heresy Plugin)). Then all plugins are pre-initialized - objects are created and dependency lists are obtained. Next, the plugin loader performs a topological sort of the plugin components, loading them in such an order that before a component is initialized, all of its dependencies must already be initialized. For a plugin component to be initialized, all of its dependencies must already be initialized.

Example:
* The OpenGL Renderer plugin defines two components:
- GLRendererComponent - "opengl.renderer" - defines basic OpenGL support in some core structures - no dependencies
- GLExtensionLoader - "opengl.extensions" - loads all OpenGL extension available and initializes vertex caching (VBO or Simple) - dependent on "opengl.window" because a rendering context is needed for extensions to be loaded
* The OpenGL Window plugin defines one component:
- GLWindowComponent - "opengl.window" - creates an OpenGL window with a rendering context; requires that some internal structures are initialized - dependent on "opengl.renderer"

The components would be initialized in the order: GLRendererComponent, GLWindowComponent, GLExtensionLoader

OpenGL is currently loaded dynamically from 4 plugins: opengl.window, opengl.renderer, opengl.cg (NVidia's Cg language) and opengl.texture (OpenIL - based texture loader)

More later Razz
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
clayasaurus



Joined: 21 May 2004
Posts: 857

PostPosted: Sat Oct 29, 2005 8:56 pm    Post subject: Re: Live feed Reply with quote

My first attempt at a 3D engine was like jumping off a plane with no parachute. I've also had to rewrite several times because of my 'bad design.' However, if I didn't try it out, I wouldn't know what bad design was in the first place. The best thing you can do before coding is to write a document outlining what you think you'll code (rough draft), then rehash it to handle all possible situations in your head. When you actually start coding things, you realize there are things you've forgotten about and then you have to go back and check your design again, repeat, until you get something that works.

What I've done in my own projects is make each piece of code as reusable as possible. I have input , window, font, sprite, etc. that can be used for any game. I seperate my game and engine code, game code you can't re-use, but engine code should be portable from game to game.

I've also learned to use D modules as namespaces + as something I would like to call a 'module class,' kinda like a singleton but not really. What I do in a module is declare a bunch of variables and functions using the module itself as the class. So I can call...

import core.input;

core.input.open();

while (!core.input.keyDown(SDL_QUIT)) { }

This nicely solves the problems of having to use 'god/engine classes' or passing a class to every single thing you need. All variables are protected by the module 'namespace' and the module becomes like a global singleton. So, instead of having every variable you need in the engine class, you can put them in seperate portable modules, and have the modules import each other as needed. It is a design that have been clean and allowed any part of my engine/game to interact with itself as needed without using any explicit code for interaction. My level module can add scrap to my player module by just using

import game.player;

game.player.addScrap();

The true test of code is reusability and readability. If you can easily use your code in another game (or others can easily use your code), then you have succeeded. For a true inspiration in design, look at the Mango trunk. The mango code looks so readable/understandable that it is hard for me to tell if it is actually doing anything.
Back to top
View user's profile Send private message AIM Address
h3r3tic



Joined: 30 Mar 2004
Posts: 261
Location: Torun, Poland

PostPosted: Wed Nov 02, 2005 1:00 am    Post subject: Re: Live feed Reply with quote

clayasaurus wrote:
The best thing you can do before coding is to write a document outlining what you think you'll code (rough draft), then rehash it to handle all possible situations in your head. When you actually start coding things, you realize there are things you've forgotten about and then you have to go back and check your design again, repeat, until you get something that works.


Well, that's called iterative software engineering and surely is one of the most frequently used approaches to programming.Not always the 'best thing' but usually it works Smile

Quote:
What I've done in my own projects is make each piece of code as reusable as possible. I have input , window, font, sprite, etc. that can be used for any game. I seperate my game and engine code, game code you can't re-use, but engine code should be portable from game to game.


I guess that's implicit when coding an *engine*. Generally engines should be reusable. Games don't have to.

Quote:
I've also learned to use D modules as namespaces + as something I would like to call a 'module class,' kinda like a singleton but not really. What I do in a module is declare a bunch of variables and functions using the module itself as the class.


But.... what you get is a bunch of globals...

Quote:
This nicely solves the problems of having to use 'god/engine classes' or passing a class to every single thing you need.


It doesn't. If you wanted to access them from a DLL, you'd first have to iniitalize them in the DLL to the same values the executable uses.

Quote:
For a true inspiration in design, look at the Mango trunk. The mango code looks so readable/understandable that it is hard for me to tell if it is actually doing anything.


Sure Mango looks nice, but I don't think I should compare my project to it. Mango's sources look so much nicer than mine because like 60? of Mango's code are comments. 3d engines are more complex than web servers so I actually have to do better than Mango Twisted Evil
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
clayasaurus



Joined: 21 May 2004
Posts: 857

PostPosted: Thu Nov 03, 2005 12:56 am    Post subject: Re: Live feed Reply with quote

Quote:

I guess that's implicit when coding an *engine*. Generally engines should be reusable. Games don't have to.


That is what I said, no? I'll repeat, game code is non portable, engine code is portable, so they should be seperated. The last thing you want is game code leaking into your engine. Also, you should be able to re-use your engine for any suitable project, who wants to recode when it's avoidable.

Quote:

But.... what you get is a bunch of globals...


True, but these globals can be private/public and are protected by the modules. I thought the problem with globals in C++ were name conflicts.

Quote:

It doesn't. If you wanted to access them from a DLL, you'd first have to iniitalize them in the DLL to the same values the executable uses.


Well then you can tell I havn't done much work with dll's.

Quote:

Sure Mango looks nice, but I don't think I should compare my project to it. Mango's sources look so much nicer than mine because like 60? of Mango's code are comments. 3d engines are more complex than web servers so I actually have to do better than Mango Twisted Evil


Code beauty and comments will save you and everyone you are working with time and energy. Mango is code beauty and comments to the extreme, therefore it is a worthy source of inspiration if that is what you want to achieve.

It seems like you are saying that in order to do better than Mango you need to forsake code beauty and comments. I advise you to keep your code as clean as possible and have as many comments as possible, it will pay off.
Back to top
View user's profile Send private message AIM Address
h3r3tic



Joined: 30 Mar 2004
Posts: 261
Location: Torun, Poland

PostPosted: Thu Nov 03, 2005 1:38 am    Post subject: Re: Live feed Reply with quote

clayasaurus wrote:
That is what I said, no? I'll repeat, game code is non portable, engine code is portable, so they should be seperated. The last thing you want is game code leaking into your engine. Also, you should be able to re-use your engine for any suitable project, who wants to recode when it's avoidable.


Yup... I was not arguing with you here, merely agreeing with you. It's obvious to anyone who's into engine programming.


Quote:
True, but these globals can be private/public and are protected by the modules. I thought the problem with globals in C++ were name conflicts.


It might be, but in my case using globals would be the *worst* approach. I want to be able to initialize all object references in a DLL/SO to the values from the main program. Using your method, I'd have to initialize every single global variable Rolling Eyes I introduced the "GlobalObjectDatabase" approach to just initialize a single reference instead of hundreds Smile

Quote:
Code beauty and comments will save you and everyone you are working with time and energy. Mango is code beauty and comments to the extreme, therefore it is a worthy source of inspiration if that is what you want to achieve.

It seems like you are saying that in order to do better than Mango you need to forsake code beauty and comments. I advise you to keep your code as clean as possible and have as many comments as possible, it will pay off.


I never said that. After I finish my current refactoring process, I'm looking forward to writing some documentation and commenting the more obscure parts of my code.
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
h3r3tic



Joined: 30 Mar 2004
Posts: 261
Location: Torun, Poland

PostPosted: Wed Nov 09, 2005 5:20 am    Post subject: Reply with quote

Shader - centered design:

The most obvious approach to creating a renderer is to make it geometry-centered. That was the first method I tried as well. You have the geometric data arranged in some forms of spatial structures, scene graphcs, etc. Then you associate shaders with these. However this methodology has some potential flaws:
- the geometrical data is completely decoupled from the method used to render it, vertex and fragment programs
- in addition to having a flexible shader system, you must have a flexible geometry-data system
- one shader can use only one geometric object

Thus, I'm trying a different path. In my current design, shaders are put in the middle of the design. Note that I don't relate the 'shader' term to GPU programs. A shader in my understanding is how YannL defines it. It's a complete description of how an object should be rendered. Shaders can geometrically model objects like
- simple meshes
- space warps (special portals with masking and view transforms)
- particle systems
- terrain systems (with LoD, etc)
plus, they can represent
- fragment and vertex programs
- effects like transparency (they may define transparency sorting)
- much more

How do geometric objects relate to shaders ? They are merely inputs to these, just like Textures or any other data a shader might need to *define an object*. A triangle mesh without a shader is not something that can be rendered. A shader might take a mesh and provide skinning for it, either using software or hardware transforms. It may take a heightmap as an input and create terrain rendering algorithms as simple as a brute-force approach. It might also create a complex curve-based terrain with fractal details.

The key concept with shaders is that they are defined independently to the normal rendering pipeline. A game might be released that uses Engine version 1.0 but after 2 years, still new rendering effects and techniques could be added to the game by just installing a single DLL/SO.

Furthermore, I want the pipeline to be absolutely configurable. If I wanted to add bloom overbrightening effects at one point, I just have to create a single class and plug it into the pipeline. If I want to have that 3d-display with the aid of red-and-blue colored glasses, no problem, I add a few objects to the pipeline and I've got it.

Currently, an internal component of the engine is a scene graph. It might at some point become just a plugin as well, but for the time being it's built in. Scene graph nodes provide a high-level access to the scene. The scene graph is composed of subclasses of the SgNode class. One of them is the Geode class. A geode is something that will probably be rendered. It may contain a list of Renderable objects.
Each Renderable contains a Shader, GeomChunks and InstanceShaderData.

InstanceShaderData may contain data created by the Shader for the given Renderable (e.g. VertexCache's).

GeomChunks are the geometrical inputs to the shader. They may come in a variety of subclasses, such as TriMesh, Bitmap /+ not sure about the name +/, Surface, ParticleSystem, etc

A Shader is what really defines how will the object be rendered. It may create arbitary new stages in the pipeline. It may also contain other shaders. E.g. one could define a 'transparency' shader that could wrap any other shader and make the geometry it produces transparent, sort it, etc.

The rendering pipeline is composed of RenderStage's. Each has an assigned priority which defines when will the stage be processed in a given frame. Priorities need not be unique.
Each RenderStage is a graph (a DAG actually) in which nodes represent pixel buffers and buffer operations. Each node defines inputs and outputs. They can be the framebuffer or pixel buffers. Special Nodes, Operators perform operations on these buffers, e.g. clearing, drawing to them, reading to a texture, etc.
When a RenderStage's function exec() is called, a topological sort is conducted, starting from buffer Sink's. Nodes are then processed in this order, implementing the operation a given graph describes. Pixel buffers are reference counted so they can be reused without a problem.


Rendering:

For the scene to be rendered, a scene graph is needed. It must contain some Geodes and a Camera node. The rendering is started by traversing the scene using a special visitor, a RenderVisitor. It creates a list of all Renderables visible from a given point of view along with their transforms relative to the viewer and frustums they're visible from /+ frustums are clipped during the rendering +/. It's essential that each Renderable figures at most once on this list. When it's visible from two different frustums (e.g. a floor in a room with two doors next to each other), these frustums are combined together. For this purpose, a MultiFrustum is used. This is essential for some rendering techniques and for performance reasons, but I won't go in too much detail now.
The list is then traversed linearly and a prepare() function is called on each Renderable's shader. This function may create additional RenderStage's, then it fills VertexCache's and inserts them into specific render lists for each RenderStage.
Finally, rendering iterates thru RenderStage's and calls exec() on each.

A Shader may define its own subclasses of RenderStage, thus allowing effects like recursive rendering of the scene for reflections, space warps, refractions, etc.
It may also define subclasses of the DrawOp pipeline node, which receives data like VertexCaches and may e.g. sort them by distance (for the transparency shader)


An important point is that shaders are referenced by Effects from the 'effects' file. An Effect is a high-level description of an object. It may e.g. define 'a bumpmapped diffusive surface'. Shaders are the worker-bees. They try to render what an Effect describes. Some may do it better than others, some may work only on specific hardware. Thus at load-time, optimal Effect-Shader configurations are found so that a given scene can be rendered at the best quality that's available on a given system.

The architecture is largely inspired by Yann Lombard's posts on the gamedev site. Some of his posts are linked to on this page:
http://triplebuffer.devmaster.net/misc/yannl.php
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
h3r3tic



Joined: 30 Mar 2004
Posts: 261
Location: Torun, Poland

PostPosted: Thu Nov 24, 2005 12:50 pm    Post subject: Reply with quote

Rolling Eyes I've been quite busy recently due to my university and RealWorld(tm) stuff... I'll write more when I finish some coding I had been planning for some time now.
/+ In the meantime, I've read 'The Pragmatic Programmer' Very Happy - excellent book
Currently parsing 'Code Complete 2nd Ed' Wink +/
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
h3r3tic



Joined: 30 Mar 2004
Posts: 261
Location: Torun, Poland

PostPosted: Wed Mar 08, 2006 7:00 pm    Post subject: Reply with quote

This 'live feed' has been quite dead. I blame everything on the RealWorld(TM) and my univ, but I still feel guilty for not writing anything in such a long time Embarassed

Anyways, I'm currently designing a simple language to define Effects that would allow a graph flow evaluation for any Effect/Shader in the engine. It will probably be enhanced by a graphical tool later (like here: http://www.projectoffset.com/images/shaderbuilder_large.jpg). So far the concept looks like this:

Code:
// declare subeffects
a : MeshEffect
b : BezierSmoothEffect
c : DepthSortEffect
d : CgEffect

// define data flow
a.out.mesh -> b.in.mesh
b.out.mesh -> c.in.mesh
c.out.mesh -> d.in.mesh

// connect inputs and outputs. works like alias in D
in   <- a.in
out   <- d.out

// set parameters
b.subdivs = 5


another example:

Code:
// very simple heightmap mesh effect
in.height   : Texture
out.mesh   : Mesh


With such definitions and a special effects file, it will be possible to create shaders that implement any of these effects and may be freely mixed.

Oh, and btw, I think I'll be trading the render-stage approach for something better. More on that later.
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> FragBots All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group