Join us at the Oolite Anniversary Party -- London, 7th July 2024, 1pm
More details in this thread.

RFC: new material model

General discussion for players of Oolite.

Moderators: another_commander, winston

Post Reply
User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

RFC: new material model

Post by JensAyton »

As I’ve mentioned, I’m rewriting the shader and texture code. This has involved introducing a concept of “materials”. These provide an increased degree of flexibility, which will be exposed through shipdata.plist (at least initially; moving on, I haven’t given up on adding a new, more powerful 3D model format).

First, texture names in DAT files have been redefined as material keys, which are used to look up a material definition in shipdata.plist. If no material definition is found, a material based on default parameters using the material key as a texture name is used – in other words, existing ships will work as normal. If you’re looking at the length of this post and going “ahhh!”, take consolation in the knowledge that all this stuff is optional, and you will probably not use most of it.

Material definitions will be looked for in two different dictionaries in a shipdata.plist entry: materials and shaders. The distinction is: if shaders are available and enabled, the shaders dictionary will be checked first, and the materials dictionary will be checked if no entry is found in shaders. If shaders are unavailable/disabled, only the materials dictionary will be checked. In other words, the shaders dictionary overrides the materials dictionary when shaders are in effect.

The format of entries is the same for both dictionaries, and is an extended form of the existing shaders dictionary. The keys in the dictionary are material keys as specified in the DAT model. The values are themselves dictionaries, for which the following keys are defined:
  • ambient (colour specifier): The ambient colour of the material; defaults to the same as diffuse.
  • diffuse (colour specifier): The diffuse colour of the material; defaults to white (1.0, 1.0, 1.0, 1.0).
  • diffuse_map (texture specifier): a single texture to be used when shaders are not in effect. If an empty string is specified, no texture is used. If no texture key is present, the material key will be used. Note, in particular, that all values in the textures array (below) are ignored.
  • emission (colour specifier): The emitted (glow) colour of the material; defaults to black (0.0, 0.0, 0.0, 1.0).
  • fragment_shader (string – file name in Shaders folder): name of GLSL fragment shader to use. Ignored if shaders are disabled.
  • glsl, glsl-vertex, glsl-fragment: deprecated, do not use.
  • shininess (integer): The “tightness” of specular reflect, ranging from 0 to 128. 0 means no specular highlight. Default TBA.
  • specular (colour specifier): The specular (shine) colour of the material. This is ignored if shininess is not 1 or more. Default TBA.
  • textures (array of texture specifiers): textures to use in shader program. Ignored if shaders are disabled.
  • uniforms (dictionary of uniform variable specifiers): uniform variables to use in shader program. Ignored if shaders are disabled.
  • vertex_shader (string – file name in Shaders folder): name of GLSL vertex shader to use. Ignored if shaders are disabled.
Types used in the above
Colour specifier:
Colour specifiers use the new extended form introduced for lasers in 1.69 (except for the minimum brightness restriction), namely any of the following:
  • Any of the following strings:
    • blackColor – 0, 0, 0, 1.0
    • darkGrayColor – 1/3, 1/3, 1/3, 1.0
    • lightGrayColor – 1/6, 1/6, 1/6, 1.0
    • whiteColor – 1, 1, 1, 1
    • grayColor – 1/2, 1/2, 1/2, 1
    • redColor – 1, 0, 0, 1
    • greenColor – 0, 1, 0, 1
    • blueColor – 0, 0, 1, 1
    • cyanColor – 0, 1, 1, 1
    • yellowColor – 1, 1, 0, 1
    • magentaColor – 1, 0, 1, 1
    • orangeColor – 1, 1/2, 0, 1
    • purpleColor – 1/2, 0, 1/2, 1
    • brownColor – 0.6, 0.4, 0.2, 1
    • clearColor – 0, 0, 0, 0
  • A string with red, green, blue and optionally alpha components, separated by single spaces. These may be in the range 0 to 1 or 0 to 255 (any value greater than 1 indicates the latter).
  • An array of red, green, blue and optionally alpha components with the same range options as above.
  • An HSB dictionary, with the following keys:
    • hue: hue angle in degrees. (Required; without this, it will be treated as an RGB dictionary.)
    • saturation: saturation in the range 0 to 1. (Optional, default 1.)
    • brightness: brightness in the range 0 to 1. (Optional, default 1.)
    • value: synonym for brightness.
    • alpha: opacity in the range 0 to 1. (Optional, default is 1.)
    • opacity: synonym for alpha.
  • An RGB dictionary, with the following keys:
    • red: red in the range 0 to 1. (Optional, default 0.)
    • green: green in the range 0 to 1. (Optional, default 0.)
    • blue: blue in the range 0 to 1. (Optional, default 0.)
    • alpha: opacity in the range 0 to 1. (Optional, default is 1.)
    • opacity: synonym for alpha.
Texture specifier:
A texture specifier may be either a string (specifying a file name in a Textures folder, except an empty string which indicates “no texture”), or a dictionary with the following keys:
  • anisotropy (real): The degree of anisotropic filtering to use. Anisotropic filtering is used to reduce blurriness of MIP mapped textures shown at an angle (and thus only applies to textures whose min_filter is mipmap or default). The anisotropy setting is multiplied by the maximum anisotropy level specified by the system, and anisotropic filtering is used only if the result is greater than 1 (For example, on a GeForce FX 5200 card, the maximum level is 8, so an anisotropy value of 0.5 means an effective anisotropic filtering strength of 4. The precise meaning of this number is hardware-dependant. Fun, eh?) A user-specified bias factor may be introduced, with a default of 1. Anisotropic filtering is disabled in Reduced Detail mode and is not available on all hardware. The default value is 0.5.
  • mag_filter (string): specifies how to scale up the texture when it is near the camera. One of the following strings:
    • nearest: use nearest-neighbour sampling, which produces a pixellated effect.
    • linear: use linear interpolation, which produces a blurry effect. This is what is currently used in Oolite.
  • min_filter (string): specifies how to scale down the texture when it is far from the camera. One of the following strings:
    • nearest: use nearest-neighbour sampling. Generally very ugly, but sometimes useful for shaders using textures for special purposes.
    • linear: use linear interpolation. This is what is currently used in Oolite. Also rather ugly for textures with significant contrast or details.
    • mipmap: use trilinear MIP map filtering to improve scaling quality. Generally also fast and least ugly, but uses 1/3 more memory, potentially slowing down systems with limited RAM or video memory. See below for an example of the effects of MIP mapping.
    • default: equivalent to mipmap, unless Reduced Detail mode is on, in which case it is equivalent to linear. Also equivalent to not specifying a min_filter setting. It is recommended that no explicit setting is made unless required for a shader, or for textures which display significant artefacts with linear filtering.
  • name (string): the file name from which to load a texture. Required, except in the case of diffuse_map where it defaults to the material key.
  • no_shrink (boolean): Indicates that the texture should not be shrunk except to fit hardware requirements. (See Texture rescaling below.) This should be used rarely, if at all. If it becomes abused, it will be disabled.
  • repeat_s (boolean): repeat textures in the s (or u) dimension, rather than clamping values beyond the range 0 to 1.
  • repeat_t (boolean): repeat textures in the t (or v) dimension, rather than clamping values beyond the range 0 to 1.
  • texture_LOD_bias (real): tweak factor for MIP mapping (which does not apply to other min_filter modes). MIP mapping involves a trade-off between scaling artefacts and blurriness; the LOD (level of detail) bias affects this. Positive values mean more blur, negative ones mean more artefacts. The default value is -0.25 (mostly so people won’t say “hey, why is everything blurrier in the new Oolite?”). Texture LOD bias is not available on all hardware.
Uniform variable specifier:
Uniform variable specifiers are used to provide information to shaders. They are specified in a dictionary, whose keys correspond to names of uniform variables declared in the shader source. These supplement the predefined uniforms specified in the Wiki.

A uniform specifier may be a number, or a dictionary with the following keys:
  • clamp (boolean): if true, the value will be clamped to the range 0 to 1.
  • type (string): the type of uniform variable. One of the following strings:
    • int: the uniform is an integer.
    • binding: the uniform is a dynamic binding to a property of the entity to which the shader applies. Each time it is rendered, the value will be fetched from the entity. (All the predefined uniforms except time and texN can be recreated with bindings.)
    • float: the uniform is a float.
    • texture: the uniform is an index into the textures array (starting from 0).
  • value: the value of the uniform variable. For uniforms of type int, texture or real, this should be a number. For uniforms of type binding, this should be a selector (a string). A list of useful selectors will be provided at a later date.
What MIP mapping can do for you:
Image
The left side of this test “ship” has min_filter = linear; the right side has min_filter = mipmap and texture_LOD_bias = 0. Anisotropic filtering does not come into effect as the textures are head-on to the camera. (The Gritty Coriolis OXP has a structurally similar texture map, and displays similar artefacts – you may remember a discussion of MIP mapping from that time, and in fact I had it partially implemented but never rolled it into the trunk.)

Texture rescaling
As before, textures will be resized so that their side lengths are powers of two, and so that they fit within the restrictions of hardware. However, additional scaling logic has been added: it may be possible for the user to specify a maximum size (the preference exists, but there is no user interface to set it yet). Also, in Reduced Detail mode, texture dimensions greater than 512 (after rounding to a power of two) will be halved (512x512 -> 256x526; 1024x256 -> 512x256). The power-of-two logic has been modified: instead of always rounding up, it will round to the nearest power of two. This means that, for instance, values from 768 to 1535 will be rounded to 1024.

Ideally, the power-of-two scaling would be removed on systems with OpenGL 2.0 support, but I don’t have such a system to test on, and there are some complications, so this probably won’t be in 1.69.

I expect to implement support for arbitrary texture sizes for HUD elements on systems which support the GL_EXT_texture_rectangle extension. This has limitations that makes it impractical to use it for ships, however, especially in combination with shaders.

A note on caching
If more than one active entity uses a given texture, it is desirable to share the texture between the entities to reduce memory usage and improve rendering performance. On the other hand, it is also sometimes desirable to be able to use the same texture file with different options in different contexts.

Oolite will reuse textures, but to identify them for reuse it will use a texture key which combines the name of the texture with the options it uses. [For the geekily curious, a texture key (currently) looks like “asciitext.png:0x0017/0/-0.75”, which means “asciitext.png with no_shrink=true, mag_filter=linear, min_filter=mipmap, anisotropy=0, texture_LOD_bias=-0.75”. Texture keys may show up in log messages from time to time.] This means that if the same texture file is used with different options, two copies of it will be kept around, even in some cases when this isn’t technically necessary.

Therefore, using the same options in each use of a texture, unless there’s a specific reason not to, is strongly recommended.
User avatar
Dr. Nil
---- E L I T E ----
---- E L I T E ----
Posts: 983
Joined: Thu Sep 28, 2006 5:11 pm
Location: Nearest Hoopy Casino
Contact:

Post by Dr. Nil »

Woaw! Great job. Can't wait to see what dream team builders will do with all these cool new options.

Shouldn't this be made sticky?
Image

300 billboards in Your Ad Here!
Astromines and more in Commies.
AVAILABLE HERE along with other Oolite eXpansion Packs.
User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

Post by JensAyton »

Nah. I’ll wikify it when it’s final.
User avatar
Rxke
Retired Assassin
Retired Assassin
Posts: 1757
Joined: Thu Aug 12, 2004 4:54 pm
Location: Belgium

Post by Rxke »

Wow, you're really on a roll with the code :D

Where do you get the inspiration?
User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

Post by JensAyton »

Inspiration is not a problem. Time and concentration are. :-)
User avatar
Selezen
---- E L I T E ----
---- E L I T E ----
Posts: 2525
Joined: Tue Mar 29, 2005 9:14 am
Location: Tionisla
Contact:

Post by Selezen »

Any chance of someone putting together a tutorial on how to create a nice modelwith all these fancy options?

Way to go, Ahruman - this is fantastic stuff!
User avatar
TGHC
---- E L I T E ----
---- E L I T E ----
Posts: 2157
Joined: Mon Jan 31, 2005 4:16 pm
Location: Berkshire, UK

Post by TGHC »

I'm afraid it's all over the top of my head, but those who know seem a bit impressed, so keep taking those tablets big A!

Is this purely for models, or will it also affect planets and clouds of space dust etc.
The Grey Haired Commander has spoken!
OK so I'm a PC user - "you know whats scary? Out of billions of sperm I was the fastest"
User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

Post by JensAyton »

Selezen wrote:
Any chance of someone putting together a tutorial on how to create a nice modelwith all these fancy options?
Possibly when they’re out in the wild. Most of the options will only be of interest in developing complex shaders (for instance, setting mag_filter to nearest will greatly simplify the electronic billboard shader I’ve got on the back burner); however, tweaking the texture_LOD_bias setting will be useful to anyone making textures with fine detail, and the ability to set specular highlights for models without custom shaders might be. (I’m going to experiment with default values for that.)
TGHC wrote:
Is this purely for models, or will it also affect planets and clouds of space dust etc.
These specific settings apply only to models, but the new texture code is already being used for text and will eventually be for planets (initially, ones with custom textures specified in planetinfo.plist) and HUD elements.
User avatar
Charlie
---- E L I T E ----
---- E L I T E ----
Posts: 262
Joined: Wed Sep 27, 2006 5:33 pm

Post by Charlie »

Wow! 8) :D

This will be great, thanks for all your hard work.

I've got to admit most of these details are over my head. Everything I've done so far has been from trial & error messing with PSP & 3D packages. I've never looked into shaders, mip-maps, texture maps, etc...
I hope someone produces a ship using all these bells + whistles as a demo of practicably how to do it. I'm already having some groovy thoughts but expect production to take a lot of head scratching.
Benulobiweed.inc
By Appointment to
--- : GalCoop : ---

Your nearest Benulobiweed.inc dealer:
http://www.box.net/public/b2tic3tjsk#main

Charlie
User avatar
drew
---- E L I T E ----
---- E L I T E ----
Posts: 2190
Joined: Fri May 19, 2006 9:29 am
Location: In front of a laptop writing a book.
Contact:

Post by drew »

Wish I understood what all that means but hey! Glad to see things are moving forward! Keep it up!

Cheers,

Drew.
Drew is an author of SF and Fantasy Novels
WebsiteFacebookTwitter
User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

Post by JensAyton »

Here, for your edification, is an example of stuff the new material model allows. It’s not designed to look good – it’s the material that will be used to highlight models whose DAT file has no TEXTURES section (different from missing textures, which will simply be white). It does, however, do stuff that currently can’t be done without a shader. It glows red, reflects blue light, and has a pink specular highlight:
Image

Code: Select all

<!-- in a shipdata.plist entry: -->
<key>materials</key>
<dict>
    <key>horrible_material</key>
    <dict>
        <key>diffuse_map</key>
        <string></string> <!-- Use no texture -->
        <key>diffuse</key>
        <string>0.0 0.0 0.4</string>
        <key>diffuse_map</key>
        <string></string>
        <key>ambient</key>
        <string>0.0 0.0 0.4</string>
        <key>emission</key>
        <string>0.5 0.1 0.0</string>
        <key>specular</key>
        <string>1.0 0.2 0.2</string>
        <key>shininess</key>
        <integer>5</integer>
    </dict>
</dict>
The asteroid nicely demonstrates the limited nature of per-vertex specular lighting, viz, it sucks. By 1.70 I intend to have per-pixel lighting for objects with no custom shader when high shader complexity is enabled.
User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

Post by JensAyton »

New material stuff is now documented on the wiki, with some minor syntax revisions. Also an example.
Makandal
Deadly
Deadly
Posts: 129
Joined: Tue Jun 12, 2007 3:48 pm
Location: Frenchman lost in Africa

Post by Makandal »

Very nice. You are roughly coding a complete ray tracing renderer ! We will have at least real metallic spacecrafts. Can you create a bump map for the texture ?
There is no theory of evolution. Just a list of creatures Chuck Norris has allowed to live.
User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

Post by JensAyton »

Makandal wrote:
Very nice. You are roughly coding a complete ray tracing renderer!
No. Really. :-) This just exposes a bit more of the standard OpenGL material model… and shaders.
Makandal wrote:
We will have at least real metallic spacecrafts. Can you create a bump map for the texture ?
Not as such. It is possible to use certain forms of normal mapping with shaders in Oolite, but most of the tools available are for tangent-space normal mapping, which requires additional information from the host application that Oolite does not provide at this time.
Post Reply