The "scene compilation" process was introduced with DIRSIG5 to facilitate faster start-up times. With DIRSIG4, the loading of the scene geometry could be a very slow process if the scene was large and contained a lot of geometry. This DIRSIG4 compute time was spent performing validity checks on the scene geometry, the association of geometry to materials, etc. The realization was that this work was repeated at the start of every simulation, even if the scene had not changed. With DIRSIG5, we replaced this frequent and compute intensive task with an infrequent pre-simulation task. The scene2hdf program included with DIRSIG5 will "compile" a DIRSIG4 scene into a single, binary Hierarchical Data Format (HDF) file that can be quickly loaded and utilized by the DIRSIG5 radiometry simulation.

Important The HDF5 library expects the filesystem to support file locking to avoid corruption introduced by simultaneous read/write commands. If your filesystem doesn’t support file locking, then the HDF5 library will issue an error. The user can disable HDF5 file locking by setting the environment variable HDF5_USE_FILE_LOCKING to FALSE.

This compilation step performs all of these checks on the geometry and materials once and then allows the DIRSIG5 model to use it many times. The introduction of this pre-simulation process has improved scene load times from 10s of minutes to 10s of seconds for large scenes.

Important The user does not need to compile the scene before every simulation. This process only needs to be repeated if the contents of the scene have been modified.

To run the scene2hdf scene compiler, you provide the name of the scene file to the program:

$ scene2hdf my_site.scene

If a DIRSIG4 simulation file is supplied, the corresponding scene file will be extracted from it:

$ scene2hdf my_demo.sim

The scene compiler will then begin the process of reading and checking all the geometry. It will then produce a file with the same name as the input .scene file but with .hdf appended to it. From our example above, the name of the final scene HDF file would be my_site.scene.hdf.

When the tool completes, it will print a series of warnings and (potentially) errors. Most warnings are for currently unsupported features that the tool can work around. Errors arise in situations where the tool cannot find a reasonable work around. Please consult the release notes document for each beta release for discussions of currently unsupported features and previously unsupported features that are now supported.

Command-Line Options

The scene2hdf tool has three command-line options:

DIRSIG4: DIRSIG5 Scene HDF Compiler
Release: 4.7.2 (r16649)
Build Date: Jun 10 2016 16:35:37
Copyright 2005-2016 Rochester Institute of Technology

usage: scene2hdf [options] <scene filename>

[options]:
--force_shared_obj_geometry
--force_numeric_material_labels
--allow_orphaned_maps

Forcing Shared OBJ Geometry

The --force_shared_obj_geometry option tells the compiler that duplicate OBJ files in any scene GLIST files will be shared. The default behavior is to create a unique copy of the OBJ each time it is requested. However, for some of RAMI test cases and some older scenes created with CityEngine, a massive number of duplicate OBJ file copies will result because of how the GLIST files were created. This results in huge scene HDF files and additional memory usage that is not required.

Specifying this option will force the scene geometry reader to share the memory for each unique OBJ file rather than allocate a copy each time the same OBJ file is requested in a scene.

Alpha-Numeric Material Label Problems

The --force_numeric_material_labels option allows users that had been using alpha-numeric material labels (IDs) in versions of DIRSIG 4.6.x and earlier. The most common situation is where a user has geometry in an OBJ or GDB file assigned the material label (ID) "10_grass" and their material database file had a material with "ID = 10". This obvious mismatch in material labels (IDs) would work in DIRSIG 4.6.3 (and earlier) because the string to numeric ID conversion would extract "10" from "10_grass". In DIRSIG 4.7.x and later, support for alpha-numeric material labels (IDs) was added (after years of being a requested feature). If the same geometry and material database setup is used, it will produce an error because there is not a material in the database labeled "10_grass" (it was labeled simply "10" in this example). If the user has a scene that took advantage of this numeric ID conversion in earlier versions of DIRSIG4 and it will not run in 4.7.x, then they need to specify the --force_numeric_material_labels option:

$ scene2hdf --force_numeric_material_labels my_site.scene

Specifying this option emulates the numeric label extraction used in DIRSIG 4.6.3 and earlier.

Orphaned Map Problems

As part of a series of improved error checks introduced in 4.7.x, a check for an "orphaned" map in the scene file was added. An "orphaned" map is when the user defines a map in the scene file, associates it with a given material label (ID) but that material label (ID) is not used on any geometry in the scene. For example, the user defines a material map that will be applied to any geometry assigned material label "10", but there is no geometry using material label "10". For DIRSIG 4.7.x and later, this is interpreted as a mistake in the scene configuration ("Why would the user bothered to setup a map that is not bound to any geometry?"). If DIRSIG 4.7.x or the DIRSIG5 scene compiler (which is currently built on DIRSIG 4.7.x code base) sees an "orphaned", it will report and error and quit.

If you have a scene with orphaned maps and want them to be ignored by the scene compiler, then the --allow_orphaned_maps option can be provided:

$ scene2hdf --allow_orphaned_maps my_site.scene

It should be noted that the scene HDF files are completely portable, so they can be relocated, shared, version controlled, etc. However, to have an existing DIRSIG4 .sim file run without modification the file should remain in the same folder as the original .scene file it was produced from.

Other Options

Controlling the spectral coverage

One of the tasks of this tool is to minimize the overhead of spectral state changes during the DIRSIG5 run-time, which is partially accomplished by resampling all the scene spectral to a common set of wavelengths during scene compilation. There are three options to control the spectral resampling when the scene is compiled.

Using the Default Sampling

At this time, the default set of wavelengths is hard-coded to span the visible through short-wave infrared (SWIR) (0.350 - 2.550 microns @ 0.001 micron sampling). This creates problems for users simulating in the mid-wave infrared (MWIR) and long-wave infrared (LWIR) regions, because the scene HDF does not contain spectral reflectance and/or emissivity data for this wavelengths by default. Therefore, it is advised that users utilize one of the following two alternatives.

Using the Scene Modality Properties

The graphical scene editor and scene files now support a way for the user to indicate which common wavelength windows a scene was created for. This allows the user to easily describe the spectral coverage of a scene using these commonly used regions:

  • Ultraviolet (UV, 0.20 - 0.40 microns)

  • Visible (VIS, 0.35 - 0.80 microns)

  • Near Infrared (NIR, 0.70 - 1.40 microns)

  • Short-wave Infrared (SWIR, 1.0 - 2.5 microns)

  • Mid-wave Infrared (MWIR, 3.0 - 5.0 microns)

  • Long-wave Infrared (LWIR, 7.5 - 14.0 microns)

The scene compiler will resolve overlaps in these pre-defined spectral windows. These windows can be selected in the graphical scene editor on the General tab (see below):

images/scene2hdf/scene_editor.png
Figure 1. The spectral coverage windows can be selected in the "Features" on the General tab in the graphical scene editor.

They can also be specified in the .scene file via the features attribute in the <properties> element:

<classicscene>
  ...
  <properties features="uv,vis,nir,swir,mwir,lwir"/>
  ...
</classicscene>

Please note that all of the corresponding window names are provided in the example above. The scene builder should only specify wavelength windows for which the scene has valid spectral data.

Using a Spectral Samples File

To override the default spectral sampling used by the scene compiler, the user needs to create a file named spectral_samples.txt that contains a list of wavelengths to be used for spectral resampling at scene compilation time.

Important The presence of a spectral_samples.txt overrides the sampling defined by the .scene file modality properties described above.

The file starts with microns (other supported units include nanometers and wavenumbers) and then includes a list of arbitrary wavelengths. These wavelengths do not need to be regularly spaced.

microns
0.4
0.41
0.42
[lines deleted for documentation purposes]
11.90
11.95
12.00
Important If a user combines a sensor plugin and a scene HDF that do not have overlapping spectral coverage, an error will be issued by DIRSIG.

Additional Output

In addition to the standard geometry_report.txt and material_report.txt files produced by the DIRSIG4 scene loading process, this tool produces the file asset_report.txt. This ASCII/Text file contains a list of all the input files that were read during the scene loading process. Each line contains the filename, the modification date and the size of the file. The last line of the file contains an MD5 hash created from the contents of all the input files.

Below is an example asset_report.txt for the Brdf1 demo:

demo.scene,Wed Jan 18 16:22:59 2017,836
geometry/demo.odb,Wed Jan 18 16:22:59 2017,2218
materials/brdf.mat,Wed Jan 18 16:23:00 2017,3171
materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841
materials/emissivity/red_panel.ems,Wed Jan 18 16:23:00 2017,1114
materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841
materials/emissivity/red_panel.ems,Wed Jan 18 16:23:00 2017,1114
materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841
materials/emissivity/red_panel.ems,Wed Jan 18 16:23:00 2017,1114
materials/brdf/au.fit,Wed Jan 18 16:23:00 2017,4010
materials/emissivity/au.ems,Wed Jan 18 16:23:00 2017,233
materials/brdf/aud.fit,Wed Jan 18 16:23:00 2017,4000
materials/emissivity/au.ems,Wed Jan 18 16:23:00 2017,233
materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841
materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841
materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841
materials/brdf/au.fit,Wed Jan 18 16:23:00 2017,4010
materials/brdf/aud.fit,Wed Jan 18 16:23:00 2017,4000
MD5 hash = ce684b876558e72777524fb2ba5f6631

Important Notes:

  • The list of files in the report only includes files that are used in the scene. Files that are not used in the scene are not included in the listing or the MD5 hash. Hence, this file listing can be used to create the definitive list of files used in a given scene folder. By extension, the list can be used to relocate/remove unused files that are cluttering up a given scene.

  • The MD5 hash will only change if one (or more) of the files in the scene is changed. Hence, it can be used to compare two copies of a scene to confirm if they are a match or not. If the MD5 hash is the same, the various scene files can be assumed to match. If they hash is not the same, the file listing, modification dates and file sizes can be used to identify which files differ.