The "scene compilation" process was introduced with DIRSIG5 to facilitate faster start-up times. With DIRSIG4, the loading of the scene geometry could be a very slow process if the scene was large and contained a lot of geometry. This DIRSIG4 compute time was spent performing validity checks on the scene geometry, the association of geometry to materials, etc. The realization was that this work was repeated at the start of every simulation, even if the scene had not changed. With DIRSIG5, we replaced this frequent and compute intensive task with an infrequent pre-simulation task. The scene2hdf program included with DIRSIG5 will "compile" a DIRSIG4 scene into a single, binary Hierarchical Data Format (HDF) file that can be quickly loaded and utilized by the DIRSIG5 radiometry simulation.
|
The HDF5 library expects the filesystem to support file locking
to avoid corruption introduced by simultaneous read/write
commands. If your filesystem doesn’t support file locking, then
the HDF5 library will issue an error. The user can disable
HDF5 file locking by setting the environment variable
HDF5_USE_FILE_LOCKING to FALSE.
|
This compilation step performs all of these checks on the geometry and materials once and then allows the DIRSIG5 model to use it many times. The introduction of this pre-simulation process has improved scene load times from 10s of minutes to 10s of seconds for large scenes.
|
|
The user does not need to compile the scene before every simulation. This process only needs to be repeated if the contents of the scene have been modified. |
To run the scene2hdf scene compiler, you provide the name of the scene file to the program:
$ scene2hdf my_site.scene
If a DIRSIG4 simulation file is supplied, the corresponding scene file will be extracted from it:
$ scene2hdf my_demo.sim
The scene compiler will then begin the process of reading and checking
all the geometry. It will then produce a file with the same name as the
input .scene file but with .hdf appended to it. From our example
above, the name of the final scene HDF file would be my_site.scene.hdf.
When the tool completes, it will print a series of warnings and (potentially) errors. Most warnings are for currently unsupported features that the tool can work around. Errors arise in situations where the tool cannot find a reasonable work around. Please consult the release notes document for each beta release for discussions of currently unsupported features and previously unsupported features that are now supported.
Command-Line Options
The scene2hdf tool has three command-line options:
DIRSIG4: DIRSIG5 Scene HDF Compiler Release: 4.7.2 (r16649) Build Date: Jun 10 2016 16:35:37 Copyright 2005-2016 Rochester Institute of Technology usage: scene2hdf [options] <scene filename> [options]: --force_shared_obj_geometry --force_numeric_material_labels --allow_orphaned_maps
Forcing Shared OBJ Geometry
The --force_shared_obj_geometry option tells the compiler that duplicate
OBJ files in any scene GLIST files will be shared. The default behavior is
to create a unique copy of the OBJ each time it is requested. However,
for some of RAMI test cases and some older scenes created with CityEngine,
a massive number of duplicate OBJ file copies will result because of how
the GLIST files were created. This results in huge scene HDF files and
additional memory usage that is not required.
Specifying this option will force the scene geometry reader to share the memory for each unique OBJ file rather than allocate a copy each time the same OBJ file is requested in a scene.
Alpha-Numeric Material Label Problems
The --force_numeric_material_labels option allows users that had been
using alpha-numeric material labels (IDs) in versions of DIRSIG 4.6.x and
earlier. The most common situation is where a user has geometry in an OBJ
or GDB file assigned the material label (ID) "10_grass" and their material
database file had a material with "ID = 10". This obvious mismatch in
material labels (IDs) would work in DIRSIG 4.6.3 (and earlier) because
the string to numeric ID conversion would extract "10" from "10_grass".
In DIRSIG 4.7.x and later, support for alpha-numeric material labels
(IDs) was added (after years of being a requested feature). If the
same geometry and material database setup is used, it will produce an
error because there is not a material in the database labeled "10_grass"
(it was labeled simply "10" in this example). If the user has a scene
that took advantage of this numeric ID conversion in earlier versions
of DIRSIG4 and it will not run in 4.7.x, then they need to specify the
--force_numeric_material_labels option:
$ scene2hdf --force_numeric_material_labels my_site.scene
Specifying this option emulates the numeric label extraction used in DIRSIG 4.6.3 and earlier.
Orphaned Map Problems
As part of a series of improved error checks introduced in 4.7.x, a check for an "orphaned" map in the scene file was added. An "orphaned" map is when the user defines a map in the scene file, associates it with a given material label (ID) but that material label (ID) is not used on any geometry in the scene. For example, the user defines a material map that will be applied to any geometry assigned material label "10", but there is no geometry using material label "10". For DIRSIG 4.7.x and later, this is interpreted as a mistake in the scene configuration ("Why would the user bothered to setup a map that is not bound to any geometry?"). If DIRSIG 4.7.x or the DIRSIG5 scene compiler (which is currently built on DIRSIG 4.7.x code base) sees an "orphaned", it will report and error and quit.
If you have a scene with orphaned maps and want them to be ignored
by the scene compiler, then the --allow_orphaned_maps option can be
provided:
$ scene2hdf --allow_orphaned_maps my_site.scene
It should be noted that the scene HDF files are completely portable, so
they can be relocated, shared, version controlled, etc. However, to
have an existing DIRSIG4 .sim file run without modification the file
should remain in the same folder as the original .scene file it was
produced from.
Other Options
Controlling the spectral coverage
One of the tasks of this tool is to minimize the overhead of spectral state changes during the DIRSIG5 run-time, which is partially accomplished by resampling all the scene spectral data to a common set of wavelengths during scene compilation. There are three options to control the spectral resampling when the scene is compiled.
Using the Default Sampling
At this time, the default set of wavelengths is hard-coded to span the visible through short-wave infrared (SWIR) (0.350 - 2.550 microns @ 0.001 micron sampling). This creates problems for users simulating in the mid-wave infrared (MWIR) and long-wave infrared (LWIR) regions, because the scene HDF does not contain spectral reflectance and/or emissivity data for this wavelengths by default. Therefore, it is advised that users utilize one of the following two alternatives.
Using the Scene Modality Properties
The graphical scene editor and scene files now support a way for the user to indicate which common wavelength windows a scene was created for. This allows the user to easily describe the spectral coverage of a scene using these commonly used regions:
-
Ultraviolet (UV, 0.20 - 0.40 microns @ 0.001 microns)
-
Visible (VIS, 0.35 - 0.80 microns @ 0.001 microns)
-
Near Infrared (NIR, 0.70 - 1.40 microns @ 0.001 microns)
-
Short-wave Infrared (SWIR, 1.0 - 2.5 microns @ 0.001 microns)
-
Mid-wave Infrared (MWIR, 3.0 - 5.0 microns @ 0.005 microns)
-
Long-wave Infrared (LWIR, 7.5 - 14.0 microns @ 0.005 microns)
The scene compiler will resolve overlaps in these pre-defined spectral windows. These windows can be selected in the graphical scene editor on the General tab (see below):
They can also be specified in the .scene file via the features
attribute in the <properties> element:
<classicscene> ... <properties features="uv,vis,nir,swir,mwir,lwir"/> ... </classicscene>
Please note that all of the corresponding window names are provided in the example above. The scene builder should only specify wavelength windows for which the scene has valid spectral data.
Using a Spectral Samples File
To override the default spectral sampling used by the scene compiler,
the user needs to create a file that contains a list of wavelengths to be used
for spectral resampling at scene compilation time. This file is then passed to
scene2hdf through the --spectral_samples command line option.
|
|
Using a spectral samples file overrides the sampling defined by the
.scene file modality properties described above. This is an
advanced feature that is only recommended for edge cases, not for
general use.
|
The file starts with microns (other supported units include nanometers
and wavenumbers) and then includes a list of arbitrary wavelengths.
These wavelengths do not need to be regularly spaced.
microns 0.4 0.41 0.42 [lines deleted for documentation purposes] 11.90 11.95 12.00
|
|
If a user combines a sensor plugin and a scene HDF that do not have overlapping spectral coverage, an error will be issued by DIRSIG. |
Additional Output
In addition to the standard geometry_report.txt and material_report.txt
files produced by the DIRSIG4 scene loading process, this tool produces the
file asset_report.txt. This ASCII/Text file contains a list of all the
input files that were read during the scene loading process. Each line
contains the filename, the modification date and the size of the file.
The last line of the file contains an MD5 hash created from the contents
of all the input files.
Below is an example asset_report.txt for the Brdf1 demo:
demo.scene,Wed Jan 18 16:22:59 2017,836 geometry/demo.odb,Wed Jan 18 16:22:59 2017,2218 materials/brdf.mat,Wed Jan 18 16:23:00 2017,3171 materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841 materials/emissivity/red_panel.ems,Wed Jan 18 16:23:00 2017,1114 materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841 materials/emissivity/red_panel.ems,Wed Jan 18 16:23:00 2017,1114 materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841 materials/emissivity/red_panel.ems,Wed Jan 18 16:23:00 2017,1114 materials/brdf/au.fit,Wed Jan 18 16:23:00 2017,4010 materials/emissivity/au.ems,Wed Jan 18 16:23:00 2017,233 materials/brdf/aud.fit,Wed Jan 18 16:23:00 2017,4000 materials/emissivity/au.ems,Wed Jan 18 16:23:00 2017,233 materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841 materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841 materials/brdf/gloss.fit,Wed Jan 18 16:23:00 2017,841 materials/brdf/au.fit,Wed Jan 18 16:23:00 2017,4010 materials/brdf/aud.fit,Wed Jan 18 16:23:00 2017,4000 MD5 hash = ce684b876558e72777524fb2ba5f6631
Important Notes:
-
The list of files in the report only includes files that are used in the scene. Files that are not used in the scene are not included in the listing or the MD5 hash. Hence, this file listing can be used to create the definitive list of files used in a given scene folder. By extension, the list can be used to relocate/remove unused files that are cluttering up a given scene.
-
The MD5 hash will only change if one (or more) of the files in the scene is changed. Hence, it can be used to compare two copies of a scene to confirm if they are a match or not. If the MD5 hash is the same, the various scene files can be assumed to match. If they hash is not the same, the file listing, modification dates and file sizes can be used to identify which files differ.