This document is meant to serve as a brief tutorial on how to use DIRSIG5 with existing DIRSIG4 scenes and simulation scenarios.

Introduction

DIRSIG5 represents a rewrite of the over 500,000 lines of code in DIRSIG4 to produce a new DIRSIG model that features higher radiometric and computational performance. This introduction attempts to briefly describe the major differences between the two versions of the software.

Plugin-Based Architecture

The DIRSIG5 software is constructed and deployed differently than DIRSIG4 was. Most features are provided via a series of plugins rather than features to a monolithic compiled code. This helps provide feature modularity and allows the radiometry core to establish stable and well tested APIs through which these plugins communicate to the radiometry engine.

The current DIRSIG5 software includes a set of plugins that provide backward compatibility with existing DIRSIG4 simulation configurations. However, the new DIRSIG5 software does not contain all the features of the mature DIRSIG4 code base that accumulated over the 12 years it was actively developed. As the DIRSIG5 software matures during this beta release period, more and more of the missing features will be introduced until nearly full parity exists between the two codes.

The list of the documented plugins employed by DIRSIG5 can be found in the Plugins Manual.

Improved and Simplified Radiometry

In DIRSIG4, the speed and fidelity of the simulation could be controlled through a series of configuration choices and settings spread across the material configurations and sensor modeling parameters. The two primary mechanisms utilized by users were:

Radiometry Solver Configurations

Radiometry solvers are the algorithms that performed the surface (or volume) leaving radiance calculations. The user could select a lower fidelity radiometry solver or dial down the parameters of a higher fidelity solver to decrease fidelity and improve run times.

Sub-Pixel Sampling Configurations

Sub-pixel sampling defines how many sample rays were used in each pixel. The more rays that are used, the better the estimate of the diversity and proportion of radiance signatures that are present within the collection area of the pixel. However, using more sub-pixel rays linearly correlates with run-time.

In general, users were often observed dialing down the fidelity of a DIRSIG4 simulation (using these two mechanisms) in order to meet compute time targets. Furthermore, there wasn’t an easy "single knob" interface to DIRSIG4 to control the fidelity vs. time trade off.

In DIRSIG5 there are no radiometry solvers because it utilizes a new, unified numerical radiometry calculation. DIRSIG5 uses "path tracing", which follows a single ray path from the pixel to either a source of energy (sun, moon, sky, light, etc.) or a termination (absorbed). Since this approach is emulating a Monte-Carlo integral of all the light paths reaching a pixel, the fidelity of the solution is proportional to how many samples (paths) are used. This "number of paths" parameter provides the sought after single knob interface controlling fidelity vs. time for the model. When a small number of paths are used, the resulting radiance estimate will be noisy. As more paths are used, the estimate improves. Controlling the number of paths used per pixel will be discussed below.

Multi-threaded Execution

Unlike DIRSIG4, the DIRSIG5 code is multi-threaded. That means more compute resources can be utilized for micro-scale parallelization on a multi-core computer. In later releases, macro-scale parallelization (using multiple computers in a cluster) will be available. Controlling how many threads are used by the model will be discussed below.

Multi-computer Execution

As of the 5.0.10 release, DIRSIG5 release include a Message Passing Interface (MPI) enabled executable that will allow the software to distribute computations on clustered computing resources. Details about using the MPI-enabled version of DIRSIG is described in a supplementary manual.

GPU Acceleration

Future releases of DIRSIG5 will utilize NVIDIAs CUDA and OptiX frameworks for GPU-accelerated ray tracing, which currently accounts for approximately 60% of the execution time.

Status and Limitations

As of November 2018, DIRISG5 is still only available through limited release to a beta user population. The software will be made available to the general user population over the winter of 2018/2019.

To date, the DIRSIG5 software effort has focused on radiometric and computational performance for passive EO simulations. The following limitations exist at this time.

  • Limited Thermal Capabilities

    • DIRSIG5 does not have a run-time temperature prediction solution at this time. The THERM algorithm used in previous generations of DIRSIG would require a massive rewrite to work in the computational approach and architecture used DIRSIG5 (10s or 100s of samples per pixel, multi-threading, etc.).

      • As of May 2021, this limitation is being actively addressed.

    • Scenes that feature user-supplied temperatures can be run. That includes the use of:

      • GDB files with per-facet temperatures,

      • Use of the data-driven temperature solver, and

      • Use of temperature maps

  • No SAR Capabilities

    • RIT has a white paper outlining the effort to get this active sensing modality supported under the new DIRSIG computational architecture.

  • No Polarization Capabilities

    • RIT currently has no plans to support this modality. The capability in DIRSIG4 was used by a very small fraction of the user population.

Installation Notes

File and Folder Layout

DIRSIG5 is bundled with a copy of the latest DIRSIG4 development release.

The bin folder

This folder contains a full set of the DIRSIG4 programs plus the DIRSIG5 programs (the main executable, the scene2hdf scene compiler, the SQT utility programs, etc.).

The demos folder

This folder contains the standard DIRSIG4 demos.

The docs folder

This folder contains all of the DIRSIG4 documentation and a handful of documents specific to DIRSIG5, including release notes and this document.

The extra folder

This folder contains various "extras" including the Blender plugins, BIN file reader for Matlab and MODTRAN6 wrapper that can be used with Linux and MacOS to resolve shared library location issues.

The lib folder

This folder (not present on Windows) contains the dynamic (shared) libraries that are used by the executables. On Windows, all dynamic libraries (DLLs) are in the bin folder.

The plugins folder

This folder contains all the DIRSIG5 plugins that provide most of the functional utility to the radiometry core. This currently includes sensor models and atmospheric plugins.

Program Wrappers

On the Linux platform, the bin folder contains a BASH shell script wrapper for the DIRSIG5 executables (see d5_wrapper.sh). Rather than require the user to customize their shell environment to setup shared library paths, it is strongly suggested that these wrappers are used, which allows the user to easily run the programs and avoids potential conflicts of the shared libraries we use with those perhaps installed on the host system. The actual executables are named with the version number appended to their filenames (e.g. the DIRSIG5 executable is named dirsig5-5.0.10) and a symbolic link to the d5_wrapper.sh BASH wrapper is created with the desired base name (e.g. dirsig5). This allows the user to execute the dirsig5 command, have the single script setup the environment and then launch the desired application.

Note On the Windows and Mac OSX platform there is no need for a wrapper script. On Windows, the required DLLs are in the bin folder and will be automatically found and used. On Mac OSX, the path to the dynamic libraries is set in the executables to be relative to the bin folder.

Configuration

There is an optional installation configuration file that allows the installation administrator to control some key parameters for all users. These parameters generally control how compute resources are utilized by individual users.

The INI formatted configuration file is called dirsig.ini and needs to be placed in the top of the installation folder (at the same level that the bin, plugins, etc. folders are located). If the file does not exist or a given parameter is not present, a set of default values are used. An example file is shown below:

[General]
max_threads=8
default_threads=4
max_threads

Defines the maximum number of threads that can be started by an individual simulation. The default value is the number of virtual cores present.

default_threads

Defines the default number of threads started by an individual simulation. The default value is the maximum number of threads. This number can be overridden for a simulation by the --threads command-line option. If the number of requested threads exceeds the maximum, an error is issued.

The idea behind the maximum and default thread count parameters is to help the installation administrator manage a compute resource being shared by many users and/or services. For example, you might have a machine with 32 cores and many users. Without setting the maximum thread parameter, a single user could start up a simulation that spawns a massive number of threads, leaving very few resources for other users. The maximum value is to define the maximum number of threads a single user could ever launch from a single simulation.

Note This parameter does not span processes or replace operating system level resource management. A single user could still launch multiple simulations that use additional thread pools.

The default thread count is the number of threads used by any simulation. This value can be less than the maximum if you want to create an environment where users can request more threads if needed (via the --threads command-line option), but by default they use a modest amount.

General Usage

To use the DIRSIG5 software, you will be running simulations from the command-line. In order to run an existing DIRSIG4 simulation, the only additional step required is to convert the scene into a pre-optimized form that the DIRSIG5 model will ingest. This conversion process is referred to as "scene compilation" and it is described below.

Scene Compilation

The scene compilation process was introduced to facilitate faster start-up times with DIRSIG5. With DIRSIG4, the loading of the scene geometry could be a very slow process if the scene was large and contained a lot of geometry. This DIRSIG4 compute time was spent performing sanity checks on the scene geometry, the association of geometry to materials, etc. The realization was that this work was repeated at the start of every simulation, even if the scene had not changed. With DIRSIG5, we replaced this frequent and compute intensive task with an infrequent pre-simulation task. The scene2hdf program included with DIRSIG5 will "compile" a DIRSIG4 scene into a single, binary Hierarchical Data Format (HDF) file that can be quickly loaded and utilized by the DIRSIG5 radiometry simulation. This compilation step performs all of these checks on the geometry and materials once and then allows the DIRSIG5 model to use it many times.

Important The user does not need to compile the scene before every simulation. This process only needs to be repeated if the contents of the scene have been modified.

To run the scene2hdf scene compiler, you provide the name of the scene file to the program:

$ scene2hdf my_site.scene
Note Please consult the scene2hdf manual to discover and understand all the options to the scene compiler.

The scene compiler will then begin the process of reading and checking all the geometry. It will then produce a file with the same name as the input .scene file but with .hdf appended to it. From our example above, the name of the final scene HDF file would be my_site.scene.hdf.

Important The HDF5 library expects the filesystem to support file locking to avoid corruption introduced by simultaneous read/write commands. If your filesystem doesn’t support file locking, then the HDF5 library will issue an error. The user can disable HDF5 file locking by setting the environment variable HDF5_USE_FILE_LOCKING to FALSE.

When the tool completes, it will print a series of warnings and (potentially) errors. Most warnings are for currently unsupported features that the tool can work around. Errors arise in situations where the tool cannot find a reasonable work around. Please consult the release notes document for each beta release for discussions of currently unsupported features and previously unsupported features that are now supported.

Running a Simulation

To run the simulation, the user simply provides the name of the DIRSIG4 .sim file to the DIRSIG5 program. The usage summary for the DIRSIG5 program is shown below:

DIRSIG5
Release: 5.0.10 (r3438)
Build Date: Dec 11 2019 13:51:12
Copyright 2015-2019 Rochester Institute of Technology

usage: dirsig5 [options] <sim filename>

basic [options]:
  --version
    Display version info
  --help
    Display this usage message
  --licenses
    Display licenses for third party code used in DIRSIG
  --random_seed=N
    Set the global random seed to 'N' (default:0)
  --threads=N
    Set the maximum number of threads to 'N' (default:<max>)
  --convergence=A,B,C
    Set min samples to 'A', max samples to 'B' and threshold to 'C' (default:20,100,1e-6)
  --max_nodes=N
    Set the maximum number of nodes per path to 'N' (default:4)
  --output_folder=<foldername>
    Specify the folder for output files
  --output_prefix=<prefix>
    Specify the prefix for output files
  --disable_earth_core
    Allow rays to pass through the Earth rather than hit the core
  --dry_run
    Do not perform simulation, build up the list of captures and print diagnostic information. Use --log_info to specify the destination for capture information to be written.
  --run_info
    A filename to which to write a JSON report with the input parameters to this simulation.
  --log_info
    A filename to which to write a JSON report with useful log output information about the simulation and captures. This is the main output of a dry run. If --run_info is not given, this file will also contain the simulation's input parameters.

BasicPlatform [options]:
  --split_channels={true,false}
    Use a unique spectral state for each channel
  --enable_direct_point_sources={true,false}
    Allow viewing points directly
  --enable_nonraster_source_truth={true,false}
    Generate non-raster (HDF) truth for visible sources

Controlling Multi-Threading

The software will detect the number of virtual cores (Intel Hyper-threading efficiently allows more than 1 thread to run on each physical core) and will automatically try to assign a thread to each. That will yield the highest utilization of the host computer’s resources, but in a shared computing environment this might not be desired. Hence, the user can specify the number of threads to start via the --threads option. For example, to use only 4 threads:

$ dirsig5 --threads=4 example.sim

Modifying the Event Schedule

Normally the sensor plugins will submit 1 or more sets of captures (referred to as "tasks") to define the scope of a simulation. In some situations, the user will not want to run all of those captures. To support that need, there are a series of options that allow the user to modify the capture event schedule generated by the sensor plugins.

To print the capture event schedule, the user can specify the --dry_run option:

$ dirsig5 --dry_run example.sim

Each capture event in the queue includes the index of the sensor plugin, the index of the task and the index of the capture within the task. This index triplet can be used specify any capture in the event queue. Indexes start at 0.

To run a specific capture in the simulation, the --capture option can be used. In the following example, capture 11 within task 7 for sensor plugin 0 will be run.

$ dirsig5 --capture=0:7:11 example.sim

To run all the captures in a specific task, the --task option can be used. In the following example, all the captures within task 7 for sensor plugin 0 will be run.

$ dirsig5 --task=0:7 example.sim

The --capture and --task options can be combined and/or supplied multiple times to specify a set of captures to be executed from the baseline queue. The following example runs all the captures in task index 1, then all the captures in task index 2, then capture index 1 in task index 3 and finally capture index 2 in task index 3:

$ dirsig5 --task=0:1 --task=0:2 --capture=0:3:1 --capture=0:3:2 example.sim

To start at a given capture and complete all the remaining captures in the event queue the --start_capture option can be used. In the following example, the event queue will start at capture 11 within task 7 for sensor plugin 0 and then continue through the remaining capture events in the queue.

$ dirsig5 --start_capture=0:7:11 example.sim

The --capture and --start_capture options also allow the user to optionally specify the starting problem within the capture (or within first capture in the case of the starting capture option). This option is useful in the situation where a capture was partially completed by a previous simulation. Consider a sensor that collects a 1280x720 frame in each capture. That translates into 1280 x 720 = 921,600 problems per capture. If the simulation only completed the first 300 lines of the capture, that means 1280 x 300 = 384,000 problems were completed. The user can finish this capture by specifying the optional starting problem index by adding a 4th field to the index triplet:

$ dirsig5 --capture=0:7:11:384000 example.sim

or, finish that capture and all the captures after it:

$ dirsig5 --start_capture=0:7:11:384000 example.sim
Note If the user supplies the starting problem index and the output for the previous problems have not been generated, most sensor plugins will pad/fill the file with 0s and then start writing the data.
Important Due to how captures are computed across multiple nodes, the problem index (4th field) option is not supported by the MPI-enabled version of DIRSIG5.

Output File and Folder Options

To support the output file and folder options available in DIRSIG4, the DIRSIG5 model has the same --output_folder=<foldername> and --output_prefix=<prefix> command-line options.

Controlling Fidelity

In DIRSIG5, the fidelity (quality) of the simulation is driven by how many paths are used in each pixel and the number of nodes (bounces) in those paths. The current algorithm supports an adaptive sampling scheme (similar to that employed in DIRSIG4) to vary the number of samples used in each pixel based on the complexity of the content within that pixel. A pixel with a larger dynamic range, more sub-pixel materials, etc. can use more samples and a pixel imaging a uniform, low variability surface can user fewer samples.

The adaptive pixel sampling algorithm has three parameters: (1) the minimum number of paths, the (2) the maximum number of paths and (3) the threshold for determining if the solution has converged.

  • The minimum number of samples specifies the minimum number of paths used in a pixel.

    • "Why wouldn’t I want this to be 1?"

      • Because the Monte-Carlo integral of a complex function with 1 sample will be poor. This will result in an incredibly very noisy image.

    • "What is the default?"

      • The default value is currently 20, which produces images that are decent quality as long as the illumination is generally high.

    • "Why would I increase the minimum number of samples?"

      • With only 20 samples, you might still get noise in the shadows because 20 wasn’t enough to detect the large dynamic range within a complex illumination environment (for example, shadows under a tree canopy).

  • The maximum number of samples specifies the maximum number of paths used in a pixel.

    • In general, this is the primary "quality" knob.

      • Turn it up to get more samples and drive down noise in the Monte-Carlo integration.

    • "What is the default?"

      • The default value is currently 100, which produces images that are decent quality as long as the illumination is generally high.

    • "Why would I increase the maximum number of samples?"

      • In general, it helps areas that are noisy. For example, shadow regions with complex illumination.

  • The threshold at which additional paths will no longer be add to a pixel.

    • Computed using the average radiance at the reference wavelength

    • This value is in units of radiance - it is not a fraction!

      • This value should be selected in terms of radiometry and not to make something pleasing to the eye.

      • Long term we expect the sensor model to define this value in terms of a minimal detectable radiance.

  • All three parameters are related:

    • At one extreme, if the threshold is very low (for example, 0) then each pixel will use the maximum number of paths.

    • At the other extreme, if the threshold is very high (for example, 1e100) then each pixel will use the minimum number of paths.

The forth key parameter is the number of nodes (bounces) along each path. This is controlled via the --max_nodes option. The default number of nodes is 4, but the user can increase it for highly scattering scenarios.

A summary of suggested convergence and node parameter configurations is listed in the table below.

Table 1. A table of suggested convergence and node count values for different simulations.
Mode Min Paths/Pixel Max Paths/Pixel Radiance Threshold Max Nodes

Default

20

100

1.0e-06

4

Better

30

500

1.0e-06

5

Even better

50

1000

1.0e-06

5

MWIR

20

100

1.0e-08

4

LWIR

20

100

1.0e-08

4

Night

30

500

1.0e-12

5

Lidar

50

100

1.0e-12

4

Preview

3

3

0.0

1

Tip The radiance levels are lower in the MWIR and LWIR regions, so the radiance threshold (which is absolute) needs to be lowered.
Tip At night the radiance values are much lower, so the radiance threshold needs to be decreased in order to match the magnitude range.
Tip The lidar instrument automatically adjusts the minimum paths/pixel and radiance threshold to at least match the values shown here. If the user specifies a minimum paths/pixel of less than 50 or a radiance threshold of greater than 1.0e-12, then the respective values will be modified automatically.
Tip The user can emulate the DIRSIG4 "preview mode" by supplying the -preview option to DIRSIG5. However unlike DIRSIG4, which would skip radiance calculations in this mode, the DIRSIG5 simulation will produce a low-fidelity radiance solution.

Using the Parallel/MPI version

Details about using the MPI-enabled version of DIRSIG is described in a supplementary manual.

Plugins

The list of the documented plugins employed by DIRSIG5 can be found in the Plugins Manual.