Summary

The NewAtmosphere plugin was created to provide a new capability supporting for physics-based atmospheric models in DIRSIG5. It attempts to address the limitations of the DIRSIG4-era make_adb and "classic" atmosphere model combination in a variety of ways. Specifically, the goals were:

  • Work with the DIRSIG5 scene HDF file directly (no need to carry around the entire scene directory).

  • Move to a binary database format that is more compact to store and faster to load.

  • Simplify the ability to integrate multiple backend physics-based atmospheric models.

  • Run multiple instances of the backend physics-based atmospheric models to build databases faster.

  • Add support for multiple times in a single database.

  • Add support for multiple altitudes in a single database.

The capability is currently divided into two major components:

  • The atm_builder tool, which interfaces with the externally developed physics-based atmospheric models to construct an HDF database of atmospheric data tables.

  • The NewAtmosphere plugin, which uses the HDF atmospheric database to drive the atmospheric contributions in a DIRSIG5 execution.

Backend Models

One of the key features of the new toolchain is that it is much simpler to implement interfaces to different backend physics models for the atmosphere. We call these different models "backend" models and there are currently 2 backend interfaces that have been created:

  • The "dummy" backend, which doesn’t interface to an external model. Instead, it has analytical models for the atmosphere that are similar to the capabilities of the Simple atmosphere model in DIRSIG4.

  • The "MODTRAN Tape" backend, which interfaces with MODTRAN using the classic "tape-based" input/output files. This should be compatible with MODTRAN4, MODTRAN5 and MODTRAN6.

Which backend model is used is part of the input configuration.

Tip A a future date, backends will be developed to (1) use MODTRAN6 via JSON input/output method and (2) interface with the 6SV model.

Input

The inputs to the plugin is a JSON document that can be either directly supplied in the JSIM file where the plugin is requested (via the inputs section) or in a separate JSON file (via the input_filename variable). The details of these two options are described later. Regardless of the mechanism used to supply that JSON document to the plugin, this section describes what that JSON input document looks like.

Below is a generalized example. It consists of an info section that describes the atmosphere setup, a backend section that describes which backend is to be used (via the name variable) and a set of common variables that have a uniform behavior across all the backends:

            "info" : {
                "name" : "A great atmosphere",
                "author" : "Joe User",
                "organization" : "Awesome, Corp",
                "description" : "This is a great atmosphere setup."
            },
            "backend" : {
                "name" : "<backend_name>",
                ...
            }
            "max_processes" : 8,
            "time_delta" : 900,
            "altitude_delta" : 500,
            "hdf_filename" : "my_sim.adb.hdf"

Common variables

The maximum number of processes

The max_processes variable defines the maximum number of processes that the backend model can use. For example, multiple simultaneous MODTRAN processes can be used to generate the paths. On a multi-core computer, you can generally run one process per physical core. On a multi-core computer that supports hyper-threading, you can usually run 2 processes per physical core. The default value is set based on the number of virtual cores available on the host computer.

Note This variable can be overridden with the --max_processes command-line argument. See below for details.

The time sampling

The time_delta variable defines the time resolution of the database. Since the atmosphere is not expected to vary rapidly, it does not make sense to compute new atmospheric values at 10s, 100s or 1000s of Hz if that is what the readout rate is of the sensor. Instead, the toolchain makes the assumption that the atmosphere is optically and geometrically (sun and moon scattering) "frozen" for some period of time. The default period or time resolution is 900 seconds (15 minutes).

Note This variable can be overridden with the --time_delta command-line argument. See below for details.

The altitude sampling

The altitude_delta variable defines the altitude resolution of the database. Since the atmosphere is not assumed to be uniform in the vertical dimension, it needs to be sampled as a function of altitude if the scene(s) span significant altitude range. When the builder loads the scene(s), it looks at the altitudes spanned by the scene bounding box(es). Using the altitude sampling delta, it will construct a list of altitudes within each scene and then merge those into a combined list of altitudes for the database.

The default value for the altitude sampling delta is 500m. In most cases, the number of altitudes used in the database is a scalar multiplier for the number of backend model runs that need to be performed.

  • Decreasing the sampling delta increases the number of altitudes used in the database.

  • Increasing the sampling delta decreases the number of altitudes used in the database.

Note This variable can be overridden with the --altitude_delta command-line argument. See below for details.

The HDF database filename

The hdf_filename variable defines the name of the HDF file to hold the resulting atmospheric database. The convention so far has been to use the compound extension ".adb.hdf".

Using the Dummy Backend

The "Dummy" backend is built-in (it does not call an external program) and it implements a set simplified calculations to approximate nominal illumination conditions:

  • The sun is modeled with a 5800K blackbody spectral distribution that is magnitude normalized to approximate the Earth reaching irradiance of the sun.

  • The moon is modeled using the same solar spectral distribution, but is magnitude normalized to approximate the Earth reaching irradiance of the moon.

  • The sky is modeled with an exponentially decaying spectral distribution that approximates clear sky scattering conditions in the visible.

  • There is no path radiance or transmission along other paths.

This backed is configured without any additional options:

        "backend" : {
          "name" : "dummy"
        }
Important The "dummy" backend is primarily for testing purposes and rapid use during simulation setup and debugging. It should not be used for data production simulations.

Using the MODTRAN Tape Backend

The "MODTRAN Tape" backend is configured with a variety of options that provide details about the external MODTRAN installation and user input files. For the examples included here, the specific paths to the MODTRAN executable, data folder, etc. may differ from your installation.

Important Although the examples below show various configurations for older versions of MODTRAN, the user is strongly encouraged to use the latest version of MODTRAN6. Visit the MODTRAN page at the Spectral Sciences, Inc. website for more information.

The option to extract atmospheric profile data

The MODTRAN tape backend can optionally extract the atmospheric profile from the MODTRAN output "tape6" file. When this profile data is extracted, it is supplied to the radiometry core to drive refraction through the atmosphere. This option is controlled via the extract_profile variable, which is set to either true or false (note, boolean values in JSON are not strings, so do not assign the variable "true").

Note This option is disabled by default.

Using a MODTRAN configuration profile from your settings

The first option for specifying the MODTRAN setup is using the MODTRAN "profiles" in the user’s DIRSIG settings (or preferences). In that case the setup includes the modtran_profile variable:

        "backend" : {
          "name" : "modtran_tape",
          "modtran_profile" : "MODTRAN5_2_4",
          "modroot_filename" : "mod5root.in",
          "tape5_filename" : "./mls.tp5",
          "extract_profile" : false
        }

In this example, the profile named "MODTRAN5_2_4" might be mapped to a MODTRAN5 2.4 installation.

Note This backend uses the "modroot" approach to naming the input and output files associated with the MODTRAN run (see the MODTRAN documentation for more information about this mechanism). The modroot_filename variable indicates the name of the file containing the base name of the MODTRAN input and output files. Since the name of this file has changed depending on the version of MODTRAN, it is provided as a user configuration variable.

Supplying the installation paths: MODTRAN4

Another option is to specify the path to the executable and data directory directly in the backend description:

        "backend" : {
          "name" : "modtran_tape",
          "modtran_exe" : "/Users/dirsig/Mod4v3r1/Mod4v3r1.exe",
          "modtran_data" : "/Users/dirsig/Mod4v3r1/Mod4v3r1/DATA",
          "modroot_filename" : "modroot.in",
          "modtran_min_delta" : 1.0,
          "tape5_filename" : "./mls.tp5",
          "extract_profile" : false
        }

In this example, the user is pointing at a MODTRAN4 Version 3, Release 1 installation in their own directory. The modtran_exe variable is assigned the full path to the MODTRAN executable, the modtran_data is assigned the full path to the MODTRAN DATA directory and the modtran_min_delta variable is assigned the limiting frequency resolution for the band model database available with that version of MODTRAN (typically 1.0 wavenumbers for MODTRAN4). The name of the "modroot" file for MODTRAN4 was modroot.in.

Supplying the installation paths: MODTRAN5

Another option is to specify the path to the executable and data directory directly in the backend description:

        "backend" : {
          "name" : "modtran_tape",
          "modtran_exe" : "/Users/dirsig/Modtran5.2.4/mod5_cons.exe",
          "modtran_data" : "/Users/dirsig/Modtran5.2.4/DATA",
          "modroot_filename" : "mod5root.in",
          "modtran_min_delta" : 0.1,
          "tape5_filename" : "./mls.tp5",
          "extract_profile" : false
        }

In this example, the user is pointing at a MODTRAN5 2.4 installation in their own directory. The modtran_exe variable is assigned the full path to the MODTRAN executable, the modtran_data is assigned the full path to the MODTRAN DATA directory and the modtran_min_delta variable is assigned the limiting frequency resolution for the band model database available with that version of MODTRAN (0.1 wavenumbers for MODTRAN5 and later). The name of the "modroot" file for MODTRAN5 was mod5root.in.

Note The specific paths to the MODTRAN executable will vary based on your installation.

Supplying the installation paths: MODTRAN6

Although MODTRAN6 introduced a new JSON input/output mechanism, the program is backward compatible with the traditional "tape" input/output mechanism utilized by this backend. This input/output option is accessed via the -modroot command line option, which is described in the manual and shown in the usage message of the console executable:

$ ./mod6c_cons -h
Usage: ./mod6c_cons input_file [data_path]
JSON and TAPE5 input formats are supported.
[data_path] specifies an optional path to the MODTRAN data directory.

optional flags
  -modroot
   Process legacy input formats using mod5root.in or tape5 files.

...

This backend employs the "modroot" approach for all versions of MODTRAN, where the basename of the input/output files for the MODTRAN run is stored in a separate file. MODTRAN5 and MODTRAN4 automatically looked for the mod5root.in and modroot.in files, respectively. MODTRAN6, however, needs the -modroot option to be specified as the 1st command-line argument for it to look in the mod5root.in file. Additionally, the MODTRAN6 executable needs to be supplied the full path to the MODTRAN6 DATA folder as the 2nd command-line argument. Hence, the complete command-line to launch MODTRAN6 needs to look like:

$ /some/path/MODTRAN6.0.0/bin/mod6c_cons.exe -modroot /some/path/MODTRAN6.0.0/DATA

This backend has an optional configuration variable that allows the user to customize the command-line arguments that are passed to the MODTRAN executable. These arguments are defined in the JSON using the modtran_options variable, which is a JSON array of strings. The example configuration below shows how these two additional arguments are provided to the MODTRAN executable using the options array variable:

        "backend" : {
          "name" : "modtran_tape",
          "modtran_exe" : "/Users/dirsig/MODTRAN6.0.0/mod6c_cons.exe",
          "modtran_data" : "/Users/dirsig/MODTRAN6.0.0/DATA",
          "modroot_filename" : "mod5root.in",
          "modtran_options" : ["-modroot","/Users/dirsig/MODTRAN6.0.0/DATA"],
          "modtran_min_delta" : 0.1,
          "tape5_filename" : "./mls.tp5",
          "extract_profile" : false
        }
Note The specific paths to the MODTRAN executable will vary based on your installation.

The input "tape5" filename

The tape5_filename variable defines the name of the input "tape5" file that is used as a template for the MODTRAN runs. In general, the backend is using the description of the atmosphere in this file (which model, aerosols, visibility, etc.) and is only updating the key parameters that are relevant to the specific DIRSIG simulation (the wavelengths, the source position, the path geometry, etc.).

The user also has the option to store the MODTRAN "tape5" file within the input JSON description via the tape5_contents variable. However, because the "tape5" file is multiple lines and you cannot embed newlines inside a JSON variable, the user will need to make the file a single string by replacing the newlines with the \n character:

    "name" : "NewAtmosphere",
    "inputs" : {
      "info" : {
        [deleted for documentation purposed]
      },
      "backend" : {
        "name" : "modtran_tape",
        "modtran_profile" : "MODTRAN4",
        "modtran_exe" : "/Users/scottbrown/pkg/Mod4v3r1/Mod4v3r1.exe",
        "modtran_options" : [""],
        "modtran_data" : "/Users/scottbrown/pkg/Mod4v3r1/DATA",
        "extract_profile" : false,
        "tape5_contents" : "M   2    2    2    0    0    0    0    0    0    0    1    1    1   0.000   0.00\n    2    0    0    1    0    0     0.000     0.000     0.000     0.000     0.218\n    20.218     0.218   180.000     0.000     0.000     0.000    0\n    1    2  236    0\n    43.000    77.000     0.000     0.000    12.000     0.000     0.000     0.000\n       450     35050       100     2\n0"
      },
      "hdf_filename" : "mls_rural_23km_dis4_scaled.adb.hdf"
    }
Important The user needs to be very careful when performing this newline replacement operation to otherwise not disrupt the precise format of the MODTRAN "tape5" file.

Usage

Input description method

As it was mentioned earlier, the plugin and JSIM file support two mechanisms for supplying the JSON document containing the input configuration.

JSIM Using an Internal Description

Below is an example where the plugin description is contained in the JSIM file:

[{
  "scene_list" : [
    { "inputs" : "./demo.scene" }
  ],
  "plugin_list" : [
    {
      "name" : "NewAtmosphere",
      "inputs" : {
        "info" : {
          "name" : "MLS Rural 23km w/ Isaac 2-stream",
          "author" : "Scott D. Brown",
          "organization" : "RIT",
          "description" : "Mid-Latitude Summer with rural aerosols and 23 km visibility, using Isaac 2-stream multiple scattering"
        },
        "backend" : {
          "name" : "modtran_tape",
          "modtran_profile" : "MODTRAN5_2_4",
          "modroot_filename" : "mod5root.in",
          "tape5_filename" : "./mls_rural_23km_dis4.tp5"
        },
        "hdf_filename" : "mls_rural_23km_dis4.adb.hdf"
      }
    },
    {
      "name" : "BasicPlatform",
      "inputs" : {
        "platform_filename" : "./demo.platform",
        "motion_filename" : "./demo.ppd",
        "tasks_filename" : "./demo.tasks",
      }
    }
  ]
}]

JSIM Using an External Description

Alternatively, the JSON can be stored in an external JSON file and the name of that file specified in the JSIM file. This approach allows a single copy of a configuration to be stored and shared among multiple simulations. The example below contains the plugin configuration using the same JSON structure but in an external file (named mls_rural_23km_dis4.jatm in this example):

{
  "info" : {
    "name" : "MLS Rural 23km w/ Isaac 2-stream",
    "author" : "Scott D. Brown",
    "organization" : "RIT",
    "description" : "Mid-Latitude Summer with rural aerosols and 23 km visibility, using Isaac 2-stream multiple scattering"
  },
  "backend" : {
    "name" : "modtran_tape",
    "modtran_profile" : "MODTRAN5_2_4",
    "modroot_filename" : "mod5root.in",
    "tape5_filename" : "./mls_rural_23km_dis4.tp5"
  },
  "max_processes" : 8,
  "hdf_filename" : "mls_rural_23km_dis4.adb.hdf"
}

and then this external file is referenced in the JSIM file:

[{
  "scene_list" : [
    { "inputs" : "./demo.scene" }
  ],
  "plugin_list" : [
    {
      "name" : "NewAtmosphere",
      "inputs" : {
        "input_filename" : "mls_rural_23km_dis4.jatm"
      }
    },
    {
      "name" : "BasicPlatform",
      "inputs" : {
        "platform_filename" : "./demo.platform",
        "motion_filename" : "./demo.ppd",
        "tasks_filename" : "./demo.tasks"
      }
    }
  ]
}]

The builder tool

The atm_builder tool is what will build the HDF atmospheric database. It loads the simulation configuration, determines what backend calculations need to be performed and then has the backend execute those calculations. Below is the command-line usage message when either no arguments or the --help option is supplied:

$ atm_builder --help
usage: atm_builder [options] <sim filename>

basic [options]:
  --version
    Display version info
  --help
    Display this usage message
  --verbose
    Enable verbose output of the tool
  --no_run
    Do not run the backend atmospheric model ('dry run' mode)
  --save_files
    Save input/output files generated by the backend atmospheric model
  --time_delta=T
    Set the interval between time steps (default: 900 seconds)
  --altitude_delta=H
    Set the interval between altitude steps (default: 500 meters)
  --fov_min_deltas=X,Y
    Set the minimum angle between sensor FOV paths (default: 5,5 degrees)
  --max_processes=N
    Set the maximum number of backend model processes (default: <virtual cores>)

General operation entails supplying the name of the JSIM file to the tool, along with any options.

$ atm_builder example.jsim

Command-Line Options

The following command-line options provide additional utility:

--no_run or --no_run=<true,false>

This option will tell the backend to configure all the backend runs, but it will not execute them. The input files will be preserved after the program completes. The default value is false.

--save_files or --save_files=<true,false>

This option will tell the backend to save all the input and output files generated to execute the associated model (overrides the save_files option in the JSON configuration). The default value is false.

--time_delta=T

This option will set the time resolution (in seconds) of the database (overrides the time_delta variable in the JSON configuration). The default value is 900 seconds (15 minutes).

--altitude_delta=H

This option will set the altitude resolution (in meters) of the database (overrides the altitude_delta variable in the JSON configuration). The default value is 500 meters (0.5 km).

--fov_min_deltas=X,Y

This option specifies the minimum angle between path samples within the field-of-view (FOV) of any given capture. The default values are 5 and 5 degrees in zenith and azimuth.

--max_processes=N

This option will tell the backed the maximum number of external model processes it is allowed to create (overrides the process_count variable in the JSON configuration). The default value is the number of virtual cores on the host computer.

Errors and Debugging

In most situations, the source of errors will be related to the external model executed by the backend. When an error occurs with either the running of the backend model process or in reading the output of the backend model, an error will be generated and the related files will be preserved in the reported path for debugging.

Under the hood of the MODTRAN Tape backend

The "MODTRAN tape" backend creates a folder for all the MODTRAN executions (named modtran_runs) within which each model execution is stored in a sub-folder using a simple numerical numbering scheme from 0 to N. The backend will purge the contents of the model request folders after the model has been successfully run and the output ingested. However, these folders can be preserved by using the --save_files command-line option to the builder.

The file listing below shows the contents of a preserved model request folder after execution:

$ ls modtran_runs/0
README.txt              modtran_stderr.txt      tmp.7sc
mod5root.in             modtran_stdout.txt      tmp.tp5

The README.txt file contain a "human readable" description of the inputs to MODTRAN. The mod5root.in contains the basename of the input/output files (in the case of this backend, it is always tmp), the modtran_stderr.txt and modtran_stdout.txt files contain the standard error and standard output (console) of the MODTRAN program, respectively. The tmp.tp5 and tmp.7sc files are the respective input and output files of the model execution.