rtdin [ parameter=value ... ] [ outputfile ... ] rtdin [ parameter=value ... ] [ directory ]
Parameters are: dmsp_types, on_pass_disk, tape_device, pass_number, files_per_pass, pass_date, pass_time, satellite, ols_res, temp_units, use_master, master_file, start_time, fine_lines, fine_samples, smth_lines, smth_samples, ss_calib, glob, t1_limb, override_form, debug_time, sync_time, sync_bits, time_diag, max_gap, archive_time, sat_time_test, dmdm_len, correct_wf, ols_pix_off.
rtdin creates TeraScan Optical Linescan System (OLS), Special Sensor Microwave Imager (SSM/I), Special Sensor Microwave Temperature Sounder (SSM/T1), Special Sensor Microwave Water Vapor Sounder (SSM/T2), SSJ4, and Direct Mode Data Message (DMDM) datasets from the Defense Meteorological Satellite Program (DMSP) real-time telemetry data (RTD).
Input telemetry data can be read from the pass disk, tape, or standard UNIX disk files. Multiple passes on tape or the pass disk can be processed at once. In the case of UNIX disk files, additional "header" information needs to be provided by the user in order to process the pass.
The OLS sensor data is composed of two channels: a visible and a thermal channel. The image data from each of these channels has approximately the same spatial resolution (< 0.5km). However, in the RTD telemetry, the data from one of these channels is averaged in the scan direction (every five pixels) and output with a sampling resolution that is lower in the scan direction by a factor of five. For example, most often for a night pass, the visible data will have low resolution and the thermal channel will have high resolution, vice versa for day passes (high latitudes are special cases). The following illustrates the line and sample characteristics of the OLS data in the DMSP telemetry stream:
High-Res Low-Res
line 1: a b c d e C (average of five samples)
line 2: f g h i j H
line 3: k l m n o M
line 4: p q r s t R
line 5: u v w x y W
The data resolution is 6-bits for the high-resolution data and 8-bits for the low resolution data. Each is stored as byte data.
The parameter ols_res gives the user several options which determine how rtdin is to extract and store the high and low resolution data. For an input pass that contains 5*M lines and 5*N samples of high resolution data, and thus 5*M lines and N samples of low resolution data, the following describes the extraction options available (note the aspect ratios are not the same for all options):
High resolution Low resolution
--------------- --------------
lines samples lines samples
--------------- ---------------
ols_res=max:
5*M 5*N 5*M N
no subsampling no subsampling
or averaging or averaging
ols_res=hi:
5*M 5*N M N
no subsampling subsample every
or averaging fifth row
ols_res=mix:
5*M N 5*M N
average every five no subsampling
pixels in scan direction or averaging
ols_res=lo:
M N M N
average every five subsample every
pixels in scan direction fifth row
subsample every
fifth row
See NOTES for a discussion on other extraction options.
The overall spatial subset of the OLS data to be extracted can be specified in one of two ways:
1) the smallest rectangular area required to cover a region defined by a master dataset (see master).
2) a rectangular area of specific size, beginning at a user-specified time and scan sample.
The SSM/I is a passive microwave radiometer with the following seven channels: 19v, 19h, 22v, 37v, 37h, 85v and 85h GHz. The SSM/I uses a conical scan so all footprints of a given frequency along a scan have the same dimensions. There are 128 samples per scan for the two 85 GHz channels and 64 samples per scan for the lower frequency channels. The spot size varies from about 50km for the 19 GHz channels to about 15 km for the 85 GHz channels. The SSM/I is primarily used to estimate various geophysical properties of the Earth and/or atmosphere. These include total columnar atmospheric precipitable water, ocean surface wind speed, sea ice concentration, etc. The TeraScan functions miedr and geoph convert SSM/I brightness temperatures into these, and other, geophysical quantities. The SSM/I sensor is only available on the f-8, f-10 and f-11 satellites (not on f-9).
The SSM/T1 is a passive microwave radiometer with the following seven channels: 50.5, 53.2, 54.35, 54.9, 58.4, 58.825 and 59.4 GHz. In the SSM/T1 output dataset these are named "mt1_1"..."mt1_7". Channel 1 is the window channel, and each higher channel "peaks" at, or has greatest sensitivity to, higher altitudes in the atmosphere, up to 22 km for channel 7. The SSM/T1 scans 7 samples cross-track, and has a spot size of approximately 175 km at nadir. The seven channels can be inverted using the TeraScan function t1edr to estimate the temperature and pressure height profiles of the atmosphere.
The SSM/T2 is a five channel passive microwave radiometer. It has three channels situated symmetrically about the 183.310 GHz water vapor resonance line, 180.310 +/-1, 183.310 +/-3, 183.310 +/-7, and two window channels, 91.655 and 150 GHz. In the SSM/T2 output dataset these are named "mt2_1"..."mt2_5". The SSM/T2 scans 28 samples cross-track, has a spot size of approximately 45 km at nadir, and essentially has 4 times the spatial resolution of the (collocated) SSM/T1. These channels can be inverted using the TeraScan function t2edr to estimate the water vapor, specific humidity, and relative humidity profiles of the atmosphere. The SSM/T2 sensor is only available on the f-11 satellite.
Note: it is known that the OLS glare obstruction bracket on the DMSP spacecraft produces signal contamination at the edges of the SSM/T1 and SSM/T2 scan lines during terminator orbits. See parameter glob below.
The SSJ4 sensor is used to measure electron and ion counts in the atmosphere. It is a nadir viewing sensor with one sample per line per second. Data output variables are ion_counts and electron_counts. These are measured for twenty energy levels from 0.948 to 948 Kev. See level dimensions and energy_level variables in the SSJ4 output datasets. Additionally, because of the nadir sampling, latitude and longitude variables for each of the sample locations is computed and provided within the output datasets.
When ingesting special sensor data, rtdin extracts the entire pass of for the special sensors listed above. The extent that calibration is performed on the data can be specified. Special sensor data is often referred to as sensor data records (SDRs) when in the form of calibrated brightness temperatures, and environmental data records (EDRs) when converted to geophysical parameters.
The DMDM is a text message containing the orbital elements for the sending satellite along with other "special messages." These special messages can take one of eight forms that include information on orbital elements for other DMSP or NOAA satellites, changes in the OLS thermal/light fine schedule, launch dates for new satellites, etc. The DMDM message is stored as six-bit ASCII characters in the DMSP real-time data telemetry, approximately one character per OLS scan line. The message is 184 characters and therefore repeats throughout the pass. The DMDM message is simply extracted and stored in a TeraScan dataset. The information in this message can be displayed using getdmdm.
There are no (user-supplied) input TeraScan datasets. The sources for input telemetry data are described above. Output is either to a user supplied directory or a user-supplied list of dataset names. If a directory name is entered, output names will be automatically generated and the files will reside in the specified directory. Names will be generated that look like f#.yyddd.hhmm.type, where # is the DMSP satellite number, yyddd is the year and julian date, hhmm is the start hour and minute of the output dataset, and type is the type of data produced [ols, dmdm, mi_s (=ssmi), mt1_s (=ssm/t1), mt2_s (=ssm/t2), j4_s (=ssj4); where the "_s" indicates sensor data record]. If a directory name is not entered, the output filename(s) given will be used with the type of dataset appended to the end of the file.
In processing the telemetry data, rtdin looks for a "good" time value to attribute to the start of the pass. A time is considered "good" when three consistent and consecutive times have been found. This "good" time helps ensure that rtdin begins processing after any "noise" at the beginning of the pass. More importantly, it allows the earth location of the data to be performed more accurately.
In establishing a "good" start time, rtdin first tries to use the satellite times embedded in the telemetry. If they cannot be extracted for some reason, rtdin will then try to use the computer/receiver clock times (hereafter archive times) that were stored during the real-time telemetry retrieval. If these are also not available then rtdin uses the time in the pass header that was put in by the operator (see parameters sync_time, sync_bits, time_diag, max_gap, archive_time, debug_time, and sat_time_test below).
This specifies the type(s) of data to be extracted from the raw DMSP telemetry. Note, more than one type can be extracted during a single invocation.
Valid responses are [ols, ss, dmdm, all]. The default is all.
Answer yes if the input is from the pass disk, or no if the input is from tape or standard UNIX disk file.
Valid responses are [yes or no]. The default is yes.
If on_pass_disk=no, this is the name of the input tape device or UNIX disk file name. Tape device names are machine specific. For example, the following devices are commonly used on the Sun SPARC Station:
/dev/nwd0 - DAT, fixed blocking, APUNIX driver /dev/nrwd0 - DAT, variable blocking, APUNIX driver /dev/nsx0 - 8mm, fixed blocking, APUNIX driver /dev/nrsx0 - 8mm, variable blocking, APUNIX driver /dev/nrmt - 8mm or CCT, variable blocking, SUN driverAll of the above device names imply no rewind on close.
The default is extracted from the UNIX environment variable TAPE. Only tape devices listed in the file $PASSDIR/devtable, or simple UNIX disk file names, are accepted.
This specifies the number(s) of the pass(es) to process, when input is from tape or the pass disk. Passes are numbered starting with 1. Input tape is always assumed to be positioned at the first pass.
Valid responses for input from the pass disk are [1 to maximum number of passes that can be stored on your pass disk (see lspass(1))]. The default response is the number of the last acquired pass.
Valid responses for input from tape are [1 to 100]. The default response is 1.
If input is from tape, this is the number of tape files per pass. If files_per_pass=2, then each pass is assumed to consist of a header file, followed by a data file. If files_per_pass=1, then the tape is assumed not to have any header files. In this case, the user must supply the additional "header" info using the three parameters listed immediately below.
Valid responses are [1 or 2]. The default response is 2.
If files_per_pass=1 or the pass is from a UNIX disk file, this is the date of the pass to be processed. See formats for valid date formats.
If files_per_pass=1 or the pass is from a UNIX disk file, this is the start time of the pass to be processed. See formats for valid time formats.
If files_per_pass=1 or the pass is from a UNIX disk file, this is the DMSP satellite name of the pass to be processed.
Valid responses include [f-8, f-9, f-10, f-11, f-12, f-13].
This determines how rtdin extracts and stores the high and low resolution data. The different options are described above in the DESCRIPTION. See NOTES for a discussion on other extraction options. Since for ols_res=[lo, mix], each pixel is the average of five samples, the data resolution is increased from 6 to 8 bits.
Valid options are [max hi mix lo]. The default is hi.
Specifies the temperature scale of the OLS thermal calibration. The thermal calibration comes from a fixed scaling determined before launch.
Valid options are [kelvin celsius fahrenheit]. The default is celsius.
Answer yes if the OLS data is to be selected from the intersection of a master dataset (number 1 in the description above, also see master). Answer no if subsets are to be specified by start time and sample (number 2 in description above).
Valid responses are [yes or no]. The default is yes.
If use_master=yes, this is the name of the master dataset used to specify a region for OLS data extraction (see master).
Valid responses are any TeraScan dataset that contains a map projection-based earth transform. The default is Master.
If use_master=no, this is the start time for OLS data extraction. The first line with a time greater than start_time is the first OLS output line. A response of 00:00:00 instructs rtdin to begin processing at the first scan line. DMDM and special sensor data start at the first valid time (see description above).
Any valid time is allowed (see formats). The default is 00:00:00.
If use_master=no, dmsp_type=[ols all] and ols_res=[hi max mix], this is the number of fine OLS lines to be written to the output OLS dataset. If the end of pass is detected before the output dataset is complete, the output dataset is truncated. The number of smooth lines written to the output dataset depends on the specification of ols_res.
The valid range is [>=1]. The default is 2000.
If use_master=no, dmsp_type=[ols max] and ols_res=[lo], this is the number of fine and smooth OLS lines to be written to the output OLS dataset (since they have the same resolution with respect to the line dimension for this ols_res option). If the end of pass is detected before the output dataset is complete, the output dataset is truncated.
The valid range is [>=1]. The default is 2000.
If use_master=no, this is the first sample to extract from the pass. Samples are numbered starting with 1.
The valid range is [1 to 7340] if ols_res=[hi max]. The valid range is [1 to 1468] if ols_res=[lo mix]. The default is 1.
If use_master=no and ols_res=[hi max], then this is the number of fine samples written to the output set. The number of smooth samples will be less by a factor of five (see DESCRIPTION).
The valid range is [>= 1 <= 7340]. The default is 7340.
If use_master=no and ols_res=[lo mix], then this is the number of fine and smooth samples written to the output set.
The valid range is [>= 0 <= 1468]. The default is 1468.
This parameter specifies the extent to which calibration is performed on the special sensor data. When counts is specified, the raw sensor counts are output along with the calibration information needed to convert the counts into antenna temperatures. When antenna_temp is specified, antenna temperatures are output. When bright_temp is specified, calibrated brightness temperatures based on the algorithm of Wentz (See NOTES) are output. When hughes_bt is specified, calibrated brightness temperatures based on the algorithm of Hughes Aircraft, Space and Communications group, are output.
Valid responses are [counts, antenna_temp, bright_temp, or hughes_bt]. The default is bright_temp.
OPTIONAL. Determines which, if any, of the SSM/T1 and SSM/T2 "edge" samples to set to bad value (see datasets) due to the signal contamination produced by the OLS glare obstruction bracket (GLOB) that is in place on the spacecraft during terminator orbits. A value of 0 indicates all samples should be processed, a value of 1 indicates SSM/T1 sample 7 and SSM/T2 samples 25-28 should not be processed (i.e. set to bad value), and a value of -1 indicates SSM/T1 sample 1 and SSM/T2 samples 1-4 should not be processed.
Valid responses are [-1, 0, 1]. The default value is [0]. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. Instructs rtdin to apply a limb correction to the SSM/T1 antenna or brightness temperatures (see calibration above). This limb correction should not be applied to data that will be used as input to the t1edr(1) retrieval program but only to data used for qualitative study or "imaging". The corrections are a function of channel and beam position, and were extracted from the DMSP Processing Guide, written by Aerospace Corporation, El Segundo, CA.
Valid responses are [yes or no]. The default value is [no]. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. Determines whether the special sensor block format indicated in the telemetry should be overridden and the "default" formatting used. The (real-time) data block format can differ from the default for tactical or other reasons, however in some cases it may just be bit errors suggesting a different format. In this latter case, the user should override the format.
Valid responses are [yes or no]. The default value is [no]. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. Instructs rtdin to print out the embedded times it encounters in processing the telemetry.
Valid responses are [yes or no]. The default value is [no]. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. Specifying yes instructs rtdin to try and find a good start time for the pass using the times embedded in the telemetry or the archive times stored when the pass was acquired. Specifying sync_time=no, instructs rtdin to bypass the satellite and clock times and use the operator given time (the time in the pass header). Note: if telemetry embedded times are not used, special sensor data cannot be processed.
The default value is yes. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. Determines the number of bit errors allowable for obtaining sync with the special sensor data block's 64-bit sync word. If syncbits=0, then the 64-bit sync word must match the predefined pattern exactly. If syncbits=N, then N out of 64 bits may be incorrect and the sync word will still be accepted. This parameter may help in extracting blocks of special sensor data that may be lost due to noisy telemetry, i.e. bad sync words. Note: the processing time will likely increase for syncbits > 0.
Valid range is [>=0 and <= 64]. The default is 0.
OPTIONAL. Determines whether rtdin prints diagnostic messages regarding time gaps in the telemetry and the insertion of missing data.
Valid responses are [yes or no]. The default value is [yes]. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. This specifies the number of seconds that is to be considered a valid time gap in the telemetry stream. If a time gap occurs that is less than this time, rtdin attempts to insert the proper amount of "missing" data. If a time gap occurs that is more than this time, rtdin assumes the apparent time gap is caused by a bit error in the time word and ignores it.
Valid responses are integers [> 0]. The default value is [90]. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. Setting archive_time=yes overrides the use of the embedded satellite times and instructs rtdin to use the clock times archived into the data at the time of acquisition to find a "good" start time for the pass. If it does not find a valid archive time after sat_time_test seconds, it will then use the operator time in the pass header (see DESCRIPTION).
Valid responses are [yes or no]. The default value is [yes]. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. This is the number of seconds to search for valid satellite and/or archive times at the beginning of the pass.
The default value is 90. This parameter can only be set by an explicit specification on the command line.
OPTIONAL. This specifies to the number of DMDM characters to extract from the pass. Since there is about one DMDM character per OLS line, and the DMDM message is 184 characters, and since getdmdm has to process three consecutive, consistent DMDM messages in order to provide an accurate DMDM, getdmdm requires that rtdin store a minimum of about 800 characters. Noise in the pass may require more characters to be stored in order to find three consecutive, matching messages.
Valid responses are [> 0]. The default is 2048.
OPTIONAL. This specifies rtdin to correct OLS wow/flutter. This procedure will produce better high resolution detail and sample matchup between successive fine lines. It also incurs a modest amount of processing time.
Valid responses are [yes or no]. The default value is [yes].
OPTIONAL. This is a (fine) pixel offset applied to successive OLS lines to try and correct any "zig-zag" effect in the data.
Valid responses are [-30 < and < 30]. The default value is [0].
Create all DMSP datasets associated with the first pass on the pass disk. This includes OLS, DMDM and all the special sensors. The area for data selection is specified by a master dataset.
[1]% rtdin output file(s) : char(255) ? [.] dmsp_types : char( 19) ? [all] on_pass_disk : char( 3) ? [yes] pass_number : int ( 2) ? [1] ols_res : char( 3) ? [hi] temp_units : char( 10) ? [celsius] use_master : char( 3) ? [yes] master_file : char(255) ? [Master] ss_calib : char( 12) ? [bright_temp] Processing pass 1 on pass disk. File 1, pass 1, satellite f-11, date 92/12/10, time 01:27:00 8792 fine OLS lines; about 391 SSMI, 23 SSMT1 and 93 SSMT2 lines. Good data starts at OLS fine line 271 Operator given pass start time 01:27:00 Satellite given pass start time 01:29:33.295 Difference in pass start time 00:02:33.295 Revised OLS start time is 01:33:49.29. OLS visible data is high resolution, thermal data is low resolution. End of file encountered. Available data is being processed. TDF dataset ./f11.92345.0129.mi_s created with 416 lines. TDF dataset ./f11.92345.0129.t1_s created with 21 lines. TDF dataset ./f11.92345.0129.t2_s created with 81 lines. TDF dataset ./f11.92345.0129.dmdm created with 2048 DMDM characters. TDF dataset ./f11.92345.0133.ols created with 1025 fine and 205 smooth lines.
Extract the special sensor data from a pass on tape. Note, since this is an f-10 pass there is no SSM/T2 data.
[2]% rtdin output file(s) : char(255) ? [.] dmsp_types : char( 19) ? [all] ss on_pass_disk : char( 3) ? [yes] no tape_device : char(255) ? [/dev/nht0] files_per_pass : int ? [2] pass_number : int (100) ? [1] ss_calib : char( 12) ? [bright_temp] Processing pass 1 on /dev/nht0. File 1, pass 1, satellite f-10, date 92/12/06, time 05:08:10 8360 fine OLS lines; about 372 SSMI, 22 SSMT1 and 0 SSMT2 lines. Good data starts at OLS fine line 196 Operator given pass start time 05:08:10 Satellite given pass start time 05:08:09.57 Difference in pass start time -00:00:00.43 Time count indicated 26 second gap. Missing SSMI data at or near lines 13 through 25. Missing SSMT1 data at or near lines 2 through 3. Time count indicated 102706 second gap -> ignored. End of file encountered. Available data is being processed. TDF dataset ./f10.92341.0508.mi_s created with 376 lines. TDF dataset ./f10.92341.0508.t1_s created with 23 lines.
$TSCANROOT/refdata/satel/f-*/orbdata $TSCANROOT/refdata/satel/f-*/ols $TSCANROOT/refdata/satel/f-*/ssmi $TSCANROOT/refdata/satel/f-*/ssmt1 $TSCANROOT/refdata/satel/f-*/ssmt2
dmsp, ssmt, ols, dmdm, ssmi, datasets, formats, etx, getdmdm, rdsin, sdfin, geoph, t1edr, t2edr, miedr, subsamp, magnify, fixline, master.
An attribute called telemetry is written to OLS output datasets to distinguish datasets derived from rtd, rds, sds and sdf telemetries.
Other extraction options can be accomplished through the use of those outlined in the DESCRIPTION section followed by the application of the magnify function.
For example, when using the option high_res=max, one may wish to replicate (or interpolate) the low resolution data pixels in the sample direction so that each channel has 5*M lines and 5*N samples. This can be done by applying the magnify function to the low resolution variable with x_scale=5.0 and method=[nn bl].
Another option that might be desirable is to have M lines and N columns of both high resolution data, similar to the high_res=no option but using averaging instead of subsampling. This can be done using the high_res=mix option followed by the application of magnify to both the low and high resolution variables with y_scale=0.2 and method=av.
Warning: In the later stages of the F-9 spacecraft, the OLS thermal data went out of specification and the temperatures may be incorrectly calibrated. Correction tables are available that can be applied to correctly calibrate the data if necessary.
DMSP Data Specifications, IS-YD-821B, 7 March 1991. Special Sensor Microwave/Imager User's Guide, NRL Rep. Branch Rep., Sept. 14, 1987. System Summary Report Passive Microwave Sounder (SSM/T). Aerojet, Report 5542, CDRL A006, 23 November 1977. System Summary Report for the SSMT2 Water Vapor Profiling Sensor Hardware Segment. Aerojet, Report 9538, CDRL 035A2, 18 June 1990. Hollinger, J.P., 1988:DMSP special sensor microwave imager calibration/validation. Naval Res. Labs, Wash. D.C. Wentz, F., 1987, JGR, Vol.91, C2, pg. 2289. DMSP Processing Guide, Aerospace Corporation, El Segundo, CA.
Last Update: $Date: 1998/06/24 20:48:56 $