Programming Guide for RSL.

Design philosophy:

The design philosophy is simple. Routines are functions. Routines are short and simple. And, higher level routines are built upon lower level routines; a hierarchical modular design.

Each procedure in the RSL is a function that returns a pointer to an object that it is designed, or self documented, to return. In the case of scalar functions, scalar values are returned. There are exceptions to this rule wherein we have procedures that modify their arguments, but, these are few. Also, each function does one thing well. This makes the implementation of each function simple and easy to understand and, therefore, easy to maintain. The power of the library has come from building higher level functions from many lower level functions. Eventually, the highest level function takes a filename as an argument and returns a Radar pointer.

The Radar data structure is designed to contain values and header information making it a superset of all the radar data formats currently encountered. Currently RSL can ingest the following file formats: nexrad, lassen both 1.3 and 1.4 file format versions (MCTEX too), UF, toga, new sigment (nsig) both version 1 and version 2 file formats, mcgill, kwajalein and rapic (Berrimah data).
 

What's New?

Future plans.

There are no plans for added functionality to the library. However, there are always new functions that pop-up. The documentation must be improved. Other than that, bug fixes and optimizations will be incorporated. Therefore all furture releases are expected to be minor.

Can read compressed files automatically (new since v0.41)

Transparent, to all ingest routines the capability of filtering the input data file through the GNU gzip program for decompressing the file has been added. This feature does not appear to slow the I/O time and, in some cases, especially on 486 pc's, improves overall throughput time. Two generic, internal routines, have been added: compress_pipe and uncompress_pipe. Each routine takes a FILE * and redirects it through gzip. Each routine returns a new FILE *. Wsr88d files occupy 1/10 the disk space when compressed and the TRMM Office plans to compress wsr88d files to CDROM and 8mm tape for overall throughput savings for the production system. It will no longer be necessary to decompress the data before processing level 1.

Similiarly, UF output can be saved using the gzip filter. The new routine added is RSL_radar_to_uf_gzip which utilizes the gzip compression filter.

Can read stdin (new since v0.39)

A new routine is provided called RSL_uf_to_radar_fp which takes an open file descriptor and returns a radar pointer. That file pointer can be stdin. This special interface syntax, for the UF ingest, is an exception to the interfaces that the other ingest routines have. All other ingest routines, including RSL_uf_to_radar, have been modified to read stdin when the input filename is NULL. The complete list of routines is: RSL_wsr88d_to_radar, RSL_nsig_to_radar, RSL_lassen_to_radar, RSL_uf_to_radar, RSL_mcgill_to_radar and RSL_toga_to_radar. The only routine that will not accept stdin will be RSL_anyformat_to_radar. That routine will not be able to handle reading stdin because it needs to read the first few bytes of the file to determine which ingest routine to call. If you plan to make a filter program, you'll just have to know what file format you expect: UF, nsig, wsr88d, etc.

Verbose print mode:

The function RSL_radar_verbose_on() and RSL_radar_verbose_off() control whether or not the radar library prints diagnostic messages during execution. These routines simply toggle an externally defined variable, radar_verbose_flag. If you're writing a new library function and you want the user (another programmer) to control the printing of diagnostic messages or not, you simply place print statement as the consequence of testing the variable radar_verbose_flag. For example:
extern int radar_verbose_flag; /* Define before usage. */
 /* Somewhere in the code. */
 if (radar_verbose_flag) printf("I'm here now. Whatever.\n");

Writing methods (structure specific interfaces for routines):

The natural hierarchy of Radar makes it easy to construct routines that have interfaces that work on each of the substructures. When writing a function that manipulates the Ray data, it is easy to provide the interface for the Sweep, Volume and Radar without much extra effort. I will illustrate with an example that adds two Rays.
Ray *add_rays(Ray *ray1, Ray *ray2)
{
int i;
Ray *ray_new;
if (ray1 == NULL) return NULL;
if (ray2 == NULL) return NULL;
ray_new = RSL_new_ray(ray1->h.nbins); /* Allocate the new ray. */
if (ray_new == NULL) return NULL;
ray_new->h = ray_new->h; /* Copy the header information. */ 
for (i=0; i <= ray_new->h.nbins; i++) /* Add each range element. */
  ray_new->range[i] = ray1->range[i] + ray2->range[i];
return ray_new; /* Return this new ray. */
}
It is valid for the routine to accept NULL's. This should be checked and appropriate values returned. In the case above, we return NULL. We could easily return the ray that is not NULL, too. The choice is up to you. The routine is a function and returns a structure pointer similar to those it manipulates; or self documented to return. Space is allocated and header information is copied. You might ask, "Why don't we call RSL_copy_ray, then call RSL_clear_ray?". This is a viable approach and it will work well at the ray level, however, for the sweep, volume, and radar interfaces we don't want to recursively copy all substructures because it is more troublesome to copy, clear and set all the levels, than to simply allocate and set at each level. To construct the interfaces for Sweeps and Volumes is easy. For the Sweep interface we recognize that a Sweep is an array of Rays. You will notice that this routine is coded almost identically to the ray routine.
Sweep *add_sweeps(Sweep *s1, Sweep *s2)
{
int i;
Sweep *s_new;
if (s1 == NULL) return NULL;
if (s2 == NULL) return NULL;
s_new = RSL_new_sweep(s1->h.nrays);
s_new->h = s1->h;
for (i=0; i <= s1->h.nrays; i++)
  s_new->ray[i] = add_rays(s1->ray[i], s2->ray[i]);
return s_new;
}
The pattern continues for making the Volume interface.
Volume *add_volumes(Volume *v1, Volume *v2)
{
int i;
Volume *v_new;
if (v1 == NULL) return NULL;
if (v2 == NULL) return NULL;
v_new = RSL_new_volume(v1->h.nsweeps);
v_new->h = v1->h;
for (i=0; i <= v1->h.nsweeps; i++)
  v_new->sweep[i] = add_sweeps(v1->sweep[i], v2->sweep[i]);
return v_new;
}
And finally, for Radar.
Radar *add_radars(Radar *r1, Radar *r2)
{
int i;
Radar *r_new;
if (r1 == NULL) return NULL;
if (r2 == NULL) return NULL;
r_new = RSL_new_radar(r1->h.nvolumes);
r_new->h = r1->h;
for (i=0; i <= r1->h.nvolumes; i++)
  r_new->v[i] = add_volumes(r1->v[i], r2->v[i]);
return r_new;
}
These four functions allow us to add two radars, add two volumes, add two sweeps or add two rays. The development was simple. For the radar routine we loop on the number of volumes and call the volume routine. For the volume routine we loop on the number of sweeps and call the sweep routine. For the sweep routine we loop on the number of rays and call the ray routine. For the ray routine we loop on the number of bins and perform the work. Simple. This is how many of the RSL routines were written.

Using h.f and h.invf and defining your own function:

The header members f and invf are provided to allow you to define the conversion function for internal/float storage. These header members are in the structures Volume, Sweep, and Ray. f takes a value of type Range and returns a value of type float. invf takes a value of type float and returns a value of type Range. f and invf should be inverse functions. That is, c == invf(f(c)) and x == f(invf(x)). Range is a datatype that represents the type of data for internal storage. Typically, Range will be unsigned char or unsigned short depending on the configuration option USE_TWO_BYTE_PRECISION the file makefile during the build and install step of RSL. There are several predefined functions and they are based on the WSR88D, or NEXRAD, encoding. There is a function for each field type: reflectivity, velocity, spectral width, etc. However, only two distinct functions are provided because the other field types can use these functions as well: DZ_F (and DZ_INVF) and VR_F (and VR_INVF). DZ_F and DZ_INVF apply to reflectivity data and VR_F and VR_INVF apply to all remaining field types by default. There are no restrictions for f and invf. Typically, you either pick the default functions DZ_F, DZ_INVF, VR_F and VR_INVF and assign them to the header member h.f and h.invf, appropriately, in the routine that is ingesting data from disk and constructing the Radar data structure. You can define your own encoding functions and assign them to h.f and h.invf. To illustrate:
static Range invf(float x) {
 return (Range) (x/2 + 10);
}

static float f (Range x) {
 return (float) (x - 10)*2; /* Not quite a perfect inverse function. */
}

Volume *volume_routine(Volume *v)
{
 v->h.f = f; /* Assign the float function. */
 v->h.invf = invf; /* Assign the Range function. */
 return v;
}
So far in the construction of RSL, the only location where you assign h.f and h.invf is in the ingest routine for a particular data format. The routines RSL_uf_to_radar, RSL_lassen_to_radar, RSL_wsr88d_to_radar, RSL_nsig_to_radar, RSL_toga_to_radar, and RSL_mcgill_to_radar each define the invf and f functions; in some cases the default functions are used. The file volume.c contains the default specification for DZ_F, DZ_INVF, etc.

Image generation:

The image generation section was initially written to test and debug the development of the library. However, some of the image generation capabilities are becoming a permanent feature of RSL. There are two parts to image generation. One, defining the color table and two, making PPM images. Assigning colors is critical to making color images, otherwise the resultant images will be black. The color table in RSL is global to the image functions and is statically allocated so that once the color table is defined, that's it. There are several routines that load color tables, but the main idea is that there is a red table, a green table and a blue table. Each of these tables is a separate file on disk stored in /usr/local/trmm/lib/colors The /usr/local/trmm prefix may be different on your system. These are the default color files and so you can use any file you like. Each file can contain up to 256 bytes, each byte represents a color intensity in the range 0 - 255. The color are ordered, in the file, by color index. The first byte is for color index 0, the second byte for color index 1, etc. There is only one type of image made, PPM. Well, I can make PBM and PGM too, but I consider them similar to PPM. To make other types of images like GIF, PICT, etc. I pipe the output into the ppmtogif, ppmtopict, command appropriately. I do this because I don't want to include all the different conversion sources in RSL. If you don't have the pbmplus software on your system, you can still use these image functions by either making your own ppmtogif command which is a csh script with the sole command 'cat' in it or modify the RSL library routine to not pass the data through the pipe. Also, when making PGM files, I pipe them into gzip so that the output files are small. The image generation functions are designed to meet our specific needs and I cannot say that they will be suitable for your needs.

Writing a new ingest routine:

Here we explain how to interface a new file format into RSL. This means we read the file format and assign values to the appropriate structure members. The interface will return a pointer to Radar, allocating all memory. Or, it will return NULL indicating an error.
Radar *RSL_something_to_radar(char *infile);
The file added to RSL will be called something.c. All the code needed to ingest the file and create the Radar structure will be in that file. It may be necessary to rely on another library that can ingest the data file utilizing a nicer interface. That is ok, go ahead and use it. This new interface definition will be placed in the prototype section of rsl.h so that other routines are aware of it. Also, code to automatically determine the type of input file must be placed in anyformat.c. If it is not possible to automatically determine the type of file, then this must be documented as such and an explanation that when anyformat_to_radar returns NULL, it may be necessary to call RSL_something_to_radar directly. Memory allocation routines you will need are: RSL_new_radar, RSL_new_volume, RSL_new_sweep and RSL_new_ray. You may not know ahead of time just how many substructures to allocate, so allocate some reasonable maximum. You can reset the value of nvolumes, nsweeps, or nrays after the ingest is complete.
radar = RSL_new_radar(20);
...load radar->v[i] ...
...keep track of the max index for v[i] ...
radar->h.nvolumes = max_vol_index;
return radar;
You don't have to worry about deallocating the extra memory. Output routines -- to disk -- will ignore it. And, after re-reading it, the right amount of memory will be allocated.

New in v0.41 is the implementation of the radar structure member radar_type. Assign a descriptive string indicating the origin or the format type of the data. Currently used strings are: "wsr88d", "lassen", "nsig", "uf", "mcgill", "toga", "kwajalein". The string may not exceed 50 characters, including the null character.

Common problems that have occurred are:

  1. Not setting the beamwidth member in the sweep header. This affects image generation because the pixel size relies on it.
  2. The number of rays, sweeps, or volumes is not set or exceeds the amount allocated.
  3. Not calling RSL_rebin_velocity_... before generating velocity images.
  4. Not calling the RSL_load_..._color_table(). This results in black images.

RSL_copy versus RSL_new:

It is important to note the differences between these two routines. RSL_copy_{radar, volume, sweep, ray} returns a complete duplicate of the input structure. It duplicates all of the substructures. This means that it allocates the exact amount of memory that the argument occupies. If a Radar occupies 15 Mbytes of RAM, then a RSL_copy_radar will result in 15Mbytes of new memory allocated. The total memory occupied will be 30 Mbytes: 15 for the original and 15 for the copy.

On the other hand, RSL_new_{radar, volume, sweep, ray} allocates memory for the header structure and only the pointers to the substructure. You have to allocate the substructure yourself. For instance, if you call

RSL_new_radar(4); /* Four volumes. */
you will have to call RSL_new_volume four times. Do that with,
for (i=0; i<=radar->h.nvolumes; i++) {
  radar->v[i] = RSL_new_volume(40); /* 40 sweep pointers. */
   ... construct the volume ...
}
Here only 40 pointers for the sweeps are allocated and not the space for the entire sweep which would include all the rays. For each sweep you must use the code,
for (i=0; i<volume->h.nsweeps; i++) {
  volume->sweep[i] = RSL_new_sweep(400);  /* 400 is a good max. */

  ... construct the sweep ...

}

Using RSL_new to copy instead of RSL_copy:

Let's say you want to create the triplet, or quadruplet, set of routines that function like the RSL_copy function in that they allocate new memory and return a pointer to the appropriate structure, but, the routines are executing an algorithm on the data and so the result is a mathematical manipulation of the input structure. In other words:
my_new_volume = RSL_some_function_of_volume(volume);
The natural hierarchical construction will be:
Radar *RSL_some_function_of_radar(radar);
Volume *RSL_some_function_of_volume(volume);
Sweep *RSL_some_function_of_sweep(sweep);
Ray *RSL_some_function_of_ray(ray);
As I've stated before, the radar routine will loop on the number of volumes and call the volume routine. The volume routine will loop on the number of sweeps and call the sweep routine. The sweep routine will loop on the number of rays and call the ray routine. The ray routine will perform the actual function. One approach might be to define the radar routine as:
Radar *RSL_some_function_of_radar(Radar *radar)
{
  int i;
  Volume *volume;
  Radar *new_radar;
  if (radar == NULL) return NULL;
  new_radar = RSL_copy_radar(radar);  /* This is bad, bad, bad. */

  for (i=0; i<=radar->h.nvolumes; i++)
    new_radar->v[i] = RSL_some_function_of_volume(radar->v[i]);

  return new_radar;
}
Now, why did I place the comment /* bad, bad, bad */ in the code? At first, it seems, it may not be all that bad. But, remember that RSL_copy_radar will copy all the volumes and sweeps and rays and bins. The line that assigns new_radar->v[i] is allocating space for a volume too. Therefore, we are more than doubling the memory requirements, if we code the volume and sweep routines similarly. The amount of memory allocated will be nvolumes*space_allocated_to_a_volume + nsweeps*space_allocated_to_a_sweep + nrays*space_allocated_to_a_ray. That is a tremendous amount of memory, you're talking 3*15*15Mbytes (675Mbytes). In order to use RSL_copy_{radar,volume,sweep,ray} you will need to clear out all the volumes, sweeps and rays with a call to RSL_clear_{volume,sweep,ray} and you will have to pass the target structure to the routine which will make it an argument that is modified. This goes against the design of the typical RSL interface where a pointer to an object is returned by a function. A better solution is to allocate a new radar and copy the header information. Instead of writing,
new_radar = RSL_copy_radar(radar);
you write,
new_radar = RSL_new_radar(radar->h.nvolumes);
new_radar->h = radar->h;
The volume, sweep and ray routine can be coded similarly. For example, here is the sweep routine:
Sweep *RSL_some_function_of_sweep(Sweep *sweep)
{
int i;
Ray *ray;
Sweep *new_sweep;
if (sweep == NULL) return NULL;
new_sweep = RSL_new_sweep(sweep->h.nrays);
new_sweep->h = sweep->h;

for (i=0; i<=sweep->h.nrays; i++)
  new_sweep->ray[i] = RSL_some_function_of_ray(sweep->ray[i]); 

return new_sweep;
}