Oh, do I miss out_track

[blesht]

I'd appreciate some help.  For years (and years and years) I've used out_track (running SeaDAS6.4) to execute the following process.  Given a set of n fixed station locations and a series of m L2 images, I want to create a table (flat file) listing the mean, standard deviation, and number of valid (non-flagged) values of a retrieved variable (e.g. chlor_a) in an l x l pixel box surrounding each station location.  The output table has m rows (one for each image) and each row has (in addition to columns with file identification information, etc.) 3n columns.  This has worked for SeaWiFS data and still works for MODIS data.  However, because the script I use depends on the old V6.4 'load' function to pass the data to out_track I have not been able to process VIIRS data the same way; 'load' does not recognize VIIRS L2 files.  From previous queries (complaints?) I've made, I've been pointed to the SeaDAS7.x pixel extraction tool and to matchup-related scripts (fd_matchup.py and mk_matchup.py), but I don't see how these can be adapted to my purpose.  Is there any hope of reproducing my process within SeaDAS7.x (if so how?) or will I have to develop my own code outside of SeaDAS?

Thanks, Barry

[jpsco]

Hi Barry,

As you suspect, I believe the mk_matchup.py script has just what you need. That script was designed to operate on SeaBASS files, but those type of files are simply ASCII files with meta-data headers that start with a "/" followed by a delimited data matrix.

I envision you could construct a psuedo-SeaBASS file (having a reduced set of headers, only the required ones for parsing) with the following contents:

/begin_header
/missing=-999
/delimiter=comma
/fields=year,month,day,hour,minute,second,lat,lon
/end_header
2018,11,5,13,45,32,46.34,-9.49
...

You would then define one row per station you desire to extract MODIS/SeaWiFS/VIIRS/etc data for, and save that file as a plain text file.

Note, there are a number of ways to represent time with SeaBASS fields: date/time, year/month/day/hour/minute/second, year/month/day/time, date/hour/minute/second, date_time, year/sdy/hour/minute/second, or year/sdy/time, all in GMT and each with a specific data format. But fundamentally, all you need are date, time, latitude, and longitude information.

Once you have your pseudo-SeaBASS file with your n station targets as separate rows, you would then call the mk_matchup.py Python3.x script, which is distributed with the OCSSW build inside SeaDAS7.x, like so:

> mk_matchup.py --sat_file <L2_fname1.nc> [<L2_fname2.nc> ...] --seabass_file <your_pseudo-SeaBASS_fname.txt>

The mk_matchup.py script applies the default exclusion criteria (5x5 satellite extract, L2flag filters, etc) of Bailey & Werdell (2006), but these are also customizable using command line options. Call mk_matchup.py with the -h or --help options to explore these in detail.

Note, that you can pass in multiple L2 satellite files via the command line argument --sat_file, simply separate the paths to the files by a space. The same is also true for --seabass_file should you need to run this command on multiple psuedo-SeaBASS files.

The mk_matchup.py script will append the L2 satellite data as additional columns to your pseudo-SeaBASS (comma-delimited, text) file, including the filtered mean and stdev, as well as L2 file identification information.

Once you have all that information appended into your pseudo-SeaBASS file, you may use any one of our SeaBASS file readers (Python, Perl, Matlab) to interact with the data, or simply open the file with a text editor or spreadsheet program.

Note, the Python SeaBASS reader (SB_support.py and its readSB class) is included in the OCSSW build within SeaDAS already.

I do admit this might be a bit of a change to your prior workflow, but I believe this will still get you the functionality you desire.

Please reply to this forum post, should you have any questions.

Cheers,
Joel

Ref: Bailey, S.W., and Werdell, P.J. (2006). A multi-sensor approach for the on-orbit validation of ocean color satellite data products. Rem. Sens. Environ. 102, 12-23.

[blesht]

Hi Joel - thank you for the detailed reply.  I'll need a bit of time to think more about it and to go over the mk_matchup.py documentation again before seeing if I can adapt it to my purpose, but I do have a couple of questions now, if you don't mind.

1.  I am assuming that in the 'normal' application of mk_matchup.py the dates and times in the SeaBASS file represent the time at which the field sample was collected.  In my case, in which my goal is to extract the satellite estimate at a sample location (no field sample involved), would I use the  L2 image time?  If that is right, my 'pseudo-SeaBASS' file would have, in addition to the header lines, n x m lines.  In a typical application of mine that might result in a file with 6000 lines.  Is there any limit to the length of the SeaBASS file?

2.  Rather than pass a number of image files to mk_matchup.py by the --sat_file parameter, can the list of images be put into a file and that file name passed?  That is, rather than
           > mk_matchup.py --sat_file <L2_fname1.nc> [<L2_fname2.nc> ...] --seabass_file <your_pseudo-SeaBASS_fname.txt>
would
           > mk_matchup.py --sat_file <my_list_of_image_files.txt> --seabass_file <your_pseudo-SeaBASS_fname.txt> work?

I'm thinking the <my_list_of_image_files.txt> would be a simple text file with lines like:

L2_fname1.nc
L2_fname2.nc
L2_fname3.nc
.
.
L2_fnameN.nc

Barry

[jpsco]

Hi Barry,

There is no maximum length of SeaBASS file (within practicality -- all OS's have maximum file size/hard drive/RAM constraints). However, the larger the file, the longer it will take to generate matchups. But, yes, without an actual in situ observation, you may use the L2 image time, or multiple times, just keep in mind that the mk_matchup.py script will be nominally looking for coincident satellite within +/- 3 hours of the target time in the SeaBASS file.

No, mk_matchup.py is not configure to parse a list-type file of filenames. However, a simple bash or python script makes it easy to programmatically make command line calls of mk_matchup.py with the contents of an 'ls' command (bash) or with the contents of a list of files and a join statement (python). Both python and bash scripts are highly recommended for use with OCSSW command-line tools, especially for multi-layer processing/binning/gridding type workflows.

Cheers,
Joel

[blesht]

Hi Joel,

Thanks again.  Building a script for this process was always a given.  I have scripts for all my regular tasks and, fortunately for me, their use of the underlying OCSSW codes has been pretty stable across all versions of SeaDAS.  For this particular application, using my_markup.py will require a bit more involved coding but I think you've pointed me in the right direction.   I still do miss out_track, but I much appreciate your suggestions.  Got to move into the 21st century, I guess.

Regards, Barry

[blesht]

Joel - One more quick question.  I don't see anything in the mk_matchup.py documentation that allows specification of the desired L2 variable (or variables).  Does that mean that every geophysical variable in the L2 file will be extracted at the station location?

Thanks, Barry

[jpsco]

Barry, you are correct. All variables for a given L2 file will be output by mk_matchup.py. Variable specification is a feature request that is currently under development and will likely make it into a future SeaDAS release.

Cheers,
Joel

[blesht]

Hi Joel.  Sorry to bother you again.  I tried running mk_matchup.py and am getting what I think is a python error.  I'm not a python user so am at a loss as to how to debug this.

Barry

bash-3.2$ python --version
Python 3.7.4
bash-3.2$ ./mk_matchup.py --sat_file=/Volumes/4TBDrive/VIIRS/L2/Erie/2019/V2019194173600.L2.SNPP_OC.x.nc --seabass_file=/Users/blesht/Desktop/ErieSBassFile.txt --out_file=/Users/blesht/Desktop/testmatch.txt
Looking for satellite/in situ match-ups for /Users/blesht/Desktop/ErieSBassFile.txt in /Volumes/4TBDrive/VIIRS/L2/Erie/2019/V2019194173600.L2.SNPP_OC.x.nc
Traceback (most recent call last):
  File "./mk_matchup.py", line 647, in <module>
    if __name__ == "__main__": main()
  File "./mk_matchup.py", line 327, in main
    pid = subprocess.run(sys_call_str, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
AttributeError: 'module' object has no attribute 'run'

[jpsco]

Hi Barry, It looks like your Python installation does not have the Subprocess module, used for making calls to executables, configured. My recommendation to solving most Python module dependencies is to use a Python package manager (such as Anaconda) to install, manage, and update your Python environment.

Cheers,
Joel

[blesht]

Hi - Well, if nothing else, I'm sure learning a lot from our exchange.  :wink:  I've been using MacPorts, but I'll give Anaconda a go.  Thanks!