OCSSW satellite validation matchup tools: val_extract failed

[arufasblanco]

Dear EarthData team,

I am currently using the SeaBASS satellite validation match-up tools (https://seabass.gsfc.nasa.gov/wiki/validation_matchup_tools) for a project involving the matching of in situ chlorophyll observations in the North Sea with satellite data from MODIS-Aqua, MERIS, VIIRS-SNNP, and VIRRS-JPSS1. While fd_matchup.py is running smoothly, I have encountered an issue with mk_matchup.py that I'm struggling to resolve.

The issue appears to be associated with val_extract. In my bash_profile script, I have configured the following, as outlined in the documentation at https://seabass.gsfc.nasa.gov/wiki/validation_matchup_tools:

export OCSSWROOT="/Users/Anna/SeaDAS/ocssw"
source $OCSSWROOT/OCSSW_bash.env
export PATH=$OCSSWROOT/bin/val_extract:$PATH

I can confirm val_extract is compiled and in the $PATH since
> which val_extract
outputs
/Users/Anna/SeaDAS/ocssw/bin/val_extract

I then run the mk_matchup.py function:

/Users/Anna/SeaDAS/ocssw/ocssw_src/src/matchup_tools/mk_matchup.py \
      --sat_file="$full_path_nc_file" \
      --seabass_file="$full_path_seabass_file" \
      --box_size=3

where "$full_path_nc_file" is the physical location in my computer where the .nc file is (and which was downloaded after running fd_matchup.py): /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc.

When I run it, I get the following message:

Looking for satellite/in situ match-ups for: /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/matchup_entries.sb
Checking: /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc

usage: mk_matchup.py [-h] --sat_file SAT_FILE [SAT_FILE ...] --seabass_file SEABASS_FILE
[SEABASS_FILE ...] [--box_size BOX_SIZE] [--min_valid_sat_pix MIN_VALID_SAT_PIX]
[--max_time_diff MAX_TIME_DIFF] [--max_coeff_variation MAX_COEFF_VARIATION]
[--slat SLAT] [--elat ELAT] [--slon SLON] [--elon ELON] [--verbose]
[--no_header_comment]
mk_matchup.py: error: val_extract failed -- verify that the val_extract binary is compiled and on your PATH and that /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc exists

If you could provide help with this I would be very grateful.
Regards,
Anna

[OB ODPS - towens]

Did you verify that /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc is a valid NetCDF file? If your login credentials are not accepted by the server, you can end up with an HTML file with an error message when you attempt to download data.

Tommy

[arufasblanco]

Dear Tommy,

I can confirm that the file /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc is a valid NetCDF file. I've done the verification by running in my terminal the following command prompt:

ncdump -h /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc > log_check_valid_nc_file.log

The first 20 lines of the log file show:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

netcdf AQUA_MODIS.20110520T132501.L2.OC {
dimensions:
number_of_lines = 2030 ;
pixels_per_line = 1354 ;
bands_per_pixel = 16 ;
number_of_reflectance_location_values = 10 ;
pixel_control_points = 1354 ;
number_of_bands = 24 ;
number_of_reflective_bands = 16 ;

// global attributes:
:title = "MODISA Level-2 Data" ;
:product_name = "AQUA_MODIS.20110520T132501.L2.OC.nc" ;
:processing_version = "R2022.0" ;
:equatorCrossingLongitude = 173.3716f ;
:orbit_number = 48105 ;
:history = "l2gen par=/data5/sdpsoper/vdc/vpu16/workbuf/A2011140132500.L1B_LAC.param metafile=AQUA_MODIS.20110520T132501.L2.OC.nc.meta" ;
:instrument = "MODIS" ;
:platform = "Aqua" ;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

What else could I try?

Many thanks,

Anna

[OB ODPS - towens]

does the --verbose option add any helpful information ?

Tommy

[arufasblanco]

Hi Tommy,
Since I'm using an old Mac OS (v. 11) I cannot upgrade netcdf to run ncdump it with the --verbose command. Instead, I've done another file verification using Python with netCDF4 library, as:

%%%%%%%%%%%%%%%%%%%

file_to_check = "/Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc"

import netCDF4

try:
with netCDF4.Dataset(file_to_check) as ncfile:
print(ncfile)
except Exception as e:
print(f"Error: {e}")

%%%%%%%%%%%%%%%%%%%

The above code has printed the below (i.e., the list of global attributes):

%%%%%%%%%%%%%%%%%%%

<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
title: MODISA Level-2 Data
product_name: AQUA_MODIS.20110520T132501.L2.OC.nc
processing_version: R2022.0
equatorCrossingLongitude: 173.3716
orbit_number: 48105
history: l2gen par=/data5/sdpsoper/vdc/vpu16/workbuf/A2011140132500.L1B_LAC.param metafile=AQUA_MODIS.20110520T132501.L2.OC.nc.meta
instrument: MODIS
platform: Aqua
Conventions: CF-1.6 ACDD-1.3
license: https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/
naming_authority: gov.nasa.gsfc.sci.oceandata
id: R2022.0/L2/AQUA_MODIS.20110520T132501.L2.OC.nc
date_created: 2022-09-07T22:53:59.000Z
keywords_vocabulary: NASA Global Change Master Directory (GCMD) Science Keywords
keywords: Earth Science > Oceans > Ocean Optics > Ocean Color
standard_name_vocabulary: CF Standard Name Table v36
institution: NASA Goddard Space Flight Center, Ocean Ecology Laboratory, Ocean Biology Processing Group
creator_name: NASA/GSFC/OBPG
creator_email: data@oceancolor.gsfc.nasa.gov
creator_url: https://oceandata.sci.gsfc.nasa.gov
project: Ocean Biology Processing Group (NASA/GSFC/OBPG)
publisher_name: NASA/GSFC/OBPG
publisher_url: https://oceandata.sci.gsfc.nasa.gov
publisher_email: data@oceancolor.gsfc.nasa.gov
processing_level: L2
cdm_data_type: swath
equatorCrossingDateTime: 2011-05-20T14:02:01.823Z
spatialResolution: 1 km
time_coverage_start: 2011-05-20T13:25:01.568Z
time_coverage_end: 2011-05-20T13:29:59.945Z
start_center_longitude: -5.5087404
start_center_latitude: 44.625565
end_center_longitude: -14.427519
end_center_latitude: 62.255764
northernmost_latitude: 63.70611
southernmost_latitude: 41.64226
easternmost_longitude: 9.333739
westernmost_longitude: -33.39269
geospatial_lat_units: degrees_north
geospatial_lon_units: degrees_east
geospatial_lat_max: 63.70611
geospatial_lat_min: 41.64226
geospatial_lon_max: 9.333739
geospatial_lon_min: -33.39269
startDirection: Ascending
endDirection: Ascending
day_night_flag: Day
earth_sun_distance_correction: 0.9766165614128113
dimensions(sizes): number_of_lines(2030), pixels_per_line(1354), bands_per_pixel(16), number_of_reflectance_location_values(10), pixel_control_points(1354), number_of_bands(24), number_of_reflective_bands(16)
variables(dimensions):
groups: sensor_band_parameters, scan_line_attributes, geophysical_data, navigation_data, processing_control

%%%%%%%%%%%%%%%%%%%

Many thanks,

Anna

[OB ODPS - towens]

I meant to add it to the matchup call:
mk_matchup.py --verbose <your command>

Tommy

[arufasblanco]

Hi Tommy,

Thanks! I followed instructions, but the addition of --verbose didn't add any new entry of information to the mk_matchup.py output, it still prints the following lines:

%%%%%%%%%%%%%%%%%%%%

The nc file is /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc
Looking for satellite/in situ match-ups for: /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/matchup_entries.sb
Checking: /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc
usage: mk_matchup.py [-h] --sat_file SAT_FILE [SAT_FILE ...] --seabass_file SEABASS_FILE [SEABASS_FILE ...] [--box_size BOX_SIZE]
[--min_valid_sat_pix MIN_VALID_SAT_PIX] [--max_time_diff MAX_TIME_DIFF]
[--max_coeff_variation MAX_COEFF_VARIATION] [--slat SLAT] [--elat ELAT] [--slon SLON] [--elon ELON]
[--verbose] [--no_header_comment]
mk_matchup.py: error: val_extract failed -- verify that the val_extract binary is compiled and on your PATH and that /Users/Anna/LocalDocuments/Academic/Projects/Agile/baseline_assessment/NASA_L2_data/data_matchups_oceancolor/SatData/AQUA_MODIS.20110520T132501.L2.OC.nc exists

%%%%%%%%%%%%%%%%%%%%

I can verify that the val_extract binary is on my PATH after running

> which val_extract
/Users/Anna/SeaDAS/ocssw/bin/val_extract

I am using SeaDAS version 8.4.0.

[arufasblanco]

Hi Tommy,
Has there been any progress on this?
Many thanks,
Anna

[OB ODPS - towens]

The SeaDAS team were all at a conference last week. They will be looking at it today.

Tommy

[OB SeaDAS - xuanyang02]

If you are using SeaDAS 8.4.0, I believe the right command is find_matchup and make_matchup.

Here are the help pages --
https://seadas.gsfc.nasa.gov/help-8.4.0/FieldMeasurements/FindMatchups.html
https://seadas.gsfc.nasa.gov/help-8.4.0/FieldMeasurements/MakeMatchups.html

Could you please try

Code: Select all

/Users/Anna/SeaDAS/ocssw/bin/make_matchup --sat_file="$full_path_nc_file" --seabass_file="$full_path_seabass_file" --box_size=3

If it still does not work, could you please share "$full_path_nc_file" and "$full_path_seabass_file", so we could try to replicate the error?