Problem with mk_matchup.py

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
blesht
Posts: 86
Joined: Mon Sep 19, 2005 3:06 pm America/New_York
Answers: 0

Problem with mk_matchup.py

by blesht » Tue Oct 13, 2020 6:41 pm America/New_York

I'm getting errors running mk_matchup.py.  I don't understand them.  Can you help me figure out where to look for the problem(s), please?  The mk_matchup.py routine is invoked within a loop in a bash script.  This is the calling statement,

   ${SEA7}mk_matchup.py --sat_file=${SFIL} --seabass_file=${HFIL} --out_file=${OFIL} --min_valid_sat_pix=75. --no_header_comment
in which {SEA7} is a token for the /Applications/seadas-7.5/ocssw/scripts directory, and {SFIL}, {HFIL}, and {OFIL} are tokens with the satellite data file, SeaBASS data file, and output file respectively.

Here are the error messages.  Because "delim" is mentioned, I thought they might have something to do with the delimiter specified in the SeaBASS input file.  An example of that file is appended below as well.  I don't see a problem with it.

Traceback (most recent call last):
  File "/Applications/seadas-7.5/ocssw/scripts/mk_matchup.py", line 647, in <module>
    if __name__ == "__main__": main()
  File "/Applications/seadas-7.5/ocssw/scripts/mk_matchup.py", line 198, in main
    no_warn=True)
  File "/Applications/seadas-7.5/ocssw/scripts/modules/SB_support_v35.py", line 237, in __init__
    if not delim:
UnboundLocalError: local variable 'delim' referenced before assignment

/begin_header
/missing=-9.999
/delimeter=comma
/fields=year,month,day,hour,minute,second,lat,lon,station
/end_header
2020,1,2,17,06,00,42.53833,-79.61667,ER09
2020,1,2,17,06,00,42.68000,-79.69167,ER10
2020,1,2,17,06,00,42.51667,-79.89333,ER15M
2020,1,2,17,06,00,42.43000,-81.20500,ER30
2020,1,2,17,06,00,42.25333,-81.10667,ER31
2020,1,2,17,06,00,42.08167,-81.01167,ER32
2020,1,2,17,06,00,41.93500,-81.47833,ER36
2020,1,2,17,06,00,42.11000,-81.57500,ER37
2020,1,2,17,06,00,42.28167,-81.67167,ER38
2020,1,2,17,06,00,41.96500,-82.04167,ER42
2020,1,2,17,06,00,41.78833,-81.94500,ER43
2020,1,2,17,06,00,41.68500,-82.93333,ER58
2020,1,2,17,06,00,41.72667,-83.15000,ER59
2020,1,2,17,06,00,41.89167,-83.19667,ER60
2020,1,2,17,06,00,41.94667,-83.04500,ER61
2020,1,2,17,06,00,42.41667,-79.80000,ER63
2020,1,2,17,06,00,41.97778,-81.75694,ER73
2020,1,2,17,06,00,42.11667,-81.25000,ER78M
2020,1,2,17,06,00,41.84083,-82.91667,ER91M
2020,1,2,17,06,00,41.95000,-82.68667,ER9

Thanks!

Tags:

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

Problem with mk_matchup.py

by gnwiii » Tue Oct 13, 2020 8:10 pm America/New_York

Is it possible that the SeaBASS input file has <CR><LF>  (DOS/Windows) line endings?  The file command may tell you:
% file t.txt
t.txt: ASCII text
% unix2dos t.txt
unix2dos: converting file t.txt to DOS format...
% file t.txt
t.txt: ASCII text, with CRLF line terminators

blesht
Posts: 86
Joined: Mon Sep 19, 2005 3:06 pm America/New_York
Answers: 0

Problem with mk_matchup.py

by blesht » Wed Oct 14, 2020 9:37 am America/New_York

Hi George - Still keeping up with the forum; sounds like a good retirement activity to me!  Hope you're doing well.

Thanks for the suggestion.  Here is what I get with the file command:

bash-3.2$ file V2020274183000RegSeaBASS.txt
V2020274183000RegSeaBASS.txt: ASCII text

Because I'm using mk_matchup as a replacement for the late, lamented, v6.4 out_track I wrote an R script that runs through a list of images, parses out the date and time information, and combines that with the information from my list of stations to create the data portion of the SeaBASS input file.  Then I use a bash script to run through the image files, create corresponding filenames for the mk_matchup.py arguments, concatenate a header with the SeaBASS data, and finally call mk_matchup.py.

Here is the result of the file command with a concatenated (header + data) file:

bash-3.2$ file V2020002183000withHead.txt
V2020002183000withHead.txt: ASCII text

I'll play with it more this morning, but I expect Joel might have some suggestion as well.

Thanks again and take care, Barry

blesht
Posts: 86
Joined: Mon Sep 19, 2005 3:06 pm America/New_York
Answers: 0

Problem with mk_matchup.py

by blesht » Wed Oct 14, 2020 10:49 am America/New_York

OK - Once again I have proven to be a poor typist.  The problem was that I misspelled 'delimiter' in my header file.  Sorry for the bother.  All is well.

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

Problem with mk_matchup.py

by gnwiii » Wed Oct 14, 2020 11:34 am America/New_York

Glad you got it going.   I'm doing well in retirement -- we built a new home (not quite finished, but liveable) in a small Nova Scotia town.  Social isolation has not been hard for us, and my wife holds regular zoom get-togethers with people she normally sees once a year while travelling.  Nova Scotia (with the exception of early problems in the largest seniors' residence in the province) has done an excellent job of managing COVID-19 so far.   We are cyclists so get benefits of reduced auto traffic.

blesht
Posts: 86
Joined: Mon Sep 19, 2005 3:06 pm America/New_York
Answers: 0

Problem with mk_matchup.py

by blesht » Wed Oct 14, 2020 11:34 am America/New_York

Well, I spoke too soon.  All is not well.  Fixing the delimiter spelling corrected the earlier problem, and now mk_matchup runs.  But now I am getting "No valid satellite match-ups found for any lat/lon/time pairs in [SeaBASS filename]" for every single image.  Given that the time (all locations have the same time) in the SeaBASS file is created by parsing the satellite image filename, the time match can't be the problem.  Similarly, I have verified that the position of the stations I use are well within the images from which I'm extracting the data. 

Looking at the code, I find that this message is generated if write_flag != 1 but it's not clear to me how this flag is set (or not set) or if it refers to only the lat/lon/time pairs or also to any of the other exclusion criterion.  The only modification I did to these is to increase the percentage of acceptable pixels in the 5x5 box to 75% from the default 50%.

Are there other things I should be looking at?  Thanks.

blesht
Posts: 86
Joined: Mon Sep 19, 2005 3:06 pm America/New_York
Answers: 0

Problem with mk_matchup.py

by blesht » Wed Oct 14, 2020 1:24 pm America/New_York

I hope this will be the last of my posts on this subject.  I activated the --verbose optional argument for mk_matchup.py and found that the failures were due to time mismatches.  Then I went back to my R script that created the pseudo-SeaBASS files and found that in the process of converting the ordinal day from the L2 filename to month and day, I was one day off (not sure why).  After I corrected the R script and regenerated the pseudo-SeaBASS files, mk_matchup.py appears to be working as expected.  I'm still getting the warning message about "no units found in the header" but that's just an annoyance.

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Problem with mk_matchup.py

by OB.DAAC - SeanBailey » Fri Oct 16, 2020 9:34 am America/New_York

> I'm still getting the warning message about "no units found in the header" but that's just an annoyance.


Well, you could add a /units= header :)

See:https://seabass.gsfc.nasa.gov/wiki/metadataheaders#units

Sean

blesht
Posts: 86
Joined: Mon Sep 19, 2005 3:06 pm America/New_York
Answers: 0

Problem with mk_matchup.py

by blesht » Fri Oct 16, 2020 10:59 am America/New_York

Hi Sean.  Really?   I guess I'll give it a try.  I thought I'd have to use a string with the unit abbreviation for each field - not that that would be so hard...:smile:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Problem with mk_matchup.py

by OB.DAAC - SeanBailey » Fri Oct 16, 2020 11:35 am America/New_York

Oh, you'll need the header to  match the number of fields in the /fields= header...you might be able to cheat with a series of none,none,..
It is just a warning there, so you can ignore it, but if it's too annoying, adding the header will eliminate the warning.

Sean

Post Reply