getanc.py for GOCI

Please enter here to ask a question about any NASA Science related topics!
Post Reply
oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

getanc.py for GOCI

by oo_processing » Tue Jan 03, 2017 1:06 pm America/New_York

I too am curious as to why other missions have no issue with getanc.py but GOCI does, eg a SeaWiFS file:

what 159 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/S1999183032627.L1A_MLAC  --ancdb=/tmp/tmp_oo/ancillary_data.db
icefile=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918306_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918306_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/1999/182/N199918200_O3_EPTOMS_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_O3_EPTOMS_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_O3_EPTOMS_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N1999183_SST_OIV2AV_24h.nc

But the GOCI file not.

what 160 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
Traceback (most recent call last):
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/getanc.py", line 183, in <module>
    g.setup()
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py", line 238, in setup
    starttime, stoptime = self.get_start_end_info_from_xml(metadata)
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py", line 138, in get_start_end_info_from_xml
    time_start_list = xml_root.findall('.//Attribute[@Name="time_coverage_start"]')
  File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 355, in findall
    return ElementPath.findall(self, path)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 198, in findall
    return _compile(path).findall(element)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 176, in _compile
    p = Path(path)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 93, in __init__
    "expected path separator (%s)" % (op or tag)
SyntaxError: expected path separator ([)
what 161 :/tmp/tmp_oo>

Surely there should be a software tweak as opposed to installing and maintaining another version of python? Especially since the older version works fine:

what 117 :/tmp/tmp_oo> which getanc.py
/optics1/software/seadas/ocssw_7.0.2/ocssw/run/scripts/getanc.py
what 118 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
icefile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/2013/117/N201311700_O3_AURAOMI_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N2013118_SST_OIV2AV_24h.nc

And if it is in the ancillary_data.db it is read fine but fails when deleted:

what 102 :/tmp/tmp_oo> SETOCSSW73
what 103 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
icefile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/2013/117/N201311700_O3_AURAOMI_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N2013118_SST_OIV2AV_24h.nc

what 104 :/tmp/tmp_oo> rm /tmp/tmp_oo/ancillary_data.db

what 105 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
Traceback (most recent call last):
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/getanc.py", line 183, in <module>
    g.setup()
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py", line 238, in setup
    starttime, stoptime = self.get_start_end_info_from_xml(metadata)
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py", line 138, in get_start_end_info_from_xml
    time_start_list = xml_root.findall('.//Attribute[@Name="time_coverage_start"]')
  File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 355, in findall
    return ElementPath.findall(self, path)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 198, in findall
    return _compile(path).findall(element)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 176, in _compile
    p = Path(path)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 93, in __init__
    "expected path separator (%s)" % (op or tag)
SyntaxError: expected path separator ([)

Tags:

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

getanc.py for GOCI

by oo_processing » Tue Jan 03, 2017 3:10 pm America/New_York

Perhaps you might consider using a more version friendly module in $OCSSWROOT/run/scripts/modules/anc_utils.py

This example this works on both CentOS 6 and CentOS 7 using up-to-date stock python with almost no changes:

what 147 :/tmp/tmp_oo> diff /optics1/software/seadas/seadas-7.2/ocssw/run/scripts/modules/anc_utils.py /optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py
8d7
< import lxml.etree
136,137c135
<         #xml_root = ElementTree.fromstring(raw_xml)
<       xml_root = lxml.etree.fromstring(raw_xml)
---

>         xml_root = ElementTree.fromstring(raw_xml)


140,141c138
<         #time_start_list = xml_root.findall('.//Attribute[@Name="time_coverage_start"]')
<       time_start_list = xml_root.xpath('.//Attribute[@Name="time_coverage_start"]')
---

>         time_start_list = xml_root.findall('.//Attribute[@Name="time_coverage_start"]')


147,148c144
<             #time_start_list = xml_root.findall('.//Attribute[@Name="Scene Start time"]')
<           time_start_list = xml_root.xpath('.//Attribute[@Name="Scene Start time"]')
---

>             time_start_list = xml_root.findall('.//Attribute[@Name="Scene Start time"]')


154,155c150
<         #time_end_list = xml_root.findall('.//Attribute[@Name="time_coverage_end"]')
<       time_end_list = xml_root.xpath('.//Attribute[@Name="time_coverage_end"]')
---

>         time_end_list = xml_root.findall('.//Attribute[@Name="time_coverage_end"]')


161,162c156
<             #time_end_list = xml_root.findall('.//Attribute[@Name="Scene end time"]')
<           time_end_list = xml_root.xpath('.//Attribute[@Name="Scene end time"]')
---

>             time_end_list = xml_root.findall('.//Attribute[@Name="Scene end time"]')


what 108 :/tmp/tmp_oo> rm ancillary_data.db

what 109 :/tmp/tmp_oo> which getanc.py
/optics1/software/seadas/seadas-7.2/ocssw/run/scripts/getanc.py

what 110 :/tmp/tmp_oo> python -V
Python 2.6.6

what 111 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/S1999183032627.L1A_MLAC --ancdb=/tmp/tmp_oo/ancillary_data.db
icefile=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918306_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918306_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/1999/182/N199918200_O3_EPTOMS_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_O3_EPTOMS_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_O3_EPTOMS_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N1999183_SST_OIV2AV_24h.nc

what 112 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
Encountered more than 1 Scene Start time tag. Using 1st value.
Encountered more than 1 Scene end time tag. Using 1st value.
icefile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/2013/117/N201311700_O3_AURAOMI_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N2013118_SST_OIV2AV_24h.nc

what 113 :/tmp/tmp_oo> cat G20131180016.L1B_COMS.he5.anc
icefile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/2013/117/N201311700_O3_AURAOMI_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N2013118_SST_OIV2AV_24h.nc

Yes, I know we are all able to change whatever we want, BUT, having a broad based common module would be greatly appreciated by users.

Brock
attachment 1

melliott
Posts: 11
Joined: Tue Jun 08, 2010 2:23 pm America/New_York

getanc.py for GOCI

by melliott » Wed Jan 04, 2017 10:02 am America/New_York

Brock,

Thanks for the suggestion. Wish I could use it. However, while it is is Anaconda, lxml doesn't seem to be in the standard Python library, so I'll be doing some debugging on the code using the xml module ...

Matt

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

getanc.py for GOCI

by oo_processing » Wed Jan 04, 2017 10:44 am America/New_York

Matt,
So, does that mean that there may be a solution that is in the standard Python library?

In the mean time, is there a way that I can use my modified code, and NOT have the script overwritten whenever I do an update?

I have the same issue another file (L1B_template.pcf) where I have to change:
800615|Write_Night_Mode_HiRes_Data|0
to
800615|Write_Night_Mode_HiRes_Data|1
Every time I update.

Brock

melliott
Posts: 11
Joined: Tue Jun 08, 2010 2:23 pm America/New_York

getanc.py for GOCI

by melliott » Wed Jan 04, 2017 1:55 pm America/New_York

Brock,

It means I'm going to work to figure out how to get things working again using Python and modules in its standard library, probably including the xml module, but don't have a fix yet.

The other issue you raise: Off the top of my head, I am not a git guru and don't know of a solution for certain. Not sure if changing the file permissions with a 'chmod -w FILENAME' would work, fail, or even cause more harm. That said, I'd expect there to be some way to do what you want, but I should be debugging the anc_utils issue (and others) right now.

-Matt

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

getanc.py for GOCI

by oo_processing » Wed Jan 04, 2017 2:29 pm America/New_York

I understand. Thanks for your efforts.

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

getanc.py for GOCI

by oo_processing » Wed Mar 08, 2017 12:35 pm America/New_York

Matt,
I was curious if the python modifications had been made for SeaDAS 7.4?

Brock

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

getanc.py for GOCI

by oo_processing » Thu Mar 09, 2017 12:13 pm America/New_York

I guess it does work. Excellent :grin:

Post Reply