getanc.py for GOCI

[oo_processing]

I too am curious as to why other missions have no issue with getanc.py but GOCI does, eg a SeaWiFS file:

what 159 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/S1999183032627.L1A_MLAC  --ancdb=/tmp/tmp_oo/ancillary_data.db
icefile=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918306_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918306_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/1999/182/N199918200_O3_EPTOMS_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_O3_EPTOMS_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_O3_EPTOMS_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N1999183_SST_OIV2AV_24h.nc

But the GOCI file not.

what 160 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
Traceback (most recent call last):
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/getanc.py", line 183, in <module>
    g.setup()
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py", line 238, in setup
    starttime, stoptime = self.get_start_end_info_from_xml(metadata)
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py", line 138, in get_start_end_info_from_xml
    time_start_list = xml_root.findall('.//Attribute[@Name="time_coverage_start"]')
  File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 355, in findall
    return ElementPath.findall(self, path)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 198, in findall
    return _compile(path).findall(element)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 176, in _compile
    p = Path(path)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 93, in __init__
    "expected path separator (%s)" % (op or tag)
SyntaxError: expected path separator ([)
what 161 :/tmp/tmp_oo>

Surely there should be a software tweak as opposed to installing and maintaining another version of python? Especially since the older version works fine:

what 117 :/tmp/tmp_oo> which getanc.py
/optics1/software/seadas/ocssw_7.0.2/ocssw/run/scripts/getanc.py
what 118 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
icefile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/2013/117/N201311700_O3_AURAOMI_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N2013118_SST_OIV2AV_24h.nc

And if it is in the ancillary_data.db it is read fine but fails when deleted:

what 102 :/tmp/tmp_oo> SETOCSSW73
what 103 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
icefile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/2013/117/N201311700_O3_AURAOMI_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N2013118_SST_OIV2AV_24h.nc

what 104 :/tmp/tmp_oo> rm /tmp/tmp_oo/ancillary_data.db

what 105 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
Traceback (most recent call last):
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/getanc.py", line 183, in <module>
    g.setup()
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py", line 238, in setup
    starttime, stoptime = self.get_start_end_info_from_xml(metadata)
  File "/optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py", line 138, in get_start_end_info_from_xml
    time_start_list = xml_root.findall('.//Attribute[@Name="time_coverage_start"]')
  File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 355, in findall
    return ElementPath.findall(self, path)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 198, in findall
    return _compile(path).findall(element)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 176, in _compile
    p = Path(path)
  File "/usr/lib64/python2.6/xml/etree/ElementPath.py", line 93, in __init__
    "expected path separator (%s)" % (op or tag)
SyntaxError: expected path separator ([)

[oo_processing]

Perhaps you might consider using a more version friendly module in $OCSSWROOT/run/scripts/modules/anc_utils.py

This example this works on both CentOS 6 and CentOS 7 using up-to-date stock python with almost no changes:

what 147 :/tmp/tmp_oo> diff /optics1/software/seadas/seadas-7.2/ocssw/run/scripts/modules/anc_utils.py /optics1/software/seadas/seadas-7.3/ocssw/run/scripts/modules/anc_utils.py
8d7
< import lxml.etree
136,137c135
<         #xml_root = ElementTree.fromstring(raw_xml)
<       xml_root = lxml.etree.fromstring(raw_xml)
---

>         xml_root = ElementTree.fromstring(raw_xml)

140,141c138
<         #time_start_list = xml_root.findall('.//Attribute[@Name="time_coverage_start"]')
<       time_start_list = xml_root.xpath('.//Attribute[@Name="time_coverage_start"]')
---

>         time_start_list = xml_root.findall('.//Attribute[@Name="time_coverage_start"]')

147,148c144
<             #time_start_list = xml_root.findall('.//Attribute[@Name="Scene Start time"]')
<           time_start_list = xml_root.xpath('.//Attribute[@Name="Scene Start time"]')
---

>             time_start_list = xml_root.findall('.//Attribute[@Name="Scene Start time"]')

154,155c150
<         #time_end_list = xml_root.findall('.//Attribute[@Name="time_coverage_end"]')
<       time_end_list = xml_root.xpath('.//Attribute[@Name="time_coverage_end"]')
---

>         time_end_list = xml_root.findall('.//Attribute[@Name="time_coverage_end"]')

161,162c156
<             #time_end_list = xml_root.findall('.//Attribute[@Name="Scene end time"]')
<           time_end_list = xml_root.xpath('.//Attribute[@Name="Scene end time"]')
---

>             time_end_list = xml_root.findall('.//Attribute[@Name="Scene end time"]')

what 108 :/tmp/tmp_oo> rm ancillary_data.db

what 109 :/tmp/tmp_oo> which getanc.py
/optics1/software/seadas/seadas-7.2/ocssw/run/scripts/getanc.py

what 110 :/tmp/tmp_oo> python -V
Python 2.6.6

what 111 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/S1999183032627.L1A_MLAC --ancdb=/tmp/tmp_oo/ancillary_data.db
icefile=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918306_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918306_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/1999/182/N199918200_O3_EPTOMS_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_O3_EPTOMS_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N199918300_O3_EPTOMS_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/1999/183/N1999183_SST_OIV2AV_24h.nc

what 112 :/tmp/tmp_oo> getanc.py /tmp/tmp_oo/G20131180016.L1B_COMS.he5 --ancdb=/tmp/tmp_oo/ancillary_data.db
Encountered more than 1 Scene Start time tag. Using 1st value.
Encountered more than 1 Scene end time tag. Using 1st value.
icefile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/2013/117/N201311700_O3_AURAOMI_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N2013118_SST_OIV2AV_24h.nc

what 113 :/tmp/tmp_oo> cat G20131180016.L1B_COMS.he5.anc
icefile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_SEAICE_NSIDC_24h.hdf
met1=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_MET_NCEPR2_6h.hdf
met2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
met3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311806_MET_NCEPR2_6h.hdf
ozone1=/optics1/software/seadas/seadas_6.1/var/anc/2013/117/N201311700_O3_AURAOMI_24h.hdf
ozone2=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
ozone3=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N201311800_O3_AURAOMI_24h.hdf
sstfile=/optics1/software/seadas/seadas_6.1/var/anc/2013/118/N2013118_SST_OIV2AV_24h.nc

Yes, I know we are all able to change whatever we want, BUT, having a broad based common module would be greatly appreciated by users.

Brock
attachment 1

[melliott]

Brock,

Thanks for the suggestion. Wish I could use it. However, while it is is Anaconda, lxml doesn't seem to be in the standard Python library, so I'll be doing some debugging on the code using the xml module ...

Matt

[oo_processing]

Matt,
So, does that mean that there may be a solution that is in the standard Python library?

In the mean time, is there a way that I can use my modified code, and NOT have the script overwritten whenever I do an update?

I have the same issue another file (L1B_template.pcf) where I have to change:
800615|Write_Night_Mode_HiRes_Data|0
to
800615|Write_Night_Mode_HiRes_Data|1
Every time I update.

Brock

[melliott]

Brock,

It means I'm going to work to figure out how to get things working again using Python and modules in its standard library, probably including the xml module, but don't have a fix yet.

The other issue you raise: Off the top of my head, I am not a git guru and don't know of a solution for certain. Not sure if changing the file permissions with a 'chmod -w FILENAME' would work, fail, or even cause more harm. That said, I'd expect there to be some way to do what you want, but I should be debugging the anc_utils issue (and others) right now.

-Matt

[oo_processing]

I understand. Thanks for your efforts.

[oo_processing]

Matt,
I was curious if the python modifications had been made for SeaDAS 7.4?

Brock

[oo_processing]

I guess it does work. Excellent :grin: