getanc.py

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

getanc.py

by avmehta » Mon Jun 01, 2020 11:01 pm America/New_York

Hello,
   I have been  trying to run OCSSW on NCCS/ADAPT system. I had help in installing SeaDAS and OCSSW on the system. When I try to run getanc.py I get the following messages - it seems that some database is missing. This does not happen when I use it on my Linux server. I'll  appreciate your help.
Thanks.
Traceback (most recent call last):
  File "/usr/local/cb/ocssw/scripts/getanc.py", line 196, in <module>
    sys.exit(main())
  File "/usr/local/cb/ocssw/scripts/getanc.py", line 189, in main
    g.findweb()
  File "/usr/local/cb/ocssw/scripts/modules/anc_utils.py", line 491, in findweb
    ancdatabase.openDB()
  File "/usr/local/cb/ocssw/scripts/modules/ancDB.py", line 18, in openDB
    conn = sqlite3.connect(self.dbfile, timeout=30)
sqlite3.OperationalError: unable to open database file

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

getanc.py

by OB.DAAC - SeanBailey » Tue Jun 02, 2020 9:02 am America/New_York

The script creates the SQLite database file.  This error indicates that it cannot do so.  Check that the user running the script has write permissions for the file.  The default location is $OCSSWROOT/var/ancillary_data.db
The path in the error message suggests the installation was done as root in a system wide location.  That may explain the permission issues.

Sean

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

getanc.py

by avmehta » Tue Jun 02, 2020 9:25 am America/New_York

Thank you Sean. Will check about the write permission.
Amita

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

getanc.py

by avmehta » Tue Jun 02, 2020 2:41 pm America/New_York

Hi Sean,
   We were able to resolve this and execute getanc.py! Thanks a lot for your help.

    I am having problem getting OLCI Level-1 data. I select dates/region from L1&L2 Browser, then order data and get a list of urls for OLCI L-1 zip files. These are supposed to be directories. When I use wget to download these files I do get the files but when I try to unzip I get the following message:

Archive:  S3A_OL_1_EFR____20190731T150748_20190731T151048_20190801T182818_0179_047_296_2340_LN1_O_NT_002.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of S3A_OL_1_EFR____20190731T150748_20190731T151048_20190801T182818_0179_047_296_2340_LN1_O_NT_002.zip or
        S3A_OL_1_EFR____20190731T150748_20190731T151048_20190801T182818_0179_047_296_2340_LN1_O_NT_002.zip.zip, and cannot find S3A_OL_1_EFR____20190731T150748_20190731T151048_20190801T182818_0179_047_296_2340_LN1_O_NT_002.zip.ZIP, period.

Thanks.
Amita

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

getanc.py

by gnwiii » Tue Jun 02, 2020 4:52 pm America/New_York

It would be good to start a new topic for the new problem.  

When downloaded files don't behave, it is useful to provide as much detail as possible about the state of your misbehaving file.   The linux file S3A_OL_1_EFR____20190731T150748_20190731T151048_20190801T182818_0179_047_296_2340_LN1_O_NT_002.zip command should  confirm or deny that the downloaded file is a zip archive.   "ls -l S3A_OL_1_EFR____20190731T150748_20190731T151048_20190801T182818_0179_047_296_2340_LN1_O_NT_002.zip" could also help.   There are a couple ways a downloaded file gets replaced by an html document.   If the "file" command says you have an html file then you should be able to rename it to replace ".zip" with ".html" and open it in a browser.  The two examples I have encountered are 1) a message from the IT people saying your file was blocked by some policy, or 2) a login page, which may mean your downloader doesn't handle the EOSDIS single-signon system or just that EOSDIS was having a bad day.

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

getanc.py

by avmehta » Tue Jun 02, 2020 5:32 pm America/New_York

Thanks George for the reply.

  -  I am having this problem for several days so may not be EOSDIS

  - Also,  I download other data from EOSDIS (Earthdata/GES DISC) with wget without any problem.  But these are not  simple files - they are directories  as I mentioned in my previous mail. I see these directories on https://oceandata.sci.gsfc.nasa.gov/S3AOLCI/L1/.  When I order data through L1&L2 Browser,  I get a list of URLS -- the names point to particular swaths (directories) from this  L1 collection.   The list of URLs when I order data looks as shown below. But if I type https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/ in my browser (which is on the URL list), I get :
ERROR .:.
OceanColor Biology Processing Group (OBPG)
Sorry, an error has occurred. Use the back button to return to the previous page or go to the Ocean Color Home Page.

Not sure how to get to those swaths and do bulk download.
Thanks again.
Amita

========
List of URLs from the L1&L2 Data Browser/selector
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190731T150748_20190731T151048_20190801T182818_0179_047_296_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190730T153359_20190730T153659_20190731T191708_0180_047_282_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190730T145426_20190730T145726_20190731T193404_0179_028_139_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190729T152037_20190729T152337_20190730T201522_0179_028_125_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190727T151133_20190727T151433_20190728T193156_0179_047_239_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190726T153744_20190726T154044_20190727T201440_0179_047_225_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190726T145811_20190726T150111_20190727T185305_0179_028_082_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190725T152422_20190725T152722_20190726T192914_0180_028_068_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190723T151518_20190723T151818_20190724T185200_0179_047_182_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190722T154129_20190722T154429_20190723T193125_0180_047_168_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190722T150155_20190722T150455_20190723T195343_0179_028_025_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190721T152806_20190721T153106_20190722T184815_0179_028_011_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190720T145252_20190720T145552_20190721T191326_0179_047_139_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190719T151903_20190719T152203_20190720T195202_0179_047_125_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190718T154514_20190718T154814_20190719T203116_0179_047_111_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190718T150539_20190718T150839_20190719T191129_0179_027_353_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190717T153150_20190717T153450_20190718T195252_0179_027_339_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190716T145636_20190716T145936_20190717T183535_0179_047_082_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20190715T152247_20190715T152547_20190716T191221_0180_047_068_2340_LN1_O_NT_002.zip
https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3B_OL_1_EFR____20190714T150923_20190714T151223_20190715T183148_0180_027_296_2340_LN1_O_NT_002.zip

~                                                                                                                                                                   
~

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

getanc.py

by OB.DAAC - SeanBailey » Wed Jun 03, 2020 12:01 pm America/New_York

Amita,

The data access page and the L1/L2 browser point to the same files.  While the unzipped OCLI "files" are in fact directories, the zip file is what you download - and that is, well,  a file.
The getfile interface requires the file to be retrieved, so yes, https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/ by itself will result in an error. 

The "sentinel" bit in the URL tells getfile that it expects the user to have accepted the EULA for Sentinel data access in their Earthdata Login profile.  If that is not true, the download will fail (well, you'll download the login page as an html file).  You can verify your EULA acceptance by checking your profile at https://urs.earthdata.nasa.gov/ (on the profile summary page, but also under Applications->Accepted EULAs).  If you don't see the Sentinel EULA as accepted, you can edit your profile and accept it (select Edit Profile and scroll down to the "Agreements" section).

As for bulk downloading, the https://oceancolor.gsfc.nasa.gov/data/download_methods/ page describes various methods, including a python script that should make it easy...

Sean

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

getanc.py

by avmehta » Thu Jun 04, 2020 10:02 am America/New_York

Hi Sean,
  Thanks for the clarification.
   I checked on Earthdata and I do have accepted EULA for Sentinel.
   I have looked at the download methods and that is how I used wget. In any method  I can not figure out how to specify geographical region ( as I don't want global swaths) that is why I am using L1/L2 Browser to get specific URLS for the region. Is there a way I can download those specific swaths using the Python script ?
  
   Another  question I have is about lonlat2pixline function. I am using the following for OLCI:
   lonlat2pixline -r 300 -x 1 -y l xfdumanifest.xml  -76.94 36.84 -75.66 38.53
   I do get line/pix for most images when the area is covered. I just don't see any return code! How can I check that?

  Thanks a lot for your help.
  Amita

OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 258
Joined: Thu Mar 05, 2009 10:25 am America/New_York
Answers: 0
Been thanked: 2 times

getanc.py

by OB SeaDAS - dshea » Thu Jun 04, 2020 10:20 am America/New_York

in bash the return code is stored in $?

so to see the return code you can:

echo $?

don

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

getanc.py

by avmehta » Thu Jun 04, 2020 10:51 am America/New_York

Thanks Don.
Amita

Post Reply