Problems to untar file with MODIS PAR L3 images

[laurare]

Hello,

last week I tried to download daily MODIS PAR l3 images from https://oceancolor.gsfc.nasa.gov/l3/
I clicked on "extract or download l3 data" bottom to reduce the region and period of time. I got the order I wanted with the manifest.txt file, and I did the same as last time when I downloaded MODIS chla data with Python:

# Download PAR data
import wget
import tarfile

output_directory = '/Users/laurare/OneDrive - cima.fcen.uba.ar/PICT-2017-2386_Cele/PAR_modis'

url = 'https://oceandata.sci.gsfc.nasa.gov/cgi ... sted_files'

file = wget.download(url,output_directory)
tar = tarfile.open(file,'r:*')
tar.extractall(path = '/Users/laurare/OneDrive - cima.fcen.uba.ar/OneDrive - cima.fcen.uba.ar/PICT-2017-2386_Cele/PAR_modis/')
tar.close()

However, this time I couldn't open the tar file. I thought it may be a mac problem, so I asked a colleague to try in linux, then on ubuntu and I also tried using wget and tar on linux terminal instead of python. I even reduced the period of time, hoping that the problem was related with the size of the file, and nothing worked.
I also thought that I wasn't downloading the file properly and I tried to use the appkey:

url = 'https://oceandata.sci.gsfc.nasa.gov/cgi ... 3bd4174e9f'

Any help is welcome. Thank you!
Laura

[OB.DAACx - amscott]

Hello,

This post scrolled past before we saw it to respond. Did you resolve this issue or can we still help you with something?
You can also look here for search and data download methods:
https://oceancolor.gsfc.nasa.gov/data/download_methods/

[laurare]

I still have problems. I've tried

echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc ; > ~/.urs_cookies
chmod 0600 ~/.netrc

on linux terminal, where I replaced USERNAME and PASSWD with my credentials. Then, I wrote the following and nothing happened:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --no-check-certificate --content-disposition -i https://oceandata.sci.gsfc.nasa.gov/cgi ... sted_files

I have a question about the netrc file, do I have to do it once? or everytime that I want to download data?

[OB.DAACx - SeanBailey]

Laura,
The .netrc set up is only needed once (unless you change your password or delete the .netrc file)

Since you are using python, I would recommend using the requests module to accomplish the file retrieval, rather than wget.

You could also just grab the obdaac_download script (available from the webpage Alicia mentioned) - it uses the requests module, but also has some other niceties for data retrieval options from our distribution servers.

Sean

[laurare]

Hello Sean,

I gave up until now, that I tried again hoping to be more wide-awake but I failed ... I downloaded the obdaac_download.py this time and I tried different syntaxes (" ", ' ',-, etc) for the HTTP_MANIFEST, ODIR, filename and there is always a problem.
When I execute the following, it says that h and v are not defined:
obdaac_download [-h] [-v] [-HTTP_MANIFEST] [-ODIR] [--uncompress] [-APPKEY]

Thank you
Laura

[laurare]

Yesterday I wrote a reply in another question and I got a suggestion about opening the http_manifest in a browser and it worked!
Even though I resolved the downloading problem, I would like to know how to use the python script.

Thank you!

[OB.DAACx - SeanBailey]

The --http_manifest option takes the manifest URL as input, e.g.:

Code: Select all

❯ obdaac_download --http_manifest=https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt\?h\=ocdist305\&p\=/data1/e030e02b2cfc4c60 -v
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist305&p=/data1/e030e02b2cfc4c60
OBPG session started
Downloading http_manifest.txt (    0.00 MBs)
[==================================================]
...Done
Output directory: /anyuser/myfavoritepath
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/requested_files_1.tar?h=ocdist305&p=/data1/e030e02b2cfc4c60/requested_files
Downloading requested_files_1.tar (    1.60 MBs)
[==================================================]
...Done

It will download the manifest file and then all the files contained within that file.

Alternatively, you can download the manifest file (via your web browser or wget or cURL) and pass the file file to the script with the --filelist option, e.g.:

Code: Select all

❯ obdaac_download --filelist=http_manifest.txt -v
Output directory: /anyuser/myfavoritepath
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/requested_files_1.tar?h=ocdist305&p=/data1/e030e02b2cfc4c60/requested_files
OBPG session started
Downloading requested_files_1.tar (    1.60 MBs)
[==================================================]
...Done

Regards,
Sean

[lylawa]

Laura,

You only need to set up the .netrc file once, unless you make changes to your password or delete the file.

Considering your use of Python, I suggest using the requests module for file retrieval instead of wget. Alternatively, you can obtain the obdaac_download script mentioned by Alicia on the webpage. This script utilizes the requests module and offers additional features for accessing data from our distribution servers.