Page 1 of 1

Problems to untar file with MODIS PAR L3 images

Posted: Wed Aug 04, 2021 1:52 pm America/New_York
by laurare
Hello,

last week I tried to download daily MODIS PAR l3 images from https://oceancolor.gsfc.nasa.gov/l3/
I clicked on "extract or download l3 data" bottom to reduce the region and period of time. I got the order I wanted with the manifest.txt file, and I did the same as last time when I downloaded MODIS chla data with Python:

# Download PAR data
import wget
import tarfile

output_directory = '/Users/laurare/OneDrive - cima.fcen.uba.ar/PICT-2017-2386_Cele/PAR_modis'

url = 'https://oceandata.sci.gsfc.nasa.gov/cgi ... sted_files'

file = wget.download(url,output_directory)
tar = tarfile.open(file,'r:*')
tar.extractall(path = '/Users/laurare/OneDrive - cima.fcen.uba.ar/OneDrive - cima.fcen.uba.ar/PICT-2017-2386_Cele/PAR_modis/')
tar.close()

However, this time I couldn't open the tar file. I thought it may be a mac problem, so I asked a colleague to try in linux, then on ubuntu and I also tried using wget and tar on linux terminal instead of python. I even reduced the period of time, hoping that the problem was related with the size of the file, and nothing worked.
I also thought that I wasn't downloading the file properly and I tried to use the appkey:

url = 'https://oceandata.sci.gsfc.nasa.gov/cgi ... 3bd4174e9f'

Any help is welcome. Thank you!
Laura

Re: Problems to untar file with MODIS PAR L3 images

Posted: Tue Sep 21, 2021 12:33 pm America/New_York
by OB.DAAC - amscott
Hello,

This post scrolled past before we saw it to respond. Did you resolve this issue or can we still help you with something?
You can also look here for search and data download methods:
https://oceancolor.gsfc.nasa.gov/data/download_methods/

Re: Problems to untar file with MODIS PAR L3 images

Posted: Tue Sep 21, 2021 5:07 pm America/New_York
by laurare
I still have problems. I've tried

echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc ; > ~/.urs_cookies
chmod 0600 ~/.netrc

on linux terminal, where I replaced USERNAME and PASSWD with my credentials. Then, I wrote the following and nothing happened:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --no-check-certificate --content-disposition -i https://oceandata.sci.gsfc.nasa.gov/cgi ... sted_files

I have a question about the netrc file, do I have to do it once? or everytime that I want to download data?

Re: Problems to untar file with MODIS PAR L3 images

Posted: Wed Sep 22, 2021 9:24 am America/New_York
by OB.DAAC - SeanBailey
Laura,
The .netrc set up is only needed once (unless you change your password or delete the .netrc file)

Since you are using python, I would recommend using the requests module to accomplish the file retrieval, rather than wget.

You could also just grab the obdaac_download script (available from the webpage Alicia mentioned) - it uses the requests module, but also has some other niceties for data retrieval options from our distribution servers.

Sean

Re: Problems to untar file with MODIS PAR L3 images

Posted: Mon Aug 28, 2023 10:00 pm America/New_York
by laurare
Hello Sean,

I gave up until now, that I tried again hoping to be more wide-awake but I failed ... I downloaded the obdaac_download.py this time and I tried different syntaxes (" ", ' ',-, etc) for the HTTP_MANIFEST, ODIR, filename and there is always a problem.
When I execute the following, it says that h and v are not defined:
obdaac_download [-h] [-v] [-HTTP_MANIFEST] [-ODIR] [--uncompress] [-APPKEY]

Thank you
Laura

Re: Problems to untar file with MODIS PAR L3 images

Posted: Tue Aug 29, 2023 10:25 pm America/New_York
by laurare
Yesterday I wrote a reply in another question and I got a suggestion about opening the http_manifest in a browser and it worked!
Even though I resolved the downloading problem, I would like to know how to use the python script.

Thank you!

Re: Problems to untar file with MODIS PAR L3 images

Posted: Fri Sep 01, 2023 2:56 pm America/New_York
by OB.DAAC - SeanBailey
The --http_manifest option takes the manifest URL as input, e.g.:

Code: Select all

❯ obdaac_download --http_manifest=https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt\?h\=ocdist305\&p\=/data1/e030e02b2cfc4c60 -v
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist305&p=/data1/e030e02b2cfc4c60
OBPG session started
Downloading http_manifest.txt (    0.00 MBs)
[==================================================]
...Done
Output directory: /anyuser/myfavoritepath
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/requested_files_1.tar?h=ocdist305&p=/data1/e030e02b2cfc4c60/requested_files
Downloading requested_files_1.tar (    1.60 MBs)
[==================================================]
...Done

It will download the manifest file and then all the files contained within that file.

Alternatively, you can download the manifest file (via your web browser or wget or cURL) and pass the file file to the script with the --filelist option, e.g.:

Code: Select all

❯ obdaac_download --filelist=http_manifest.txt -v
Output directory: /anyuser/myfavoritepath
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/requested_files_1.tar?h=ocdist305&p=/data1/e030e02b2cfc4c60/requested_files
OBPG session started
Downloading requested_files_1.tar (    1.60 MBs)
[==================================================]
...Done
Regards,
Sean

Re: Problems to untar file with MODIS PAR L3 images

Posted: Sun Oct 08, 2023 11:45 am America/New_York
by lylawa
Laura,

You only need to set up the .netrc file once, unless you make changes to your password or delete the file.

Considering your use of Python, I suggest using the requests module for file retrieval instead of wget. Alternatively, you can obtain the obdaac_download script mentioned by Alicia on the webpage. This script utilizes the requests module and offers additional features for accessing data from our distribution servers.

Re: Problems to untar file with MODIS PAR L3 images

Posted: Sun Mar 24, 2024 1:23 am America/New_York
by sunflower36002
Hi Laura,

It seems you're encountering issues downloading MODIS PAR L3 images from the NASA website despite trying different methods. Have you double-checked the URL and reached out to NASA support or forums for assistance? Hopefully, you'll find a solution soon.