Problems to untar file with MODIS PAR L3 images

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
laurare
Posts: 6
Joined: Wed Aug 04, 2021 1:36 pm America/New_York
Answers: 0

Problems to untar file with MODIS PAR L3 images

by laurare » Wed Aug 04, 2021 1:52 pm America/New_York

Hello,

last week I tried to download daily MODIS PAR l3 images from https://oceancolor.gsfc.nasa.gov/l3/
I clicked on "extract or download l3 data" bottom to reduce the region and period of time. I got the order I wanted with the manifest.txt file, and I did the same as last time when I downloaded MODIS chla data with Python:

# Download PAR data
import wget
import tarfile

output_directory = '/Users/laurare/OneDrive - cima.fcen.uba.ar/PICT-2017-2386_Cele/PAR_modis'

url = 'https://oceandata.sci.gsfc.nasa.gov/cgi ... sted_files'

file = wget.download(url,output_directory)
tar = tarfile.open(file,'r:*')
tar.extractall(path = '/Users/laurare/OneDrive - cima.fcen.uba.ar/OneDrive - cima.fcen.uba.ar/PICT-2017-2386_Cele/PAR_modis/')
tar.close()

However, this time I couldn't open the tar file. I thought it may be a mac problem, so I asked a colleague to try in linux, then on ubuntu and I also tried using wget and tar on linux terminal instead of python. I even reduced the period of time, hoping that the problem was related with the size of the file, and nothing worked.
I also thought that I wasn't downloading the file properly and I tried to use the appkey:

url = 'https://oceandata.sci.gsfc.nasa.gov/cgi ... 3bd4174e9f'

Any help is welcome. Thank you!
Laura

Tags:

OB.DAAC - amscott
User Services
User Services
Posts: 345
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Problems to untar file with MODIS PAR L3 images

by OB.DAAC - amscott » Tue Sep 21, 2021 12:33 pm America/New_York

Hello,

This post scrolled past before we saw it to respond. Did you resolve this issue or can we still help you with something?
You can also look here for search and data download methods:
https://oceancolor.gsfc.nasa.gov/data/download_methods/

laurare
Posts: 6
Joined: Wed Aug 04, 2021 1:36 pm America/New_York
Answers: 0

Re: Problems to untar file with MODIS PAR L3 images

by laurare » Tue Sep 21, 2021 5:07 pm America/New_York

I still have problems. I've tried

echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc ; > ~/.urs_cookies
chmod 0600 ~/.netrc

on linux terminal, where I replaced USERNAME and PASSWD with my credentials. Then, I wrote the following and nothing happened:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --no-check-certificate --content-disposition -i https://oceandata.sci.gsfc.nasa.gov/cgi ... sted_files

I have a question about the netrc file, do I have to do it once? or everytime that I want to download data?

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Problems to untar file with MODIS PAR L3 images

by OB.DAAC - SeanBailey » Wed Sep 22, 2021 9:24 am America/New_York

Laura,
The .netrc set up is only needed once (unless you change your password or delete the .netrc file)

Since you are using python, I would recommend using the requests module to accomplish the file retrieval, rather than wget.

You could also just grab the obdaac_download script (available from the webpage Alicia mentioned) - it uses the requests module, but also has some other niceties for data retrieval options from our distribution servers.

Sean

laurare
Posts: 6
Joined: Wed Aug 04, 2021 1:36 pm America/New_York
Answers: 0

Re: Problems to untar file with MODIS PAR L3 images

by laurare » Mon Aug 28, 2023 10:00 pm America/New_York

Hello Sean,

I gave up until now, that I tried again hoping to be more wide-awake but I failed ... I downloaded the obdaac_download.py this time and I tried different syntaxes (" ", ' ',-, etc) for the HTTP_MANIFEST, ODIR, filename and there is always a problem.
When I execute the following, it says that h and v are not defined:
obdaac_download [-h] [-v] [-HTTP_MANIFEST] [-ODIR] [--uncompress] [-APPKEY]

Thank you
Laura

laurare
Posts: 6
Joined: Wed Aug 04, 2021 1:36 pm America/New_York
Answers: 0

Re: Problems to untar file with MODIS PAR L3 images

by laurare » Tue Aug 29, 2023 10:25 pm America/New_York

Yesterday I wrote a reply in another question and I got a suggestion about opening the http_manifest in a browser and it worked!
Even though I resolved the downloading problem, I would like to know how to use the python script.

Thank you!

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Problems to untar file with MODIS PAR L3 images

by OB.DAAC - SeanBailey » Fri Sep 01, 2023 2:56 pm America/New_York

The --http_manifest option takes the manifest URL as input, e.g.:

Code: Select all

❯ obdaac_download --http_manifest=https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt\?h\=ocdist305\&p\=/data1/e030e02b2cfc4c60 -v
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist305&p=/data1/e030e02b2cfc4c60
OBPG session started
Downloading http_manifest.txt (    0.00 MBs)
[==================================================]
...Done
Output directory: /anyuser/myfavoritepath
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/requested_files_1.tar?h=ocdist305&p=/data1/e030e02b2cfc4c60/requested_files
Downloading requested_files_1.tar (    1.60 MBs)
[==================================================]
...Done

It will download the manifest file and then all the files contained within that file.

Alternatively, you can download the manifest file (via your web browser or wget or cURL) and pass the file file to the script with the --filelist option, e.g.:

Code: Select all

❯ obdaac_download --filelist=http_manifest.txt -v
Output directory: /anyuser/myfavoritepath
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/requested_files_1.tar?h=ocdist305&p=/data1/e030e02b2cfc4c60/requested_files
OBPG session started
Downloading requested_files_1.tar (    1.60 MBs)
[==================================================]
...Done
Regards,
Sean

lylawa
Posts: 1
Joined: Sun Oct 08, 2023 11:41 am America/New_York
Answers: 0

Re: Problems to untar file with MODIS PAR L3 images

by lylawa » Sun Oct 08, 2023 11:45 am America/New_York

Laura,

You only need to set up the .netrc file once, unless you make changes to your password or delete the file.

Considering your use of Python, I suggest using the requests module for file retrieval instead of wget. Alternatively, you can obtain the obdaac_download script mentioned by Alicia on the webpage. This script utilizes the requests module and offers additional features for accessing data from our distribution servers. Visit https://apkhexo.com/
Last edited by lylawa on Thu Oct 19, 2023 3:09 am America/New_York, edited 2 times in total.

Post Reply