MODIS data download

avmehta · by **avmehta** » Mon Oct 12, 2020 10:54 am America/New_York

Hello, I am trying to download MODIS L1A data that I ordered using wget but I get the following -wget worked before so not sure what might be the problem. I'd appreciate your help. My data Order number is Data order [5080c0f01685fc5a].Thank you.Amita========wget "(link=)oceandata.sci.gsfc.nasa.gov/api/file_search?subID=avmehta&subType=1&format=txt&addurl=1" -O - | wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --no-check-certificate --content-disposition -i - https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist204&p=/data1/5080c0f01685fc5a[1] 1265735sh-4.4$ --2020-10-12 10:47:44-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist204Redirecting output to â€˜wget-logâ€™.Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0

128 :: 84Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.HTTP request sent, awaiting response... 302 FoundLocation: /ob/getfile/http_manifest.txt?h=ocdist204 [following]--2020-10-12 10:47:45-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/http_manifest.txt?h=ocdist204Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.HTTP request sent, awaiting response... 302 FoundLocation: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict [following]--2020-10-12 10:47:45-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2FrestrictResolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0

4081::89Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.HTTP request sent, awaiting response... 302 FoundLocation: https://oceandata.sci.gsfc.nasa.gov (/ob/getfile/restrict?code=533bdbda353e0a5f19c5f37c0e26aee5ca516c4eb6fb8660d8ae15e174684ecf) [following]--2020-10-12 10:47:45-- https://oceandata.sci.gsfc.nasa.gov (/ob/getfile/restrict?code=533bdbda353e0a5f19c5f37c0e26aee5ca516c4eb6fb8660d8ae15e174684ecf). Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.HTTP request sent, awaiting response... 302 FoundLocation: /ob/getfile/http_manifest.txt?h=ocdist204 [following]--2020-10-12 10:47:45-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/http_manifest.txt?h=ocdist204Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.HTTP request sent, awaiting response... 409 Conflict2020-10-12 10:47:50 ERROR 409: Conflict.No URLs found in -.

by **OB ODPS - jgwilding** » Mon Oct 12, 2020 11:33 am America/New_York

Hi. It looks like the URL to the manifest file is getting cut off at the '&' by the shell. Have you tried putting the manifest URL in quotes?

john

avmehta · by **avmehta** » Mon Oct 12, 2020 12:10 pm America/New_York

Thank you John.
I used quotes but the manifest url still did not work. But when I tried 'requested_files' url from the manifest.txt file it worked! I was wondering if there are multiple tar 'requested_files' then that is what is required! In any case - at least first bunch of files are now download!
Thanks again.
Amita

by **OB ODPS - jgwilding** » Mon Oct 12, 2020 12:59 pm America/New_York

Hi Amita,

I missed in your first post that you are piping the results of the call to file-search into a wget for the manifest file. While i still think you need quotes around the manifest URL to protect the special characters, there should not be a call to the file-search. What you want is,

wget -O - 'manifest-url-from-email' | wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --no-check-certificate --content-disposition -i -

This page has examples:

https://oceancolor.gsfc.nasa.gov/data/download_methods/

john