VIIRS and MODIS Level 2 Subscription Access

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
oceani
Posts: 68
Joined: Wed Aug 23, 2006 12:32 pm America/New_York
Answers: 0

Re: VIIRS and MODIS Level 2 Subscription Access

by oceani » Thu Sep 28, 2023 6:33 pm America/New_York

I just updated wget on one machine and the most recent version I can get for RHEL seems to be 1.14. All of our systems run RHEL for very specific reasons. Changing all of our scripts over to use curl would be a large task. Is there anyway to backtrack to your old version in order to give me some time to switch. Otherwise, we are going to be in a very bad situation here.

PS - V1.14 had the same error

Mark

Tags:

OB WebDev - masoudso
Subject Matter Expert
Subject Matter Expert
Posts: 20
Joined: Tue Dec 22, 2020 2:11 pm America/New_York
Answers: 0
Has thanked: 2 times

Re: VIIRS and MODIS Level 2 Subscription Access

by OB WebDev - masoudso » Thu Sep 28, 2023 8:05 pm America/New_York

Hi Mark,
The old version is back in place. We will investigate solutions for the 308 message.

Thanks,
Masoud

oceani
Posts: 68
Joined: Wed Aug 23, 2006 12:32 pm America/New_York
Answers: 0

Re: VIIRS and MODIS Level 2 Subscription Access

by oceani » Fri Sep 29, 2023 8:14 am America/New_York

Masoud,

Thank you very much for doing this. I see that things are running normally on my end now. If you find that you need to perform the switch eventually, please send me a sample curl command line to download a file delivered via subscription and I will start working on switching our retrieval method from wget to curl.

Thanks again,

Mark

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 138
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0
Been thanked: 1 time

Re: VIIRS and MODIS Level 2 Subscription Access

by OB ODPS - jgwilding » Fri Sep 29, 2023 9:00 am America/New_York

Mark,

We have some redhat systems (RHEL 8.8) as well, and they have wget 1.18, which doesn't have the problem. I'm guessing your redhat systems are older? As Masoud said, we are looking at what might be behind the redirect

There's nothing in the usage of getfile that is specific to subscriptions. It's the call to file_search that accepts the subID that returns the list of URLs, and that can continue to use wget if you prefer. This would be the equivalent curl command to retrieve a file. You can try it manually from a command line to verify. Sub in your user and pass for the -u option:

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -u user:pass https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20230928T152501.L2.SST.NRT.nc

Code: Select all

% curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -u user:pass https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20230928T152501.L2.SST.NRT.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 19.0M  100 19.0M    0     0  36.1M      0 --:--:-- --:--:-- --:--:--  110M
% echo $?
0
% ls -l AQUA_MODIS.20230928T152501.L2.SST.NRT.nc
-rw-rw-r-- 1 opuser opuser 19967451 Sep 29 12:49 AQUA_MODIS.20230928T152501.L2.SST.NRT.nc


oceani
Posts: 68
Joined: Wed Aug 23, 2006 12:32 pm America/New_York
Answers: 0

Re: VIIRS and MODIS Level 2 Subscription Access

by oceani » Fri Sep 29, 2023 10:02 am America/New_York

John and Masoud,

Thank you for all your help on this. You are correct our systems are running relatively old versions of RHEL (6.1 and 7.2). This is because an older image processing application we are using requires older Linux. Unfortunately, we are stuck with it for a while. I imagine the two older versions of wget are the problem. I will work on switching over to curl next week. I try to avoid implementing changes over the weekend. If all goes well, I'll notify you via this thread and let you know that you can switch back to your redirect method. I will also see if I can install wget 1.18 on any of our machines.

Cheers,

Mark

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 138
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0
Been thanked: 1 time

Re: VIIRS and MODIS Level 2 Subscription Access

by OB ODPS - jgwilding » Fri Sep 29, 2023 1:47 pm America/New_York

Mark,

Can you try a manual test of the modified getfile? Use the same URL for a file but remove the ob/ or cgi/ path element, so it will be oceandata.sci.gsfc.nasa.gov/getfile/name-of-file

Try that with your usualy wget options.

john

oceani
Posts: 68
Joined: Wed Aug 23, 2006 12:32 pm America/New_York
Answers: 0

Re: VIIRS and MODIS Level 2 Subscription Access

by oceani » Fri Sep 29, 2023 2:03 pm America/New_York

John,

Looks like it worked:
AQUA_MODIS.20230929T073501.L2.SST.NRT.nc

--2023-09-29 11:56:30-- https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20230929T073501.L2.SST.NRT.nc
Resolving oceandata.sci.gsfc.nasa.gov... 169.154.128.84
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FAQUA_MODIS.20230929T073501.L2.SST.NRT.nc [following]
--2023-09-29 11:56:31-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FAQUA_MODIS.20230929T073501.L2.SST.NRT.nc
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/ [following]
--2023-09-29 11:56:32-- https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/
Resolving urs.earthdata.nasa.gov... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=2954b146155f8fac85edbff25812c8b548ecb256cc6d95256991f5af6d4a22a1 [following]
--2023-09-29 11:56:32-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=2954b146155f8fac85edbff25812c8b548ecb256cc6d95256991f5af6d4a22a1
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20230929T073501.L2.SST.NRT.nc [following]
--2023-09-29 11:56:41-- https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20230929T073501.L2.SST.NRT.nc
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20690714 (20M) [application/octet-stream]
Saving to: “AQUA_MODIS.20230929T073501.L2.SST.NRT.nc”

100%[===============================================>] 20,690,714 1.69M/s in 16s

2023-09-29 11:57:01 (1.21 MB/s) - “AQUA_MODIS.20230929T073501.L2.SST.NRT.nc” saved [20690714/20690714]

Can I ask that you wait until Monday to make any changes to remove the cgi from the path? Unfortunately I have to leave the office this weekend and will be away from access to our scripts/Internet. I can change the URLs in our scripts first thing Monday morning, if that is OK.

Thanks!

Mark

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 138
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0
Been thanked: 1 time

Re: VIIRS and MODIS Level 2 Subscription Access

by OB ODPS - jgwilding » Fri Sep 29, 2023 2:10 pm America/New_York

yes, but you won't need to change anything. our original redirects should work. this test bypassed those. we'll leave everything be until Monday.

Thanks.
john

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 138
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0
Been thanked: 1 time

Re: VIIRS and MODIS Level 2 Subscription Access

by OB ODPS - jgwilding » Mon Oct 02, 2023 10:59 am America/New_York

Mark,

We put in the new version of getfile this morning. Hopefully things on your end will continue to work okay, but let us know if not.


john

oceani
Posts: 68
Joined: Wed Aug 23, 2006 12:32 pm America/New_York
Answers: 0

Re: VIIRS and MODIS Level 2 Subscription Access

by oceani » Mon Oct 02, 2023 6:04 pm America/New_York

Looks like all is working fine. Thanks very much for your help.

Mark

Post Reply