VIIRS and MODIS Level 2 Subscription Access

[oceani]

I just updated wget on one machine and the most recent version I can get for RHEL seems to be 1.14. All of our systems run RHEL for very specific reasons. Changing all of our scripts over to use curl would be a large task. Is there anyway to backtrack to your old version in order to give me some time to switch. Otherwise, we are going to be in a very bad situation here.

PS - V1.14 had the same error

Mark

[OB WebDev - masoudso]

Hi Mark,
The old version is back in place. We will investigate solutions for the 308 message.

Thanks,
Masoud

[oceani]

Masoud,

Thank you very much for doing this. I see that things are running normally on my end now. If you find that you need to perform the switch eventually, please send me a sample curl command line to download a file delivered via subscription and I will start working on switching our retrieval method from wget to curl.

Thanks again,

Mark

[OB ODPS - jgwilding]

Mark,

We have some redhat systems (RHEL 8.8) as well, and they have wget 1.18, which doesn't have the problem. I'm guessing your redhat systems are older? As Masoud said, we are looking at what might be behind the redirect

There's nothing in the usage of getfile that is specific to subscriptions. It's the call to file_search that accepts the subID that returns the list of URLs, and that can continue to use wget if you prefer. This would be the equivalent curl command to retrieve a file. You can try it manually from a command line to verify. Sub in your user and pass for the -u option:

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -u user:pass https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20230928T152501.L2.SST.NRT.nc

Code: Select all

% curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -u user:pass https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20230928T152501.L2.SST.NRT.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 19.0M  100 19.0M    0     0  36.1M      0 --:--:-- --:--:-- --:--:--  110M
% echo $?
0
% ls -l AQUA_MODIS.20230928T152501.L2.SST.NRT.nc
-rw-rw-r-- 1 opuser opuser 19967451 Sep 29 12:49 AQUA_MODIS.20230928T152501.L2.SST.NRT.nc

[oceani]

John and Masoud,

Thank you for all your help on this. You are correct our systems are running relatively old versions of RHEL (6.1 and 7.2). This is because an older image processing application we are using requires older Linux. Unfortunately, we are stuck with it for a while. I imagine the two older versions of wget are the problem. I will work on switching over to curl next week. I try to avoid implementing changes over the weekend. If all goes well, I'll notify you via this thread and let you know that you can switch back to your redirect method. I will also see if I can install wget 1.18 on any of our machines.

Cheers,

Mark

[OB ODPS - jgwilding]

Mark,

Can you try a manual test of the modified getfile? Use the same URL for a file but remove the ob/ or cgi/ path element, so it will be oceandata.sci.gsfc.nasa.gov/getfile/name-of-file

Try that with your usualy wget options.

john

[oceani]

John,

Looks like it worked:
AQUA_MODIS.20230929T073501.L2.SST.NRT.nc

--2023-09-29 11:56:30-- https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20230929T073501.L2.SST.NRT.nc
Resolving oceandata.sci.gsfc.nasa.gov... 169.154.128.84
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FAQUA_MODIS.20230929T073501.L2.SST.NRT.nc [following]
--2023-09-29 11:56:31-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FAQUA_MODIS.20230929T073501.L2.SST.NRT.nc
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/ [following]
--2023-09-29 11:56:32-- https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/
Resolving urs.earthdata.nasa.gov... 198.118.243.33, 2001:4d04081::89
Connecting to urs.earthdata.nasa.gov|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=2954b146155f8fac85edbff25812c8b548ecb256cc6d95256991f5af6d4a22a1 [following]
--2023-09-29 11:56:32-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=2954b146155f8fac85edbff25812c8b548ecb256cc6d95256991f5af6d4a22a1
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20230929T073501.L2.SST.NRT.nc [following]
--2023-09-29 11:56:41-- https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20230929T073501.L2.SST.NRT.nc
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20690714 (20M) [application/octet-stream]
Saving to: “AQUA_MODIS.20230929T073501.L2.SST.NRT.nc”

100%[===============================================>] 20,690,714 1.69M/s in 16s

2023-09-29 11:57:01 (1.21 MB/s) - “AQUA_MODIS.20230929T073501.L2.SST.NRT.nc” saved [20690714/20690714]

Can I ask that you wait until Monday to make any changes to remove the cgi from the path? Unfortunately I have to leave the office this weekend and will be away from access to our scripts/Internet. I can change the URLs in our scripts first thing Monday morning, if that is OK.

Thanks!

Mark

[OB ODPS - jgwilding]

yes, but you won't need to change anything. our original redirects should work. this test bypassed those. we'll leave everything be until Monday.

Thanks.
john

[OB ODPS - jgwilding]

Mark,

We put in the new version of getfile this morning. Hopefully things on your end will continue to work okay, but let us know if not.


john

[oceani]

Looks like all is working fine. Thanks very much for your help.

Mark