Functionality of cksum

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
mansi278
Posts: 3
Joined: Wed Feb 19, 2020 9:11 am America/New_York
Answers: 0

Functionality of cksum

by mansi278 » Mon Mar 16, 2020 5:11 am America/New_York

Hello,
I am trying to use cksum in my command however I am not able to find the checksum for each file in the subscription.txt
wget --no-check-certificate --post-data="subID=2452&addurl=1&results_as_file=1&cksums=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi> search.cksums

this is the kind of error I keep getting
2452_path.txt: Scheme missing.

Can anyone please help me understand the functionality of checksum in oceancolor data.

Tags:

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 118
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0

Functionality of cksum

by OB ODPS - jgwilding » Mon Mar 16, 2020 8:09 am America/New_York

First, i think you want cksum=1 instead of cksums=1.  The API ignores cksums=1 (at least for me) and provides me with a list of files for the subscription when i use the URL in your post.

Second, are you running this from a script?  From what i've seen of the Scheme missing error, it is commonly produced if the URL given to wget has too many quotes around it, i.e. single quotes within double quotes or vice versa.

john

mansi278
Posts: 3
Joined: Wed Feb 19, 2020 9:11 am America/New_York
Answers: 0

Functionality of cksum

by mansi278 » Tue Mar 17, 2020 6:22 am America/New_York

Hello John,
Thank you for your response. I added cksum=1 it gives me the text file with checksum values for each image. However, I am not able to download the files. As it keeps throwing the error of scheme missing and value error.
Yes I am running it from a script.

command=('wget --no-check-certificate --post-data="subID=%s&addurl=1&results_as_file=1&cksum=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi>%s' %( subs,downloadfile))

Is there anything I can do it to avoid it or is it not a good option to use cksum? Also will the checksum allow the corrupt files not to be downloaded at all. Considering that it gives the checksum for each image in text file? This might be silly but I am just trying to understand how it works with oceancolor data.
Please suggest.

Thank you.

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 118
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0

Functionality of cksum

by OB ODPS - jgwilding » Tue Mar 17, 2020 4:01 pm America/New_York

The checksums are for you to use on your end once you download a file.  You can generate a checksum (SHA1) and verify that it matches that received from the file-search call.

Generally the file-search call you are using results in output that looks like this:

ace2af707de98f8a2340b7a6a2f9a513cf1f266a  AQUA_MODIS.20200315T092001.L2.SST.NRT.nc

There is nothing in that line that is a download URL.  Although you are specifying the addurl=1 parameter, the cksum=1 negates it.  If you leave off the cksum=1, you then will get URLs that will work with a subsequent wget call.  Discounting all of the authentication arguments, it should be possible to do something like this:

     wget --no-check-certificate --post-data="subID=2452&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi | wget -i -

where the first wget sends its output to stdout, and the second one reads its input from stdin.

If you first saved the file-search output that included the cksum=1, you could then run through the downloaded files, generating checksums and verifying them against the expected ones.

john

Post Reply