Page 1 of 1

Functionality of cksum

Posted: Mon Mar 16, 2020 5:11 am America/New_York
by mansi278
I am trying to use cksum in my command however I am not able to find the checksum for each file in the subscription.txt
wget --no-check-certificate --post-data="subID=2452&addurl=1&results_as_file=1&cksums=1" -O -> search.cksums

this is the kind of error I keep getting
2452_path.txt: Scheme missing.

Can anyone please help me understand the functionality of checksum in oceancolor data.

Functionality of cksum

Posted: Mon Mar 16, 2020 8:09 am America/New_York
by OB ODPS - jgwilding
First, i think you want cksum=1 instead of cksums=1.  The API ignores cksums=1 (at least for me) and provides me with a list of files for the subscription when i use the URL in your post.

Second, are you running this from a script?  From what i've seen of the Scheme missing error, it is commonly produced if the URL given to wget has too many quotes around it, i.e. single quotes within double quotes or vice versa.


Functionality of cksum

Posted: Tue Mar 17, 2020 6:22 am America/New_York
by mansi278
Hello John,
Thank you for your response. I added cksum=1 it gives me the text file with checksum values for each image. However, I am not able to download the files. As it keeps throwing the error of scheme missing and value error.
Yes I am running it from a script.

command=('wget --no-check-certificate --post-data="subID=%s&addurl=1&results_as_file=1&cksum=1" -O ->%s' %( subs,downloadfile))

Is there anything I can do it to avoid it or is it not a good option to use cksum? Also will the checksum allow the corrupt files not to be downloaded at all. Considering that it gives the checksum for each image in text file? This might be silly but I am just trying to understand how it works with oceancolor data.
Please suggest.

Thank you.

Functionality of cksum

Posted: Tue Mar 17, 2020 4:01 pm America/New_York
by OB ODPS - jgwilding
The checksums are for you to use on your end once you download a file.  You can generate a checksum (SHA1) and verify that it matches that received from the file-search call.

Generally the file-search call you are using results in output that looks like this:


There is nothing in that line that is a download URL.  Although you are specifying the addurl=1 parameter, the cksum=1 negates it.  If you leave off the cksum=1, you then will get URLs that will work with a subsequent wget call.  Discounting all of the authentication arguments, it should be possible to do something like this:

     wget --no-check-certificate --post-data="subID=2452&addurl=1&results_as_file=1" -O - | wget -i -

where the first wget sends its output to stdout, and the second one reads its input from stdin.

If you first saved the file-search output that included the cksum=1, you could then run through the downloaded files, generating checksums and verifying them against the expected ones.