Hello,
I am trying to use cksum in my command however I am not able to find the checksum for each file in the subscription.txt
wget --no-check-certificate --post-data="subID=2452&addurl=1&results_as_file=1&cksums=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi> search.cksums
this is the kind of error I keep getting
2452_path.txt: Scheme missing.
Can anyone please help me understand the functionality of checksum in oceancolor data.
Functionality of cksum
-
OB ODPS - jgwilding
- Subject Matter Expert
- Posts: 139
- Joined: Fri Feb 19, 2021 1:09 pm America/New_York
- Been thanked: 1 time
Functionality of cksum
First, i think you want cksum=1 instead of cksums=1. The API ignores cksums=1 (at least for me) and provides me with a list of files for the subscription when i use the URL in your post.
Second, are you running this from a script? From what i've seen of the Scheme missing error, it is commonly produced if the URL given to wget has too many quotes around it, i.e. single quotes within double quotes or vice versa.
john
Second, are you running this from a script? From what i've seen of the Scheme missing error, it is commonly produced if the URL given to wget has too many quotes around it, i.e. single quotes within double quotes or vice versa.
john
Functionality of cksum
Hello John,
Thank you for your response. I added cksum=1 it gives me the text file with checksum values for each image. However, I am not able to download the files. As it keeps throwing the error of scheme missing and value error.
Yes I am running it from a script.
command=('wget --no-check-certificate --post-data="subID=%s&addurl=1&results_as_file=1&cksum=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi>%s' %( subs,downloadfile))
Is there anything I can do it to avoid it or is it not a good option to use cksum? Also will the checksum allow the corrupt files not to be downloaded at all. Considering that it gives the checksum for each image in text file? This might be silly but I am just trying to understand how it works with oceancolor data.
Please suggest.
Thank you.
Thank you for your response. I added cksum=1 it gives me the text file with checksum values for each image. However, I am not able to download the files. As it keeps throwing the error of scheme missing and value error.
Yes I am running it from a script.
command=('wget --no-check-certificate --post-data="subID=%s&addurl=1&results_as_file=1&cksum=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi>%s' %( subs,downloadfile))
Is there anything I can do it to avoid it or is it not a good option to use cksum? Also will the checksum allow the corrupt files not to be downloaded at all. Considering that it gives the checksum for each image in text file? This might be silly but I am just trying to understand how it works with oceancolor data.
Please suggest.
Thank you.
-
OB ODPS - jgwilding
- Subject Matter Expert
- Posts: 139
- Joined: Fri Feb 19, 2021 1:09 pm America/New_York
- Been thanked: 1 time
Functionality of cksum
The checksums are for you to use on your end once you download a file. You can generate a checksum (SHA1) and verify that it matches that received from the file-search call.
Generally the file-search call you are using results in output that looks like this:
ace2af707de98f8a2340b7a6a2f9a513cf1f266a AQUA_MODIS.20200315T092001.L2.SST.NRT.nc
There is nothing in that line that is a download URL. Although you are specifying the addurl=1 parameter, the cksum=1 negates it. If you leave off the cksum=1, you then will get URLs that will work with a subsequent wget call. Discounting all of the authentication arguments, it should be possible to do something like this:
wget --no-check-certificate --post-data="subID=2452&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi | wget -i -
where the first wget sends its output to stdout, and the second one reads its input from stdin.
If you first saved the file-search output that included the cksum=1, you could then run through the downloaded files, generating checksums and verifying them against the expected ones.
john
Generally the file-search call you are using results in output that looks like this:
ace2af707de98f8a2340b7a6a2f9a513cf1f266a AQUA_MODIS.20200315T092001.L2.SST.NRT.nc
There is nothing in that line that is a download URL. Although you are specifying the addurl=1 parameter, the cksum=1 negates it. If you leave off the cksum=1, you then will get URLs that will work with a subsequent wget call. Discounting all of the authentication arguments, it should be possible to do something like this:
wget --no-check-certificate --post-data="subID=2452&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi | wget -i -
where the first wget sends its output to stdout, and the second one reads its input from stdin.
If you first saved the file-search output that included the cksum=1, you could then run through the downloaded files, generating checksums and verifying them against the expected ones.
john