Recommendations to avoid Blacklisting
Posted: Fri Jan 21, 2022 3:45 am America/New_York
Hi,
We have a download chain since a long time to download your L2 products, and it seems that we were blacklisted from (nearly) 2022-01-19 17:00 UTC to 2022-01-20 12:00 UTC. I think this is related to the recent reprocessing concerning ozone issue because our chain have downloaded more files, but our chain applies the following logic to avoid blacklisting and we try to understand why is it not enough: our download chain ensure a minimum delay of 5 seconds between each HTTP request to your website. Is it enough? Because sometimes we have this kind of error:
HTTPError: 429 Client Error: Too Many Requests for url: https://oceandata.sci.gsfc.nasa.gov/ob/ ... SNPP_OC.nc
and I think we did not have this error in the past.
I've just tried to increase the delay to 10 seconds and it seems better for the moment.
Maybe we could improve our download approach to avoid these blacklistings? What delay do yo recommend between 2 HTTP requests? Are there other recommendations to avoid blacklisting?
Note that it is also difficult on our side to ensure that no other people from our company does other manual access to your website because the same IP is also used for general usage (firefox, etc...). Do you have recommendations about this problem?
Many thanks in advance,
Julien
We have a download chain since a long time to download your L2 products, and it seems that we were blacklisted from (nearly) 2022-01-19 17:00 UTC to 2022-01-20 12:00 UTC. I think this is related to the recent reprocessing concerning ozone issue because our chain have downloaded more files, but our chain applies the following logic to avoid blacklisting and we try to understand why is it not enough: our download chain ensure a minimum delay of 5 seconds between each HTTP request to your website. Is it enough? Because sometimes we have this kind of error:
HTTPError: 429 Client Error: Too Many Requests for url: https://oceandata.sci.gsfc.nasa.gov/ob/ ... SNPP_OC.nc
and I think we did not have this error in the past.
I've just tried to increase the delay to 10 seconds and it seems better for the moment.
Maybe we could improve our download approach to avoid these blacklistings? What delay do yo recommend between 2 HTTP requests? Are there other recommendations to avoid blacklisting?
Note that it is also difficult on our side to ensure that no other people from our company does other manual access to your website because the same IP is also used for general usage (firefox, etc...). Do you have recommendations about this problem?
Many thanks in advance,
Julien