Authentication failure - using wget via HPC based Linux

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Tue May 02, 2023 3:34 pm America/New_York

GES DISC - mgreene wrote:
> Following up to my last post to see if you found the information to be
> helpful. thanks

I downloaded the one .nc file manually. However, I get an error message when I try to read it in R or QGIS. In QGIS it says "Unsupported Data Source C:\path-to-the-file is not a supported raster data source". However, other .nc files open without any incident or error on both platforms. Information from my r console

> library(sf)
Linking to GEOS 3.11.1, GDAL 3.6.2, PROJ 9.1.1; sf_use_s2() is TRUE
> library(terra)
terra 1.7.29
> nc_file <- rast("C:/Users/xxx/Documents/sst.mnmean.nc")
>
> nc_file <- rast("C:/Users/xxx/Documents/MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc")
Error: [rast] cannot open this file as a SpatRaster: C:/Users/pma311/Documents/MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc
In addition: Warning message:
`C:/Users/xxx/Documents/MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc' not recognized as a supported file format. (GDAL error 4)
>
Attachments
Screenshot 2023-05-02 134430.jpg
Screenshot 2023-05-02 134430.jpg (57.35 KiB) Not viewed yet

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 176
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Tue May 02, 2023 3:52 pm America/New_York

Thank you, we are looking into this.

GES DISC - mgreene
User Services
User Services
Posts: 176
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Tue May 02, 2023 4:18 pm America/New_York

Hello, please check the size of
'C:/Users/xxx/Documents/MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc' file and
provide us with the URL from which this file was obtained.

pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Wed May 03, 2023 10:33 am America/New_York

GES DISC - mgreene wrote:
> Hello, please check the size of
> 'C:/Users/xxx/Documents/MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc' file
> and
> provide us with the URL from which this file was obtained.

MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc is 2.1GB and I downloaded from https://disc.gsfc.nasa.gov/datasets/M2I3NVAER_5.12.4/summary?keywords=pm1

After defining the date range etc (as shown in the attached image) I click on Get Data. This brings the download link list (shown in the second image). Ideally to download all files, I will download the Download link list and use wget to access or download the dataset. In this case, I click on the individual .nc file name and it downloads the data for that specific file (see the second image beneath the Download list link and Download Instructions tab). I did download the single file to test and that's when I encountered the error reading or open the file in R or QGIS. By the way, I try ArcGIS Pro too and there was an error opening the file.
Attachments
Selecting date range etc
Selecting date range etc
Screenshot 2023-05-03 102604.jpg (220.27 KiB) Not viewed yet
download link list
download link list
Screenshot 2023-05-03 102452.jpg (226.38 KiB) Not viewed yet

pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Wed May 03, 2023 1:42 pm America/New_York

GES DISC - mgreene wrote:
> Hello, please check the size of
> 'C:/Users/xxx/Documents/MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc' file
> and
> provide us with the URL from which this file was obtained.

The MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc file is 2.1GB. I obtained the data from https://disc.gsfc.nasa.gov/datasets/M2I3NVAER_5.12.4/summary?keywords=pm1

Clicked on subset or Get data, then defined the time or date range etc (as shown in the attached picture). Clicked Get Data which took my to the download list link. Beneath the download list link, clicking the individual file name downloads the file directly.
Attachments
Screenshot 2023-05-03 102604.jpg
Screenshot 2023-05-03 102604.jpg (220.27 KiB) Not viewed yet
Screenshot 2023-05-03 102452.jpg
Screenshot 2023-05-03 102452.jpg (226.38 KiB) Not viewed yet

pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Fri May 05, 2023 3:39 pm America/New_York

GES DISC - mgreene wrote:
> Hello, please check the size of
> 'C:/Users/xxx/Documents/MERRA2_400.inst3_3d_aer_Nv.20230101.SUB.nc' file
> and
> provide us with the URL from which this file was obtained.

Considering the size of the M2I3NVAER dataset, is it possible to work with the data remotely without downloading it? That is, do you offer cloud computing options where researchers can work directly with the data and export or download the output? I am trying to model monthly PM10 and PM1 concentrations from the M2I3NVAER dataset.

GES DISC - mgreene
User Services
User Services
Posts: 176
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Wed Jun 07, 2023 1:11 pm America/New_York

Following up to see if you were able to download your data of need successfully.
Please let us know if you need further assistance.

pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Wed Jun 07, 2023 8:49 pm America/New_York

GES DISC - mgreene wrote:
> Following up to see if you were able to download your data of need
> successfully.
> Please let us know if you need further assistance.

I have tried again since my last update due to commitments elsewhere. I will try again and provide feedback. Also, I asked earlier if it is possible to work with the datasets remotely without downloading them. That is, if NASA offers such an opportunity to researchers considering the size of the M2I3NVAER dataset. Thank you.

GES DISC - mgreene
User Services
User Services
Posts: 176
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Fri Jun 09, 2023 5:12 pm America/New_York

Thank you, looking forward to your feedback. In regards to your question a subject matter expert will be in contact with you as soon as possible.

GES DISC - mgreene
User Services
User Services
Posts: 176
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Tue Jun 13, 2023 4:15 pm America/New_York

regarding the inquiry about remote accessing MERRA-2 data.

----------------------------------
We do provide options for remote accessing MERRA-2 data.
1) AWS cloud: MERRA-2 collections have been migrated to the AWS cloud US-west-2 region. So you are in the US-west-2 region, you can access S3 bucket following this How-to: How to Directly Access MERRA-2 Data from an S3 Bucket with Python from a Cloud Environment (https://disc.gsfc.nasa.gov/information/howto?keywords=%22Earthdata%20Cloud%22&title=How%20to%20Directly%20Access%20MERRA-2%20Data%20from%20an%20S3%20Bucket%20with%20Python%20from%20a%20Cloud%20Environment)

2) OPeNDAP: MERRA-2 collections could be remotely accessed through the OPeNDAP data service. You can access it from any place, say, your local machine, following this How-to "How to Access MERRA-2 Data using OPeNDAP with Python3 and Calculate Daily/Weekly/Monthly Statistics from Hourly Data" (https://disc.gsfc.nasa.gov/information/howto?keywords=OPendap&title=How%20to%20Access%20MERRA-2%20Data%20using%20OPeNDAP%20with%20Python3%20and%20Calculate%20Daily%2FWeekly%2FMonthly%20Statistics%20from%20Hourly%20Data%20).

In addition, I would suggest you try an API to subset and download MERRA-2 data, following "How to Use the Web Services API for Subsetting MERRA-2 Data"
(https://disc.gsfc.nasa.gov/information/howto?keywords=API&title=How%20to%20Use%20the%20Web%20Services%20API%20for%20Subsetting%20MERRA-2%20Data)

Note that the speed might be slow in all options above due to the large volume of the 3D collection.

Feel free to let us know if you have any questions.

GES DISC MERRA-2 support team

Post Reply