Reading MERRA-2 data

I’m having the same issue as noted in #63, trying to read MERRA-2 data. I have updated my Pydap to the latest master version from Github, using pip install -e git+https://github.com/pydap/pydap.git#egg=pydap (I think this is the correct way to do it, but please let me know if I’ve gone astray - I haven’t installed packages other than the official release versions before, so I searched around somewhat blindly until finding something that seemed applicable.) I added a print() statement to the open_url() source code to confirm that I was importing the new version in my ipython shell, and it looks like I’m running the new version. However, I’m getting an error for 3-D, as noted in #63, and also now the same error for 2-D data (whereas I was able to read 2-D data with the release version of pydap).

from pydap.client import open_url
from pydap.cas.urs import setup_session

url2 = ('https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2/'
        'M2I1NXASM.5.12.4/2015/08/MERRA2_400.inst1_2d_asm_Nx.20150802.nc4')
url3 = ('https://goldsmr5.gesdisc.eosdis.nasa.gov/opendap/hyrax/MERRA2/'
       'M2I3NVASM.5.12.4/2015/08/MERRA2_400.inst3_3d_asm_Nv.20150802.nc4')

session = setup_session(username, password)
ds2 = open_url(url2, session=session)
ds3 = open_url(url3, session=session)

print(ds2['TS'].shape)
print(ds2['TS'][0, 0, 0])
print(ds3['T'].shape)
print(ds3['T'][0, 0, 0, 0])

I can print the shapes of the arrays and other metadata, but when I try to access the array values in either ds2 or ds3 I get this error:

ValueError                                Traceback (most recent call last)
<ipython-input-23-80aa8f6dccdc> in <module>()
----> 1 ds3['T'][:, 0, 0, 0]

/home/jwalker/dynamics/python/src/pydap/src/pydap/model.pyc in __getitem__(self, index)
    251             return np.vectorize(decode_np_strings)(self.data[index])
    252         else:
--> 253             return self.data[index]
    254
    255     def __len__(self):

/home/jwalker/dynamics/python/src/pydap/src/pydap/handlers/dap.pyc in __getitem__(self, index)
    137         r = GET(url, self.application, self.session)
    138         raise_for_status(r)
--> 139         dds, data = r.body.split(b'\nData:\n', 1)
    140         dds = dds.decode(r.content_encoding or 'ascii')
    141

ValueError: need more than 1 value to unpack

I’m not sure what is going on. Can you please try reading the files in my example and look into it? Any help with this issue would be greatly appreciated. And thank you so so so much for developing these updates to handle the new authentication procedures for NASA data. Such a lifesaver!!

[laliberte]

@jenfly Simply replace the line

session = setup_session(username, password)

by

session = setup_session(username, password, check_url=url2)

This is explained in the updated documentation but unfortunately the readthedocs version was not updated after the last commit so there was no way for you to find out. @jameshiebert any chance the online documentation could be updated?

The reason is that the NASA URS authentication only gives you the right cookies when you actually query some data…

[empeeu]

Wow, nevermind. I followed the link and just followed the instructions. Turns out I had to authorize the NASA GESDISC DATA ARCHIVE applications as part of my Earthdata account.