Add BSRN format reader to iotools

I need a parser for the NASA Langley CAPABLE BSRN site. Given the importance and the quality of the BSRN sites, I expect that others would benefit from this parser as well.

Does anyone have experience with this site or have a parser for the format?

https://capable.larc.nasa.gov/data/

https://cove.larc.nasa.gov/BSRN/LRC49/

Data from December 2014 to present.

1 month of data per file. Appears to be uploaded in the first few days of the following month. 1 minute intervals.

Another fun fixed width file. Entries like:

  1  987   1003 -99.9 -999 -999    975 -99.9 -999 -999
             72 -99.9 -999 -999    287 -99.9 -999 -999     19.9  37.2 1026

  1  988   1006 -99.9 -999 -999    977 -99.9 -999 -999
             72 -99.9 -999 -999    290 -99.9 -999 -999     19.8  36.9 1026

  1 1438     22 -99.9 -999 -999      0 -99.9 -999 -999
             21 -99.9 -999 -999    307 -99.9 -999 -999     17.7  57.9 1023

  1 1439     21 -99.9 -999 -999      0 -99.9 -999 -999
             20 -99.9 -999 -999    307 -99.9 -999 -999     17.6  56.8 1023

The first number is the day of the month. The second is the minute of the day. Times appear to be in UTC.

I believe the ordering is:

• CM22 pyranometer GHI (upper left)
• CM31 pyranometer DHI (lower left)
• CH1 pyrheliometer DNI (upper right)
• PIR infrared (lower right)

I’d probably read the file into a DataFrame without meaningful columns, split it into two DataFrames using .iloc[::2] and .iloc[1::2], parse the date time information into and index, then stitch the data back together.

CAPABLE Site Coordinates: Latitude: 37.1038 Longitude: -76.3872 Elevation: 3 m ASL

cross post from https://github.com/SolarArbiter/solarforecastarbiter-core/issues/541

[kanderso-nrel]

I created an account but I still don’t have permission to download the tsv file. I also tried to download the file via ftp with wget but got a “login incorrect” message.

Ah, there is a username and password (and for whatever reason, the account you can create yourself doesn’t work). I think there is just a global login that everyone uses – I don’t feel comfortable sharing it publicly, but you can email Amelie Driemel for it: https://bsrn.awi.de/?id=393.

Do you know if the data available over ftp is in tsv format too?

Looks like the FTP files are in the same format as your example, which I think is called “station-to-archive” format: https://bsrn.awi.de/data/station-to-archive-file-format/

I’d like to automate the fetch and parsing so I’d prefer a regular naming scheme to random DOIs.

I wrote a scraper a while back that uses the pangaea search function to list the datasets I wanted: https://www.pangaea.de/?q=project%3Alabel%3ABSRN+%2Bevent%3Alabel%3ALRC+%2Bcitation%3ABasic+-guidelines

Fetching data from the FTP archive would have been cleaner. I don’t think I knew about the FTP archive back then. So maybe for your use case, implementing the more complex “station-to-archive” format would be better. Seems like the choice is a trade-off between nicer data format and easier file retrieval. Side note: I assume you’ll want to be fetching the data automatically in the future, but if you just want historical BSRN data, I have local copies of all the US station data and can share if you want.

Possibly helpful links:

• PANGAEA data warehouse API? https://www.pangaea.de/about/services.php
• Python3 library to fetch PANGAEA data by DOI: https://github.com/pangaea-data-publisher/pangaeapy
  • Don’t think it knows how to list/search for datasets
• R library for fetching PANGAEA data: https://github.com/ropensci/pangaear
  • Can list/search for datasets

[AdamRJensen]

@wholmgren I have rewritten the function a bit to make it simpler and tested it on a few thousand of BSRN files. It’s my first pull request, so perhaps you could review it and tell me if I am missing something?