Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Transformation grid unavailable via network won't fall back to data dir

See original GitHub issue

Problem description

Hello. I am setting up a transformation from EPSG 9306 to EPSG 4326. The best transformation uses a grid HS2TN15_NTv2.gsb. Since this grid is proprietary, it’s not available via the network:

>>> import pyproj
>>> pyproj.network.set_network_enabled(True)
>>> tg = pyproj.transformer.TransformerGroup('epsg:9306', 'epsg:4326', always_xy=True)
/opt/homebrew/lib/python3.9/site-packages/pyproj/transformer.py:81: UserWarning: Best transformation is not available due to missing Grid(short_name=HS2TN15_NTv2.gsb, full_name=, package_name=, url=, direct_download=False, open_license=False, available=False)
>>> tg.transformers[0].transform(282000, 287000)
(-0.30282086906056793, 51.50241100037383)

Fortunately, my client has provided this grid file to me. I’ve put it in the pyproj.datadir.get_data_dir(). If I have the network turned off, it works fine:

>>> import pyproj, os
>>> pyproj.network.set_network_enabled(False)
>>> assert os.path.exists(os.path.join(pyproj.datadir.get_data_dir(), 'HS2TN15_NTv2.gsb'))
>>> tg = pyproj.transformer.TransformerGroup('epsg:9306', 'epsg:4326', always_xy=True)
>>> tg.transformers[0].transform(282000, 287000)
(-0.276600729863347, 51.5176500133351)
>>> tg.transformers[0].definition
'proj=pipeline step inv proj=tmerc lat_0=52.3 lon_0=-1.5 k=1 x_0=198873.0046 y_0=375064.3871 ellps=GRS80 step proj=hgridshift grids=HS2TN15_NTv2.gsb step proj=unitconvert xy_in=rad xy_out=deg'

But if I have the network turned on, like I usually do for loading other transformation grids, it does not find the file in the local directory, and it uses the ballpark transformation instead.

>>> import pyproj, os
>>> pyproj.network.set_network_enabled(True)
>>> assert os.path.exists(os.path.join(pyproj.datadir.get_data_dir(), 'HS2TN15_NTv2.gsb'))
>>> tg = pyproj.transformer.TransformerGroup('epsg:9306', 'epsg:4326', always_xy=True)
/usr/local/lib/python3.7/dist-packages/pyproj/transformer.py:86: UserWarning: Best transformation is not available due to missing Grid(short_name=HS2TN15_NTv2.gsb, full_name=HS2TN15_NTv2.gsb, package_name=, url=, direct_download=False, open_license=False, available=False)
>>> tg.transformers[0].transform(282000, 287000)
(-0.30282086906056793, 51.502411000373826)
>>> tg.transformers[0].definition
'proj=pipeline step inv proj=tmerc lat_0=52.3 lon_0=-1.5 k=1 x_0=198873.0046 y_0=375064.3871 ellps=GRS80 step proj=unitconvert xy_in=rad xy_out=deg'

Should PyProj be falling back to the local data directory if the transformation grid is unavailable via the network? The debugging output (below) shows the grid file being opened, it’s just not being used as a fallback like I would expect.

Full debugging output with PROJ_DEBUG=3 
import logging
console_handler = logging.StreamHandler()
formatter = logging.Formatter("%(levelname)s:%(message)s")
console_handler.setFormatter(formatter)
logger = logging.getLogger("pyproj")
logger.addHandler(console_handler)
logger.setLevel(logging.DEBUG)

import pyproj, os
pyproj.network.set_network_enabled(True)
assert os.path.exists(os.path.join(pyproj.datadir.get_data_dir(), 'HS2TN15_NTv2.gsb'))
tg = pyproj.transformer.TransformerGroup('epsg:9306', 'epsg:4326', always_xy=True)
tg.transformers[0].transform(282000, 287000)
tg.transformers[0].definition

DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/usr/local/lib/python3.7/dist-packages/pyproj/proj_dir/share/proj/proj.ini) - succeeded
DEBUG:PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
DEBUG:PROJ_DEBUG: pj_open_lib(HS2TN15_NTv2.gsb): call fopen(/usr/local/lib/python3.7/dist-packages/pyproj/proj_dir/share/proj/HS2TN15_NTv2.gsb) - succeeded
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 0
DEBUG:PROJ_TRACE: Pipeline: init - inv, 8
DEBUG:PROJ_TRACE:     proj=tmerc
DEBUG:PROJ_TRACE:     lat_0=52.3
DEBUG:PROJ_TRACE:     lon_0=-1.5
DEBUG:PROJ_TRACE:     k=1
DEBUG:PROJ_TRACE:     x_0=198873.0046
DEBUG:PROJ_TRACE:     y_0=375064.3871
DEBUG:PROJ_TRACE:     ellps=GRS80
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: Pipeline: Step 0 (inv) at 0x143b6f0
DEBUG:PROJ_TRACE: Pipeline at [0x1332140]:    step at [0x143b6f0] (inv) done
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 1
DEBUG:PROJ_TRACE: Pipeline: init - proj=hgridshift, 2
DEBUG:PROJ_TRACE:     grids=HS2TN15_NTv2.gsb
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: Pipeline: Step 1 (proj=hgridshift) at 0x143b9e0
DEBUG:PROJ_TRACE: Pipeline at [0x1332140]:    step at [0x143b9e0] (proj=hgridshift) done
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 2
DEBUG:PROJ_TRACE: Pipeline: init - proj=unitconvert, 3
DEBUG:PROJ_TRACE:     xy_in=rad
DEBUG:PROJ_TRACE:     xy_out=deg
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: xy_in unit: Radian
DEBUG:PROJ_TRACE: xy_out unit: Degree
DEBUG:PROJ_TRACE: Pipeline: Step 2 (proj=unitconvert) at 0x143bd50
DEBUG:PROJ_TRACE: Pipeline at [0x1332140]:    step at [0x143bd50] (proj=unitconvert) done
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 3
DEBUG:PROJ_TRACE: Pipeline: init - proj=axisswap, 2
DEBUG:PROJ_TRACE:     order=2,1
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: Pipeline: Step 3 (proj=axisswap) at 0x143c0a0
DEBUG:PROJ_TRACE: Pipeline at [0x1332140]:    step at [0x143c0a0] (proj=axisswap) done
DEBUG:PROJ_TRACE: Pipeline: 4 steps built. Determining i/o characteristics
DEBUG:PROJ_ERROR: proj_coordoperation_get_method_info: Object is not a DerivedCRS or BoundCRS
/usr/local/lib/python3.7/dist-packages/pyproj/transformer.py:86: UserWarning: Best transformation is not available due to missing Grid(short_name=HS2TN15_NTv2.gsb, full_name=HS2TN15_NTv2.gsb, package_name=, url=, direct_download=False, open_license=False, available=False)
  area_of_interest=area_of_interest,
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 0
DEBUG:PROJ_TRACE: Pipeline: init - inv, 8
DEBUG:PROJ_TRACE:     proj=tmerc
DEBUG:PROJ_TRACE:     lat_0=52.3
DEBUG:PROJ_TRACE:     lon_0=-1.5
DEBUG:PROJ_TRACE:     k=1
DEBUG:PROJ_TRACE:     x_0=198873.0046
DEBUG:PROJ_TRACE:     y_0=375064.3871
DEBUG:PROJ_TRACE:     ellps=GRS80
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: Pipeline: Step 0 (inv) at 0x165ac20
DEBUG:PROJ_TRACE: Pipeline at [0x165a8d0]:    step at [0x165ac20] (inv) done
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 1
DEBUG:PROJ_TRACE: Pipeline: init - proj=unitconvert, 3
DEBUG:PROJ_TRACE:     xy_in=rad
DEBUG:PROJ_TRACE:     xy_out=deg
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: xy_in unit: Radian
DEBUG:PROJ_TRACE: xy_out unit: Degree
DEBUG:PROJ_TRACE: Pipeline: Step 1 (proj=unitconvert) at 0x165afb0
DEBUG:PROJ_TRACE: Pipeline at [0x165a8d0]:    step at [0x165afb0] (proj=unitconvert) done
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 2
DEBUG:PROJ_TRACE: Pipeline: init - proj=axisswap, 2
DEBUG:PROJ_TRACE:     order=2,1
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: Pipeline: Step 2 (proj=axisswap) at 0x165b370
DEBUG:PROJ_TRACE: Pipeline at [0x165a8d0]:    step at [0x165b370] (proj=axisswap) done
DEBUG:PROJ_TRACE: Pipeline: 3 steps built. Determining i/o characteristics
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 0
DEBUG:PROJ_TRACE: Pipeline: init - inv, 8
DEBUG:PROJ_TRACE:     proj=tmerc
DEBUG:PROJ_TRACE:     lat_0=52.3
DEBUG:PROJ_TRACE:     lon_0=-1.5
DEBUG:PROJ_TRACE:     k=1
DEBUG:PROJ_TRACE:     x_0=198873.0046
DEBUG:PROJ_TRACE:     y_0=375064.3871
DEBUG:PROJ_TRACE:     ellps=GRS80
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: Pipeline: Step 0 (inv) at 0x1471270
DEBUG:PROJ_TRACE: Pipeline at [0x14cf550]:    step at [0x1471270] (inv) done
DEBUG:PROJ_TRACE: Pipeline: Building arg list for step no. 1
DEBUG:PROJ_TRACE: Pipeline: init - proj=unitconvert, 3
DEBUG:PROJ_TRACE:     xy_in=rad
DEBUG:PROJ_TRACE:     xy_out=deg
DEBUG:PROJ_TRACE: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
DEBUG:PROJ_TRACE: pj_ellipsoid - final:    ellps=GRS80
DEBUG:PROJ_TRACE: xy_in unit: Radian
DEBUG:PROJ_TRACE: xy_out unit: Degree
DEBUG:PROJ_TRACE: Pipeline: Step 1 (proj=unitconvert) at 0x1471660
DEBUG:PROJ_TRACE: Pipeline at [0x14cf550]:    step at [0x1471660] (proj=unitconvert) done
DEBUG:PROJ_TRACE: Pipeline: 2 steps built. Determining i/o characteristics

I also tried putting the file into pyproj.datadir.get_user_data_dir(), and got the same result. I also tried this whole process with PyProj 3.3.1 and got the same result.

Workaround

As a temporary workaround, I am using Transformer.from_pipeline() to instantiate the transformer directly from the PROJ pipeline definition string, and that appears to be working. But managing these pipeline strings is, of course, less convenient than a simple EPSG code.

>>> import pyproj, os
>>> pyproj.network.set_network_enabled(True)
>>> assert os.path.exists(os.path.join(pyproj.datadir.get_data_dir(), 'HS2TN15_NTv2.gsb'))
>>> definition = 'proj=pipeline step inv proj=tmerc lat_0=52.3 lon_0=-1.5 k=1 x_0=198873.0046 y_0=375064.3871 ellps=GRS80 step proj=hgridshift grids=HS2TN15_NTv2.gsb step proj=unitconvert xy_in=rad xy_out=deg'
>>> transformer = pyproj.Transformer.from_pipeline(definition)
>>> transformer.transform(282000, 287000)
(-0.276600729863347, 51.5176500133351)

Environment Information

pyproj info:
    pyproj: 3.0.1
      PROJ: 7.2.1
  data dir: /usr/local/lib/python3.7/dist-packages/pyproj/proj_dir/share/proj
user_data_dir: /root/.local/share/proj

System:
    python: 3.7.5 (default, Dec  9 2021, 17:04:37)  [GCC 8.4.0]
executable: /usr/bin/python3.7
   machine: Linux-5.10.76-linuxkit-x86_64-with-Ubuntu-18.04-bionic

Python deps:
/usr/local/lib/python3.7/dist-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
       pip: 22.0.4
setuptools: 62.1.0
    Cython: None

Installation method

python3.7 -m pip install pyproj==3.0.1

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:9 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
rouaultcommented, Aug 8, 2022

are you able to confirm the source of the fix?

hard to tell. That’s plausible, but I honestly can’t grasp all the consequences of the changes I do 😃 But if it works, that’s the main thing

0reactions
dmahr1commented, Aug 11, 2022

@snowman2 Opened an issue in PROJ: https://github.com/OSGeo/PROJ/issues/3284

We can continue the discussion over there. Thanks for your help. Closing this issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Transformation grid management · Issue #747 · pyproj4/pyproj
If I run proj transformation for this coordinate in both images I get the following result: docker run pyproj:3.0.0 echo 6452293.98 ...
Read more >
QGIS cannot find transformation grid - GIS Stack Exchange
When I open an existing QGIS project I get an error bar at the top of the mapping window stating that some data...
Read more >
Troubleshooting transforms | Elasticsearch Guide [8.5] | Elastic
Use the information in this section to troubleshoot common problems. For issues that you cannot fix yourself … we're here to help. If...
Read more >
How to Fix the DataSource.Error Message in Power Query
If you're sharing or collaborating with other intermediate Power Query users and source data from large files, you don't want to send those ......
Read more >
Transformation Grids - pyproj 3.4.1rc0 documentation
pyproj API for managing the Data Directory and PROJ Network Settings. Note. pyproj 3 wheels do not include any transformation grids.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Segmentation Fault on pyproj.transform

Next page

Unable to install using pip on Cygwin