More on overlay performance

This is just a follow up to #338, but wanted to make sure someone sees my posts. I was trying to use overlay and noticed it is impossibly slow. So I ended up coding some functions to take care of this. Using the example in #338 I tested and the new functions are much faster, so I am wondering if there is interest and I could create a pull that improves performance. Here’s the function (for now it only implements intersection and difference, but I could generalize it):

def spatial_overlays(df1, df2, how='intersection'):
    '''Compute overlay intersection of two 
        GeoPandasDataFrames df1 and df2
    '''
    df1 = df1.copy()
    df2 = df2.copy()
    df1['geometry'] = df1.geometry.buffer(0)
    df2['geometry'] = df2.geometry.buffer(0)
    if how=='intersection':
        # Spatial Index to create intersections
        spatial_index = df2.sindex
        df1['bbox'] = df1.geometry.apply(lambda x: x.bounds)
        df1['histreg']=df1.bbox.apply(lambda x:list(spatial_index.intersection(x)))
        pairs = df1['histreg'].to_dict()
        nei = []
        for i,j in pairs.items():
            for k in j:
                nei.append([i,k])
        
        pairs = gp.GeoDataFrame(nei, columns=['idx1','idx2'], crs=df1.crs)
        pairs = pairs.merge(df1, left_on='idx1', right_index=True)
        pairs = pairs.merge(df2, left_on='idx2', right_index=True, suffixes=['_1','_2'])
        pairs['Intersection'] = pairs.apply(lambda x: (x['geometry_1'].intersection(x['geometry_2'])).buffer(0), axis=1)
        pairs = gp.GeoDataFrame(pairs, columns=pairs.columns, crs=df1.crs)
        cols = pairs.columns.tolist()
        cols.remove('geometry_1')
        cols.remove('geometry_2')
        cols.remove('histreg')
        cols.remove('bbox')
        cols.remove('Intersection')
        dfinter = pairs[cols+['Intersection']].copy()
        dfinter.rename(columns={'Intersection':'geometry'}, inplace=True)
        dfinter = gp.GeoDataFrame(dfinter, columns=dfinter.columns, crs=pairs.crs)
        dfinter = dfinter.loc[dfinter.geometry.is_empty==False]
        return dfinter
    elif how=='difference':
        spatial_index = df2.sindex
        df1['bbox'] = df1.geometry.apply(lambda x: x.bounds)
        df1['histreg']=df1.bbox.apply(lambda x:list(spatial_index.intersection(x)))
        df1['new_g'] = df1.apply(lambda x: reduce(lambda x, y: x.difference(y).buffer(0), [x.geometry]+list(df2.iloc[x.histreg].geometry)) , axis=1)
        df1.geometry = df1.new_g
        df1 = df1.loc[df1.geometry.is_empty==False].copy()
        df1.drop(['bbox', 'histreg', new_g], axis=1, inplace=True)
        return df1

and the example

import geopandas as gpd
world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
capitals = gpd.read_file(gpd.datasets.get_path('naturalearth_cities'))
countries = world[['geometry', 'name']]
countries = countries.to_crs('+init=epsg:3395')[countries.name!="Antarctica"]
capitals = capitals.to_crs('+init=epsg:3395')
capitals['geometry']= capitals.buffer(500000)

%time gpd.overlay(countries, capitals, how='intersection')
CPU times: user 36.5 s, sys: 357 ms, total: 36.8 s
Wall time: 38.8 s

%time spatial_overlays(countries, capitals, how='intersection')
CPU times: user 1.53 s, sys: 11.3 ms, total: 1.54 s
Wall time: 1.59 s

As you can see an major improvement in performance due to the use of the spatial index.

[jliebrand]

Curious to get all these performance improvements… but it seems there hasn’t been a lot of activity on this for some time. Is there still a plan to merge this in and get it released? Same for #338 ?

[amine-aboufirass]

Has this been integrated into the main repo? My intersections take forever to run