'Connection closed by server: input data too slow'

cartoframes version: 1.0b1

when I write a dataframe with shape (159811, 26), it failed with this Error Message

CartoException: [u'Connection closed by server: input data too slow']

CartoExceptionTraceback (most recent call last)
<command-3936245970158062> in <module>()
----> 1 dj.write(data8, 'data8_sample', overwrite=True)

/local_disk0/pythonVirtualEnvDirs/virtualEnv-815f9d21-7f16-4535-965c-e17131b11050/local/lib/python2.7/site-packages/cartoframes/context.pyc in write(self, df, table_name, temp_dir, overwrite, lnglat, encode_geom, geom_col, **kwargs)
    349             if_exists = Dataset.REPLACE
    350 
--> 351         dataset.upload(with_lnglat=lnglat, if_exists=if_exists, table_name=table_name, context=self)
    352 
    353         tqdm.write('Table successfully written to CARTO: {table_url}'.format(

/local_disk0/pythonVirtualEnvDirs/virtualEnv-815f9d21-7f16-4535-965c-e17131b11050/local/lib/python2.7/site-packages/cartoframes/data/dataset.pyc in upload(self, with_lnglat, if_exists, table_name, schema, context)
    419                 raise already_exists_error
    420 
--> 421             self._copyfrom(with_lnglat)
    422 
    423         elif self._query is not None:

/local_disk0/pythonVirtualEnvDirs/virtualEnv-815f9d21-7f16-4535-965c-e17131b11050/local/lib/python2.7/site-packages/cartoframes/data/dataset.pyc in _copyfrom(self, with_lnglat)
    553             """COPY {table_name}({columns},the_geom)
    554                FROM stdin WITH (FORMAT csv, DELIMITER '|');""".format(table_name=self._table_name, columns=columns),
--> 555             self._rows(self._df, [c for c in self._df.columns if c != 'cartodb_id'], with_lnglat, geom_col)
    556         )
    557 

/local_disk0/pythonVirtualEnvDirs/virtualEnv-815f9d21-7f16-4535-965c-e17131b11050/local/lib/python2.7/site-packages/carto/sql.pyc in copyfrom(self, query, iterable_data, compress, compression_level)
    382             raise e
    383         except Exception as e:
--> 384             raise CartoException(e)
    385 
    386         return response_json

CartoException: [u'Connection closed by server: input data too slow']

[andy-esch]

Just tried this and didn’t experience the reported issues.

I mocked Dongjie’s dataset like so:

N = 110000
visits = {f"n_{n}": [randint(0, 25) for _ in range(N)] for n in range(24)}
visits["quadkey_id"] = (
    "".join(np.random.choice(list("0123"), size=16)) for _ in range(N)
)

# then create the polygon from the quadkey

df = gpd.GeoDataFrame(visits)
CartoDataFrame(df).to_carto(table_name="iss908_test", if_exists='replace')

The shape is approximately the same as the one Dongjie tried to upload as are most other data qualities.

[andy-esch]

Btw: you can create a CartoDataFrame directly from the visits variable.

Yes, it works! 😃 I was trying to save time with things I knew worked so didn’t want to risk it before. Turns out it would’ve saved me 15 seconds 😉