Import API performance: Importing 'nothing' takes 5 seconds
See original GitHub issueContext
I’m trying to create an easy, programmatic way to create a table on CARTO that’s builder friendly and engine friendly (for use in cartoframes), so I tried a function like this out:
import requests
import time
APIKEY = 'abcdefg'
def _table_touch(table_name, schema):
resp = requests.post(
'https://andycopy2.carto-staging.com/api/v1/imports',
params={'sql': 'SELECT {} LIMIT 0'.format(schema), 'table_name': table_name, 'api_key': APIKEY}
)
while True:
r = requests.get(
'https://andycopy2.carto-staging.com/api/v1/imports/{}'.format(resp.json()['item_queue_id']),
params={'api_key': APIKEY}
)
if r.json()['state'] == 'complete':
break
elif r.json()['state'] == 'failure':
raise Exception(str(r))
time.sleep(0.1)
return r.json()['table_name']
This create a table with a specific schema and a specific name using the Import API’s sql
and table_name
args. The problem is it takes ~4-10+ seconds to create a table with no rows. I would expect this type of operation to be on the order of 0.5 seconds instead.
Steps to Reproduce
- Update that script above with an API key and correct base url
- Run it with a custom table name and ‘schema’ like so:
schema = 'null::numeric as mag, null::text as place, null::text as net, null::numeric as depth, null::text as id, null::numeric as longitude, null::numeric as latitude, null::text as magtype, null::numeric as rms' t_start = time.time() _table_touch('abcdefghij', schema) print(time.time() - t_start)
- Observe the time to create the table (in my tests ~4-10+ seconds)
Current Result
Creating a table with no rows over the Import API takes ~5 seconds
Expected result
That creating a table with no rows would only take a small fraction of a second
Browser and version
n/a
.carto file
n/a
Additional info
nope, thanks
cc @pramsey
Issue Analytics
- State:
- Created 5 years ago
- Reactions:1
- Comments:9 (7 by maintainers)
Top Results From Across the Web
Speeding up the python "import" loader - Stack Overflow
For a complicated project environment, with tons of different directories, this can take around 5 seconds -- all to run a script that...
Read more >import tensorflow is very slow · Issue #37729 - GitHub
However, every time I import tf, it takes about 5 seconds. import tensorflow is 10 times slower than import torch .
Read more >API Design Guidance: Bulk and Batch Import - Tyk.io
For bulk imports that will take longer than a few seconds, the API should return validate the request, queue the work to be...
Read more >Python import: Advanced Techniques and Tips
The Python import system is as powerful as it is useful. In this in-depth tutorial, you'll learn how to harness this power to...
Read more >A Guide to Troubleshooting Import Set Issues - ServiceNow
Finally, we must make sure the expectation is correct. For example, importing 100 records per second or transforming 20 records per second is ......
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
There needs to be a fast, reliable, programmatic way to create a table that is registered to the UI, to support CartoFrames and anyone else who wants to bring data automatically. Otherwise the data is imported but users cannot tell, as it doesn’t show up in the UI until (pages are refreshed|a timer tickets over|something).
I’d think about this request in the lights of a new dashboard vs editor/builder, plus the ability to instantiate maps (isn’t that right, @andy-esch ?) , @inigomedina and @rochoa