bulk index connection timeout caused by readtimeout error

I’m trying to index using bulk index api in es.But index processing seems to get slower and slower even cause connection timeout.The next is my python code.

def bulk_index():
    chunk = []
    for count, doc in enumerate(doc_generator()):
        id = doc.get('_id')
        chunk.append({
            'index': {
                '_id': id
            }
        })
        chunk.append(doc)
        if (count + 1) % 10000 == 0:
            assert len(chunk) == 20000
            res = es.bulk(index='weibo', doc_type='user', body=chunk)
            assert res['errors'] is False
            print 'keke'
            chunk = []
    print 'docs total count: %s' % (count + 1)

[honzakral]

Bulk indexing can be an expensive process so it can indeed take a long time. If it takes more than 10 seconds the only soution is to raise the timeout parameter of the es client. Either for the whole instance or you can specify request_timeout=30 as part of the es.bulk call. There is no way to speed it up from the client - either send smaller bulk requests or raise the timeout value.

also note that you are replicating the chunking logic that we already have as part of the bulk helper: http://elasticsearch-py.readthedocs.org/en/latest/helpers.html#elasticsearch.helpers.bulk

Hope this helps…

[fxdgear]

@vovavovavovavova as of right now we do not support tuple timeouts in elasticsearch-py. Though if that’s something you feel we should add you can create an issue and I’ll look into it.

Personally I’ve never used tuples as timeouts. But I do see that there’d be some value there on the requests side. urllib3 does not support this and as I write this I’m not sure how much work it’d be to do to ensure that it works with requests and not with urllib3. But am willing to investigate.