Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
DataFrame merge significantly slower in Python 3.6 and 3.7
See original GitHub issueCode Sample, a copy-pastable example if possible
import numpy as np
import pandas as pd
from line_profiler import LineProfiler
left_on = right_on = ['object_7', 'object_3', 'int64_21']
column_types = {
'object_5': 'object', 'int32_5': 'int32', 'object_14': 'object', 'int64_6': 'int64',
'object_6': 'object', 'object_10': 'object', 'object_16': 'object', 'int64_23': 'int64',
'int32_4': 'int32', 'object_31': 'object', 'int64_11': 'int64',
'int32_9': 'int32', 'int64_5': 'int64', 'int32_10': 'int32', 'int64_24': 'int64',
'object_27': 'object', 'object_22': 'object', 'object_20': 'object',
'object_8': 'object', 'int64_16': 'int64', 'int64_1': 'int64', 'object_29': 'object',
'object_7': 'object', 'object_9': 'object', 'int64_13': 'int64', 'int64_9': 'int64',
'object_11': 'object', 'object_28': 'object', 'int32_2': 'int32',
'int32_6': 'int32', 'int64_21': 'int64', 'object_25': 'object', 'int32_1': 'int32',
'int64_8': 'int64', 'object_15': 'object', 'object_21': 'object', 'int32_12': 'int32',
'int64_3': 'int64', 'int64_7': 'int64', 'object_13': 'object', 'int64_2': 'int64',
'object_30': 'object', 'int64_4': 'int64', 'int64_18': 'int64', 'int32_14': 'int32',
'object_2': 'object', 'int64_19': 'int64', 'int64_10': 'int64', 'float64_2': 'float64',
'object_32': 'object', 'object_19': 'object', 'object_26': 'object',
'int32_8': 'int32', 'int64_17': 'int64', 'object_1': 'object', 'int32_3': 'int32',
'object_17': 'object', 'object_4': 'object', 'int32_13': 'int32', 'object_3': 'object',
'int64_15': 'int64', 'object_18': 'object', 'int64_22': 'int64',
'int32_15': 'int32', 'int64_12': 'int64', 'object_23': 'object',
'int64_14': 'int64', 'int32_11': 'int32', 'int32_7': 'int32', 'int64_20': 'int64',
'object_12': 'object', 'float64_1': 'float64', 'object_24': 'object'}
left_cols = [
'float64_1', 'object_3', 'object_4', 'object_5', 'object_6', 'object_7', 'int64_1',
'object_8', 'int32_1', 'object_9', 'float64_2', 'int64_3', 'int64_4',
'object_10', 'object_11', 'int64_7', 'int64_8', 'int64_10', 'int32_4',
'int64_11', 'int64_12', 'int64_6', 'int32_5', 'int32_6',
'int64_14', 'int64_15', 'int32_8', 'object_12', 'object_13', 'object_14', 'object_15',
'int32_11', 'object_16', 'int64_16', 'object_17', 'object_18',
'object_19', 'object_20', 'object_21', 'object_22',
'object_23', 'object_24', 'object_25',
'object_26', 'int64_18', 'int32_14', 'object_27', 'object_28',
'int64_20', 'int64_21', 'object_29', 'object_30', 'object_31', 'int64_22',
'int64_23', 'object_32', 'int32_15', 'int64_17', 'object_1', 'object_2',
'int32_9', 'int32_10', 'int32_13', 'int32_12', 'int64_13', 'int32_7', 'int32_2',
'int32_3', 'int64_9', 'int64_24', 'int64_2', 'int64_19']
left_merge_tuples = \
([('4', '2', 1)] * 1) + \
([('4', '2', 18)] * 2) + \
([('4', '2', 19)] * 1) + \
([('5', '1', 1)] * 1) + \
([('5', '1', 22)] * 1) + \
([('5', '1', 19)] * 4) + \
([('5', '1', 4)] * 3) + \
([('5', '1', 6)] * 4) + \
([('6', '7', 2)] * 3) + \
([('6', '7', 18)] * 3) + \
([('6', '1', 2)] * 10) + \
([('6', '1', 18)] * 10) + \
([('6', '1', 19)] * 10) + \
([('6', '1', 3)] * 10) + \
([('6', '1', 17)] * 10) + \
([('6', '1', 4)] * 10) + \
([('6', '1', 23)] * 10) + \
([('6', '1', 5)] * 4) + \
([('6', '2', 1)] * 5437) + \
([('7', '7', 1)] * 1577) + \
([('7', '3', 1)] * 277) + \
([('7', '6', 1)] * 24) + \
([('7', '1', 1)] * 7945) + \
([('11', '2', 11)] * 4515) + \
([('13', '2', 12)] * 3978) + \
([('14', '2', 4)] * 5489) + \
([('15', '2', 13)] * 3478) + \
([('16', '2', 5)] * 5479) + \
([('17', '2', 14)] * 3281) + \
([('18', '2', 2)] * 5742) + \
([('19', '2', 6)] * 5439) + \
([('20', '2', 15)] * 3153) + \
([('21', '2', 7)] * 5357) + \
([('22', '2', 16)] * 2701) + \
([('23', '2', 8)] * 5290) + \
([('24', '7', 2)] * 3154) + \
([('24', '3', 2)] * 283) + \
([('24', '6', 2)] * 44) + \
([('24', '1', 2)] * 8276) + \
([('24', '2', 3)] * 5490) + \
([('25', '2', 9)] * 5011) + \
([('26', '2', 10)] * 4674) + \
([('37', '7', 11)] * 2729) + \
([('37', '3', 11)] * 330) + \
([('37', '6', 11)] * 49) + \
([('37', '1', 11)] * 15534) + \
([('39', '7', 12)] * 2600) + \
([('39', '3', 12)] * 259) + \
([('39', '6', 12)] * 46) + \
([('39', '1', 12)] * 14711) + \
([('40', '7', 4)] * 1999) + \
([('40', '3', 4)] * 480) + \
([('40', '6', 4)] * 40) + \
([('40', '1', 4)] * 16869) + \
([('41', '7', 13)] * 2382) + \
([('41', '3', 13)] * 224) + \
([('41', '6', 13)] * 41) + \
([('41', '1', 13)] * 13263) + \
([('42', '7', 5)] * 1949) + \
([('42', '3', 5)] * 566) + \
([('42', '6', 5)] * 57) + \
([('42', '1', 5)] * 16967) + \
([('43', '7', 14)] * 2213) + \
([('43', '3', 14)] * 171) + \
([('43', '6', 14)] * 38) + \
([('43', '1', 14)] * 12044) + \
([('44', '7', 6)] * 1943) + \
([('44', '3', 6)] * 568) + \
([('44', '6', 6)] * 57) + \
([('44', '1', 6)] * 16746) + \
([('45', '7', 15)] * 2178) + \
([('45', '3', 15)] * 127) + \
([('45', '6', 15)] * 37) + \
([('45', '1', 15)] * 11506) + \
([('46', '7', 7)] * 3131) + \
([('46', '3', 7)] * 570) + \
([('46', '6', 7)] * 57) + \
([('46', '1', 7)] * 16781) + \
([('47', '7', 16)] * 2076) + \
([('47', '3', 16)] * 90) + \
([('47', '6', 16)] * 34) + \
([('47', '1', 16)] * 9999) + \
([('48', '7', 8)] * 3016) + \
([('48', '3', 8)] * 475) + \
([('48', '6', 8)] * 53) + \
([('48', '1', 8)] * 16506) + \
([('49', '7', 3)] * 3524) + \
([('49', '3', 3)] * 477) + \
([('49', '6', 3)] * 51) + \
([('49', '1', 3)] * 16585) + \
([('50', '7', 9)] * 2839) + \
([('50', '3', 9)] * 403) + \
([('50', '6', 9)] * 53) + \
([('50', '1', 9)] * 16089) + \
([('51', '7', 10)] * 2779) + \
([('51', '3', 10)] * 376) + \
([('51', '6', 10)] * 49) + \
([('51', '1', 10)] * 15783) + \
([('57', '7', 1)] * 3) + \
([('57', '7', 2)] * 3) + \
([('57', '7', 18)] * 3) + \
([('57', '7', 19)] * 3) + \
([('57', '7', 3)] * 3) + \
([('57', '7', 17)] * 3) + \
([('57', '7', 4)] * 3) + \
([('57', '7', 23)] * 3) + \
([('68', '2', 1)] * 10) + \
([('68', '2', 18)] * 10) + \
([('68', '2', 19)] * 10) + \
([('68', '2', 3)] * 10) + \
([('68', '2', 17)] * 10) + \
([('68', '2', 4)] * 10) + \
([('68', '2', 23)] * 7) + \
([('68', '2', 5)] * 7) + \
([('68', '2', 6)] * 7) + \
([('68', '2', 7)] * 4) + \
([('74', '1', 1)] * 10) + \
([('74', '1', 18)] * 10) + \
([('74', '1', 19)] * 10) + \
([('74', '1', 3)] * 10) + \
([('74', '1', 17)] * 10) + \
([('74', '1', 4)] * 10) + \
([('74', '1', 23)] * 10) + \
([('74', '1', 5)] * 4)
left_merge_df = pd.DataFrame(left_merge_tuples, columns=left_on)
for col in right_on:
left_merge_df[col] = left_merge_df[col].astype(column_types[col])
right_cols = ['object_7', 'object_3', 'int64_21', 'int64_5']
right_merge_tuples = \
[('1', '1', 1)] + \
[('1', '1', 2)] + \
[('1', '1', 3)] + \
[('1', '1', 4)] + \
[('1', '1', 5)] + \
[('1', '1', 6)] + \
[('1', '1', 7)] + \
[('1', '1', 8)] + \
[('1', '1', 9)] + \
[('1', '1', 10)] + \
[('1', '1', 11)] + \
[('1', '1', 12)] + \
[('1', '1', 13)] + \
[('1', '1', 14)] + \
[('1', '1', 15)] + \
[('1', '1', 16)] + \
[('1', '2', 1)] + \
[('1', '2', 2)] + \
[('1', '2', 3)] + \
[('1', '2', 17)] + \
[('1', '2', 4)] + \
[('1', '2', 5)] + \
[('1', '2', 6)] + \
[('1', '2', 7)] + \
[('1', '2', 8)] + \
[('1', '2', 9)] + \
[('1', '2', 10)] + \
[('1', '2', 11)] + \
[('1', '2', 12)] + \
[('1', '2', 13)] + \
[('1', '2', 14)] + \
[('1', '2', 15)] + \
[('1', '2', 16)] + \
[('2', '3', 1)] + \
[('2', '3', 2)] + \
[('2', '3', 3)] + \
[('2', '3', 4)] + \
[('2', '3', 5)] + \
[('2', '3', 6)] + \
[('2', '3', 7)] + \
[('2', '3', 8)] + \
[('2', '3', 9)] + \
[('2', '3', 10)] + \
[('2', '3', 11)] + \
[('2', '3', 12)] + \
[('2', '3', 13)] + \
[('2', '3', 14)] + \
[('2', '3', 15)] + \
[('2', '3', 16)] + \
[('3', '1', 1)] + \
[('3', '1', 2)] + \
[('3', '1', 3)] + \
[('3', '1', 4)] + \
[('3', '1', 5)] + \
[('3', '1', 6)] + \
[('3', '1', 7)] + \
[('3', '1', 8)] + \
[('3', '1', 9)] + \
[('3', '1', 10)] + \
[('3', '1', 11)] + \
[('3', '1', 12)] + \
[('3', '1', 13)] + \
[('3', '1', 14)] + \
[('3', '1', 15)] + \
[('3', '1', 16)] + \
[('4', '2', 1)] + \
[('4', '2', 2)] + \
[('4', '2', 3)] + \
[('4', '2', 4)] + \
[('4', '2', 5)] + \
[('4', '2', 6)] + \
[('4', '2', 7)] + \
[('4', '2', 8)] + \
[('4', '2', 9)] + \
[('4', '2', 10)] + \
[('4', '2', 11)] + \
[('4', '2', 20)] + \
[('4', '2', 12)] + \
[('4', '2', 13)] + \
[('4', '2', 21)] + \
[('4', '2', 14)] + \
[('5', '4', 4)] + \
[('5', '5', 1)] + \
[('5', '5', 2)] + \
[('5', '5', 3)] + \
[('5', '5', 4)] + \
[('5', '5', 5)] + \
[('5', '5', 6)] + \
[('5', '5', 7)] + \
[('5', '5', 8)] + \
[('5', '5', 9)] + \
[('5', '5', 10)] + \
[('5', '5', 11)] + \
[('5', '5', 20)] + \
[('5', '5', 12)] + \
[('5', '5', 13)] + \
[('5', '5', 21)] + \
[('5', '5', 14)] + \
[('5', '3', 1)] + \
[('5', '3', 2)] + \
[('5', '3', 3)] + \
[('5', '3', 4)] + \
[('5', '3', 5)] + \
[('5', '3', 6)] + \
[('5', '3', 7)] + \
[('5', '3', 8)] + \
[('5', '3', 9)] + \
[('5', '3', 10)] + \
[('5', '3', 11)] + \
[('5', '3', 20)] + \
[('5', '3', 12)] + \
[('5', '3', 13)] + \
[('5', '3', 21)] + \
[('5', '3', 14)] + \
[('5', '6', 1)] + \
[('5', '6', 2)] + \
[('5', '6', 3)] + \
[('5', '6', 4)] + \
[('5', '6', 5)] + \
[('5', '6', 6)] + \
[('5', '6', 7)] + \
[('5', '6', 8)] + \
[('5', '6', 9)] + \
[('5', '6', 10)] + \
[('5', '6', 20)] + \
[('5', '6', 12)] + \
[('5', '6', 13)] + \
[('5', '6', 21)] + \
[('5', '6', 14)] + \
[('5', '1', 1)] + \
[('5', '1', 2)] + \
[('5', '1', 3)] + \
[('5', '1', 4)] + \
[('5', '1', 5)] + \
[('5', '1', 6)] + \
[('5', '1', 7)] + \
[('5', '1', 8)] + \
[('5', '1', 9)] + \
[('5', '1', 10)] + \
[('5', '1', 11)] + \
[('5', '1', 20)] + \
[('5', '1', 12)] + \
[('5', '1', 13)] + \
[('5', '1', 21)] + \
[('5', '1', 14)] + \
[('6', '7', 1)] + \
[('6', '7', 2)] + \
[('6', '7', 3)] + \
[('6', '7', 17)] + \
[('6', '7', 4)] + \
[('6', '7', 5)] + \
[('6', '7', 6)] + \
[('6', '7', 7)] + \
[('6', '7', 8)] + \
[('6', '7', 9)] + \
[('6', '7', 10)] + \
[('6', '7', 11)] + \
[('6', '7', 12)] + \
[('6', '7', 13)] + \
[('6', '7', 14)] + \
[('6', '7', 15)] + \
[('6', '7', 16)] + \
[('6', '3', 1)] + \
[('6', '3', 2)] + \
[('6', '3', 3)] + \
[('6', '3', 4)] + \
[('6', '3', 5)] + \
[('6', '3', 6)] + \
[('6', '3', 7)] + \
[('6', '3', 8)] + \
[('6', '3', 9)] + \
[('6', '3', 10)] + \
[('6', '3', 11)] + \
[('6', '3', 12)] + \
[('6', '3', 13)] + \
[('6', '3', 14)] + \
[('6', '3', 15)] + \
[('6', '3', 16)] + \
[('6', '6', 1)] + \
[('6', '6', 2)] + \
[('6', '6', 3)] + \
[('6', '6', 4)] + \
[('6', '6', 5)] + \
[('6', '6', 6)] + \
[('6', '6', 7)] + \
[('6', '6', 8)] + \
[('6', '6', 9)] + \
[('6', '6', 10)] + \
[('6', '6', 11)] + \
[('6', '6', 12)] + \
[('6', '6', 13)] + \
[('6', '6', 14)] + \
[('6', '6', 15)] + \
[('6', '6', 16)] + \
[('6', '8', 1)] + \
[('6', '8', 22)] + \
[('6', '8', 2)] + \
[('6', '8', 3)] + \
[('6', '8', 4)] + \
[('6', '8', 5)] + \
[('6', '8', 6)] + \
[('6', '8', 7)] + \
[('6', '8', 8)] + \
[('6', '8', 9)] + \
[('6', '8', 10)] + \
[('6', '8', 11)] + \
[('6', '8', 12)] + \
[('6', '8', 13)] + \
[('6', '8', 14)] + \
[('6', '8', 15)] + \
[('6', '8', 16)] + \
[('6', '1', 1)] + \
[('6', '1', 2)] + \
[('6', '1', 3)] + \
[('6', '1', 4)] + \
[('6', '1', 5)] + \
[('6', '1', 6)] + \
[('6', '1', 7)] + \
[('6', '1', 8)] + \
[('6', '1', 9)] + \
[('6', '1', 10)] + \
[('6', '1', 11)] + \
[('6', '1', 12)] + \
[('6', '1', 13)] + \
[('6', '1', 14)] + \
[('6', '1', 15)] + \
[('6', '1', 16)] + \
[('6', '2', 1)] + \
[('6', '2', 2)] + \
[('6', '2', 3)] + \
[('6', '2', 4)] + \
[('6', '2', 5)] + \
[('6', '2', 6)] + \
[('6', '2', 7)] + \
[('6', '2', 8)] + \
[('6', '2', 9)] + \
[('6', '2', 10)] + \
[('6', '2', 11)] + \
[('6', '2', 20)] + \
[('6', '2', 12)] + \
[('6', '2', 13)] + \
[('6', '2', 21)] + \
[('6', '2', 14)] + \
[('6', '2', 15)] + \
[('6', '2', 16)] + \
[('7', '7', 1)] + \
[('7', '7', 22)] + \
[('7', '7', 2)] + \
[('7', '7', 3)] + \
[('7', '7', 4)] + \
[('7', '7', 5)] + \
[('7', '7', 6)] + \
[('7', '7', 7)] + \
[('7', '7', 8)] + \
[('7', '7', 9)] + \
[('7', '7', 10)] + \
[('7', '7', 11)] + \
[('7', '7', 12)] + \
[('7', '7', 13)] + \
[('7', '7', 14)] + \
[('7', '3', 1)] + \
[('7', '3', 22)] + \
[('7', '3', 2)] + \
[('7', '3', 3)] + \
[('7', '3', 4)] + \
[('7', '3', 5)] + \
[('7', '3', 6)] + \
[('7', '3', 7)] + \
[('7', '3', 8)] + \
[('7', '3', 9)] + \
[('7', '3', 10)] + \
[('7', '3', 11)] + \
[('7', '3', 12)] + \
[('7', '3', 13)] + \
[('7', '3', 14)] + \
[('7', '3', 15)] + \
[('7', '3', 16)] + \
[('7', '8', 1)] + \
[('7', '8', 22)] + \
[('7', '8', 2)] + \
[('7', '8', 3)] + \
[('7', '8', 4)] + \
[('7', '8', 5)] + \
[('7', '8', 6)] + \
[('7', '8', 7)] + \
[('7', '8', 8)] + \
[('7', '8', 9)] + \
[('7', '8', 10)] + \
[('7', '8', 11)] + \
[('7', '8', 12)] + \
[('7', '8', 13)] + \
[('7', '8', 14)] + \
[('7', '8', 15)] + \
[('7', '1', 1)] + \
[('7', '1', 2)] + \
[('7', '1', 3)] + \
[('7', '1', 4)] + \
[('7', '1', 5)] + \
[('7', '1', 6)] + \
[('7', '1', 7)] + \
[('7', '1', 8)] + \
[('7', '1', 9)] + \
[('7', '1', 10)] + \
[('7', '1', 11)] + \
[('7', '1', 20)] + \
[('7', '1', 12)] + \
[('7', '1', 13)] + \
[('7', '1', 21)] + \
[('7', '1', 14)] + \
[('7', '1', 15)] + \
[('7', '1', 16)] + \
[('7', '9', 1)] + \
[('7', '9', 2)] + \
[('7', '9', 3)] + \
[('7', '9', 4)] + \
[('7', '9', 5)] + \
[('7', '9', 6)] + \
[('7', '9', 7)] + \
[('7', '9', 8)] + \
[('7', '9', 9)] + \
[('7', '9', 10)] + \
[('7', '9', 11)] + \
[('7', '9', 12)] + \
[('7', '9', 13)] + \
[('7', '9', 14)] + \
[('7', '9', 15)] + \
[('7', '9', 16)] + \
[('7', '10', 1)] + \
[('7', '10', 2)] + \
[('7', '10', 3)] + \
[('7', '10', 4)] + \
[('7', '10', 5)] + \
[('7', '10', 6)] + \
[('7', '10', 7)] + \
[('7', '10', 8)] + \
[('7', '10', 9)] + \
[('7', '10', 10)] + \
[('7', '10', 11)] + \
[('7', '10', 12)] + \
[('7', '10', 13)] + \
[('7', '10', 14)] + \
[('7', '10', 15)] + \
[('7', '10', 16)] + \
[('8', '1', 1)] + \
[('8', '1', 2)] + \
[('8', '1', 3)] + \
[('8', '1', 4)] + \
[('8', '1', 5)] + \
[('8', '1', 6)] + \
[('8', '1', 7)] + \
[('8', '1', 8)] + \
[('8', '1', 9)] + \
[('8', '1', 10)] + \
[('8', '1', 11)] + \
[('8', '1', 12)] + \
[('8', '1', 13)] + \
[('8', '1', 14)] + \
[('8', '1', 15)] + \
[('8', '1', 16)] + \
[('8', '2', 1)] + \
[('8', '2', 2)] + \
[('8', '2', 3)] + \
[('8', '2', 17)] + \
[('8', '2', 4)] + \
[('8', '2', 5)] + \
[('8', '2', 6)] + \
[('8', '2', 7)] + \
[('8', '2', 8)] + \
[('8', '2', 9)] + \
[('8', '2', 10)] + \
[('8', '2', 11)] + \
[('8', '2', 12)] + \
[('8', '2', 13)] + \
[('8', '2', 14)] + \
[('8', '2', 15)] + \
[('8', '2', 16)] + \
[('9', '3', 1)] + \
[('9', '3', 2)] + \
[('9', '3', 3)] + \
[('9', '3', 4)] + \
[('9', '3', 5)] + \
[('9', '3', 6)] + \
[('9', '3', 7)] + \
[('9', '3', 8)] + \
[('9', '3', 9)] + \
[('9', '3', 10)] + \
[('9', '3', 11)] + \
[('9', '3', 12)] + \
[('9', '3', 13)] + \
[('9', '3', 14)] + \
[('9', '3', 15)] + \
[('9', '3', 16)] + \
[('10', '1', 1)] + \
[('10', '1', 2)] + \
[('10', '1', 3)] + \
[('10', '1', 4)] + \
[('10', '1', 5)] + \
[('10', '1', 6)] + \
[('10', '1', 7)] + \
[('10', '1', 8)] + \
[('10', '1', 9)] + \
[('10', '1', 10)] + \
[('10', '1', 11)] + \
[('10', '1', 12)] + \
[('10', '1', 13)] + \
[('10', '1', 14)] + \
[('10', '1', 15)] + \
[('10', '1', 16)] + \
[('12', '2', 1)] + \
[('12', '2', 2)] + \
[('12', '2', 3)] + \
[('12', '2', 4)] + \
[('12', '2', 5)] + \
[('12', '2', 6)] + \
[('12', '2', 7)] + \
[('12', '2', 8)] + \
[('12', '2', 9)] + \
[('12', '2', 10)] + \
[('12', '2', 11)] + \
[('12', '2', 12)] + \
[('12', '2', 13)] + \
[('12', '2', 14)] + \
[('12', '2', 15)] + \
[('12', '2', 16)] + \
[('24', '7', 1)] + \
[('24', '7', 2)] + \
[('24', '7', 3)] + \
[('24', '7', 4)] + \
[('24', '7', 5)] + \
[('24', '7', 6)] + \
[('24', '7', 7)] + \
[('24', '7', 8)] + \
[('24', '7', 9)] + \
[('24', '7', 10)] + \
[('24', '7', 11)] + \
[('24', '7', 12)] + \
[('24', '7', 13)] + \
[('24', '7', 21)] + \
[('24', '7', 14)] + \
[('24', '7', 15)] + \
[('24', '7', 16)] + \
[('24', '5', 1)] + \
[('24', '5', 2)] + \
[('24', '5', 3)] + \
[('24', '5', 4)] + \
[('24', '5', 5)] + \
[('24', '5', 6)] + \
[('24', '5', 7)] + \
[('24', '5', 8)] + \
[('24', '5', 9)] + \
[('24', '5', 10)] + \
[('24', '5', 11)] + \
[('24', '5', 12)] + \
[('24', '5', 13)] + \
[('24', '5', 14)] + \
[('24', '5', 15)] + \
[('24', '5', 16)] + \
[('24', '11', 1)] + \
[('24', '11', 2)] + \
[('24', '11', 3)] + \
[('24', '11', 4)] + \
[('24', '11', 5)] + \
[('24', '11', 6)] + \
[('24', '11', 7)] + \
[('24', '11', 8)] + \
[('24', '11', 9)] + \
[('24', '11', 10)] + \
[('24', '11', 11)] + \
[('24', '11', 12)] + \
[('24', '11', 13)] + \
[('24', '11', 14)] + \
[('24', '11', 15)] + \
[('24', '11', 16)] + \
[('24', '12', 1)] + \
[('24', '12', 2)] + \
[('24', '12', 3)] + \
[('24', '12', 4)] + \
[('24', '12', 5)] + \
[('24', '12', 6)] + \
[('24', '12', 7)] + \
[('24', '12', 8)] + \
[('24', '12', 9)] + \
[('24', '12', 10)] + \
[('24', '12', 11)] + \
[('24', '12', 12)] + \
[('24', '12', 13)] + \
[('24', '12', 14)] + \
[('24', '12', 15)] + \
[('24', '12', 16)] + \
[('24', '3', 1)] + \
[('24', '3', 2)] + \
[('24', '3', 3)] + \
[('24', '3', 4)] + \
[('24', '3', 5)] + \
[('24', '3', 6)] + \
[('24', '3', 7)] + \
[('24', '3', 8)] + \
[('24', '3', 9)] + \
[('24', '3', 10)] + \
[('24', '3', 11)] + \
[('24', '3', 12)] + \
[('24', '3', 13)] + \
[('24', '3', 21)] + \
[('24', '3', 14)] + \
[('24', '3', 15)] + \
[('24', '3', 16)] + \
[('24', '6', 1)] + \
[('24', '6', 2)] + \
[('24', '6', 3)] + \
[('24', '6', 4)] + \
[('24', '6', 5)] + \
[('24', '6', 6)] + \
[('24', '6', 7)] + \
[('24', '6', 8)] + \
[('24', '6', 9)] + \
[('24', '6', 10)] + \
[('24', '6', 11)] + \
[('24', '6', 12)] + \
[('24', '6', 13)] + \
[('24', '6', 14)] + \
[('24', '6', 15)] + \
[('24', '6', 16)] + \
[('24', '8', 1)] + \
[('24', '8', 22)] + \
[('24', '8', 2)] + \
[('24', '8', 3)] + \
[('24', '8', 4)] + \
[('24', '8', 5)] + \
[('24', '8', 6)] + \
[('24', '8', 7)] + \
[('24', '8', 8)] + \
[('24', '8', 9)] + \
[('24', '8', 10)] + \
[('24', '8', 11)] + \
[('24', '8', 12)] + \
[('24', '8', 13)] + \
[('24', '8', 14)] + \
[('24', '8', 15)] + \
[('24', '13', 1)] + \
[('24', '13', 2)] + \
[('24', '13', 3)] + \
[('24', '13', 4)] + \
[('24', '13', 5)] + \
[('24', '13', 6)] + \
[('24', '13', 7)] + \
[('24', '13', 8)] + \
[('24', '13', 9)] + \
[('24', '13', 10)] + \
[('24', '13', 11)] + \
[('24', '13', 12)] + \
[('24', '13', 13)] + \
[('24', '13', 14)] + \
[('24', '13', 15)] + \
[('24', '13', 16)] + \
[('24', '1', 1)] + \
[('24', '1', 2)] + \
[('24', '1', 3)] + \
[('24', '1', 4)] + \
[('24', '1', 5)] + \
[('24', '1', 6)] + \
[('24', '1', 7)] + \
[('24', '1', 8)] + \
[('24', '1', 9)] + \
[('24', '1', 10)] + \
[('24', '1', 11)] + \
[('24', '1', 12)] + \
[('24', '1', 13)] + \
[('24', '1', 14)] + \
[('24', '1', 15)] + \
[('24', '1', 16)] + \
[('24', '14', 1)] + \
[('24', '14', 2)] + \
[('24', '14', 3)] + \
[('24', '14', 4)] + \
[('24', '14', 5)] + \
[('24', '14', 6)] + \
[('24', '14', 7)] + \
[('24', '14', 8)] + \
[('24', '14', 9)] + \
[('24', '14', 10)] + \
[('24', '14', 11)] + \
[('24', '14', 12)] + \
[('24', '14', 13)] + \
[('24', '14', 14)] + \
[('24', '14', 15)] + \
[('24', '14', 16)] + \
[('24', '15', 1)] + \
[('24', '15', 2)] + \
[('24', '15', 3)] + \
[('24', '15', 4)] + \
[('24', '15', 5)] + \
[('24', '15', 6)] + \
[('24', '15', 7)] + \
[('24', '15', 8)] + \
[('24', '15', 9)] + \
[('24', '15', 10)] + \
[('24', '15', 11)] + \
[('24', '15', 12)] + \
[('24', '15', 13)] + \
[('24', '15', 14)] + \
[('24', '15', 15)] + \
[('24', '15', 16)] + \
[('24', '16', 1)] + \
[('24', '16', 2)] + \
[('24', '16', 3)] + \
[('24', '16', 4)] + \
[('24', '16', 5)] + \
[('24', '16', 6)] + \
[('24', '16', 7)] + \
[('24', '16', 8)] + \
[('24', '16', 9)] + \
[('24', '16', 10)] + \
[('24', '16', 11)] + \
[('24', '16', 12)] + \
[('24', '16', 13)] + \
[('24', '16', 14)] + \
[('24', '16', 15)] + \
[('24', '16', 16)] + \
[('24', '17', 1)] + \
[('24', '17', 2)] + \
[('24', '17', 3)] + \
[('24', '17', 4)] + \
[('24', '17', 5)] + \
[('24', '17', 6)] + \
[('24', '17', 7)] + \
[('24', '17', 8)] + \
[('24', '17', 9)] + \
[('24', '17', 10)] + \
[('24', '17', 11)] + \
[('24', '17', 12)] + \
[('24', '17', 13)] + \
[('24', '17', 14)] + \
[('24', '17', 15)] + \
[('24', '17', 16)] + \
[('24', '9', 1)] + \
[('24', '9', 2)] + \
[('24', '9', 3)] + \
[('24', '9', 4)] + \
[('24', '9', 5)] + \
[('24', '9', 6)] + \
[('24', '9', 7)] + \
[('24', '9', 8)] + \
[('24', '9', 9)] + \
[('24', '9', 10)] + \
[('24', '9', 11)] + \
[('24', '9', 12)] + \
[('24', '9', 13)] + \
[('24', '9', 14)] + \
[('24', '9', 15)] + \
[('24', '9', 16)] + \
[('24', '10', 1)] + \
[('24', '10', 2)] + \
[('24', '10', 3)] + \
[('24', '10', 4)] + \
[('24', '10', 5)] + \
[('24', '10', 6)] + \
[('24', '10', 7)] + \
[('24', '10', 8)] + \
[('24', '10', 9)] + \
[('24', '10', 10)] + \
[('24', '10', 11)] + \
[('24', '10', 12)] + \
[('24', '10', 13)] + \
[('24', '10', 14)] + \
[('24', '10', 15)] + \
[('24', '10', 16)] + \
[('24', '2', 1)] + \
[('24', '2', 2)] + \
[('24', '2', 3)] + \
[('24', '2', 4)] + \
[('24', '2', 5)] + \
[('24', '2', 6)] + \
[('24', '2', 7)] + \
[('24', '2', 8)] + \
[('24', '2', 9)] + \
[('24', '2', 10)] + \
[('24', '2', 11)] + \
[('24', '2', 12)] + \
[('24', '2', 13)] + \
[('24', '2', 14)] + \
[('24', '2', 15)] + \
[('24', '2', 16)] + \
[('27', '1', 1)] + \
[('27', '1', 2)] + \
[('27', '1', 3)] + \
[('27', '1', 4)] + \
[('27', '1', 5)] + \
[('27', '1', 6)] + \
[('27', '1', 7)] + \
[('27', '1', 8)] + \
[('27', '1', 9)] + \
[('27', '1', 10)] + \
[('27', '1', 11)] + \
[('27', '1', 12)] + \
[('27', '1', 13)] + \
[('27', '1', 14)] + \
[('27', '1', 15)] + \
[('27', '1', 16)] + \
[('27', '2', 1)] + \
[('27', '2', 2)] + \
[('27', '2', 3)] + \
[('27', '2', 17)] + \
[('27', '2', 4)] + \
[('27', '2', 5)] + \
[('27', '2', 6)] + \
[('27', '2', 7)] + \
[('27', '2', 8)] + \
[('27', '2', 9)] + \
[('27', '2', 10)] + \
[('27', '2', 11)] + \
[('27', '2', 12)] + \
[('27', '2', 13)] + \
[('27', '2', 14)] + \
[('27', '2', 15)] + \
[('27', '2', 16)] + \
[('28', '2', 2)] + \
[('28', '2', 3)] + \
[('28', '2', 17)] + \
[('28', '2', 4)] + \
[('28', '2', 5)] + \
[('28', '2', 6)] + \
[('28', '2', 8)] + \
[('28', '2', 9)] + \
[('28', '2', 10)] + \
[('28', '2', 11)] + \
[('28', '2', 13)] + \
[('28', '2', 14)] + \
[('29', '3', 1)] + \
[('29', '3', 2)] + \
[('29', '3', 3)] + \
[('29', '3', 4)] + \
[('29', '3', 5)] + \
[('29', '3', 6)] + \
[('29', '3', 7)] + \
[('29', '3', 8)] + \
[('29', '3', 9)] + \
[('29', '3', 10)] + \
[('29', '3', 11)] + \
[('29', '3', 12)] + \
[('29', '3', 13)] + \
[('29', '3', 14)] + \
[('29', '3', 15)] + \
[('29', '3', 16)] + \
[('30', '1', 1)] + \
[('30', '1', 2)] + \
[('30', '1', 3)] + \
[('30', '1', 4)] + \
[('30', '1', 5)] + \
[('30', '1', 6)] + \
[('30', '1', 7)] + \
[('30', '1', 8)] + \
[('30', '1', 9)] + \
[('30', '1', 10)] + \
[('30', '1', 11)] + \
[('30', '1', 12)] + \
[('30', '1', 13)] + \
[('30', '1', 14)] + \
[('30', '1', 15)] + \
[('30', '1', 16)] + \
[('31', '1', 1)] + \
[('31', '1', 2)] + \
[('31', '1', 3)] + \
[('31', '1', 17)] + \
[('31', '1', 4)] + \
[('31', '1', 5)] + \
[('31', '1', 6)] + \
[('31', '1', 7)] + \
[('31', '1', 8)] + \
[('31', '1', 9)] + \
[('31', '1', 10)] + \
[('31', '1', 11)] + \
[('31', '1', 12)] + \
[('31', '1', 13)] + \
[('31', '1', 14)] + \
[('31', '1', 15)] + \
[('31', '1', 16)] + \
[('32', '5', 24)] + \
[('32', '5', 1)] + \
[('32', '5', 2)] + \
[('32', '5', 3)] + \
[('32', '5', 4)] + \
[('32', '5', 5)] + \
[('32', '5', 6)] + \
[('32', '5', 8)] + \
[('32', '5', 9)] + \
[('32', '5', 10)] + \
[('32', '5', 11)] + \
[('32', '5', 12)] + \
[('32', '5', 13)] + \
[('32', '3', 24)] + \
[('32', '3', 1)] + \
[('32', '3', 2)] + \
[('32', '3', 3)] + \
[('32', '3', 4)] + \
[('32', '3', 5)] + \
[('32', '3', 6)] + \
[('32', '3', 8)] + \
[('32', '3', 9)] + \
[('32', '3', 10)] + \
[('32', '3', 11)] + \
[('32', '3', 12)] + \
[('32', '3', 13)] + \
[('32', '6', 24)] + \
[('32', '6', 1)] + \
[('32', '6', 2)] + \
[('32', '6', 3)] + \
[('32', '6', 4)] + \
[('32', '6', 5)] + \
[('32', '6', 6)] + \
[('32', '6', 8)] + \
[('32', '6', 9)] + \
[('32', '6', 10)] + \
[('32', '6', 11)] + \
[('32', '6', 12)] + \
[('32', '6', 13)] + \
[('32', '8', 24)] + \
[('32', '8', 1)] + \
[('32', '8', 2)] + \
[('32', '8', 3)] + \
[('32', '8', 4)] + \
[('32', '8', 5)] + \
[('32', '8', 6)] + \
[('32', '8', 8)] + \
[('32', '8', 9)] + \
[('32', '8', 10)] + \
[('32', '8', 11)] + \
[('32', '8', 12)] + \
[('32', '8', 13)] + \
[('32', '1', 24)] + \
[('32', '1', 1)] + \
[('32', '1', 2)] + \
[('32', '1', 3)] + \
[('32', '1', 4)] + \
[('32', '1', 5)] + \
[('32', '1', 6)] + \
[('32', '1', 8)] + \
[('32', '1', 9)] + \
[('32', '1', 10)] + \
[('32', '1', 11)] + \
[('32', '1', 12)] + \
[('32', '1', 13)] + \
[('32', '2', 24)] + \
[('32', '2', 1)] + \
[('32', '2', 2)] + \
[('32', '2', 3)] + \
[('32', '2', 4)] + \
[('32', '2', 5)] + \
[('32', '2', 6)] + \
[('32', '2', 8)] + \
[('32', '2', 9)] + \
[('32', '2', 10)] + \
[('32', '2', 11)] + \
[('32', '2', 12)] + \
[('32', '2', 13)] + \
[('33', '5', 24)] + \
[('33', '5', 1)] + \
[('33', '5', 2)] + \
[('33', '5', 3)] + \
[('33', '5', 4)] + \
[('33', '5', 5)] + \
[('33', '5', 6)] + \
[('33', '5', 8)] + \
[('33', '5', 9)] + \
[('33', '5', 10)] + \
[('33', '5', 11)] + \
[('33', '5', 12)] + \
[('33', '5', 13)] + \
[('33', '3', 24)] + \
[('33', '3', 1)] + \
[('33', '3', 2)] + \
[('33', '3', 3)] + \
[('33', '3', 4)] + \
[('33', '3', 5)] + \
[('33', '3', 6)] + \
[('33', '3', 8)] + \
[('33', '3', 9)] + \
[('33', '3', 10)] + \
[('33', '3', 11)] + \
[('33', '3', 12)] + \
[('33', '3', 13)] + \
[('33', '6', 24)] + \
[('33', '6', 1)] + \
[('33', '6', 2)] + \
[('33', '6', 3)] + \
[('33', '6', 4)] + \
[('33', '6', 5)] + \
[('33', '6', 6)] + \
[('33', '6', 8)] + \
[('33', '6', 9)] + \
[('33', '6', 10)] + \
[('33', '6', 11)] + \
[('33', '6', 12)] + \
[('33', '6', 13)] + \
[('33', '8', 24)] + \
[('33', '8', 1)] + \
[('33', '8', 2)] + \
[('33', '8', 3)] + \
[('33', '8', 4)] + \
[('33', '8', 5)] + \
[('33', '8', 6)] + \
[('33', '8', 8)] + \
[('33', '8', 9)] + \
[('33', '8', 10)] + \
[('33', '8', 11)] + \
[('33', '8', 12)] + \
[('33', '8', 13)] + \
[('33', '1', 24)] + \
[('33', '1', 1)] + \
[('33', '1', 2)] + \
[('33', '1', 3)] + \
[('33', '1', 4)] + \
[('33', '1', 5)] + \
[('33', '1', 6)] + \
[('33', '1', 8)] + \
[('33', '1', 9)] + \
[('33', '1', 10)] + \
[('33', '1', 11)] + \
[('33', '1', 12)] + \
[('33', '1', 13)] + \
[('33', '2', 24)] + \
[('33', '2', 1)] + \
[('33', '2', 2)] + \
[('33', '2', 3)] + \
[('33', '2', 4)] + \
[('33', '2', 5)] + \
[('33', '2', 6)] + \
[('33', '2', 8)] + \
[('33', '2', 9)] + \
[('33', '2', 10)] + \
[('33', '2', 11)] + \
[('33', '2', 12)] + \
[('33', '2', 13)] + \
[('34', '1', 1)] + \
[('34', '1', 2)] + \
[('34', '1', 3)] + \
[('34', '1', 17)] + \
[('34', '1', 4)] + \
[('34', '1', 5)] + \
[('34', '1', 6)] + \
[('34', '1', 7)] + \
[('34', '1', 8)] + \
[('34', '1', 9)] + \
[('34', '1', 10)] + \
[('34', '1', 11)] + \
[('34', '1', 12)] + \
[('34', '1', 13)] + \
[('34', '1', 14)] + \
[('34', '1', 15)] + \
[('34', '1', 16)] + \
[('35', '2', 2)] + \
[('35', '2', 3)] + \
[('35', '2', 17)] + \
[('35', '2', 4)] + \
[('35', '2', 5)] + \
[('35', '2', 6)] + \
[('35', '2', 8)] + \
[('35', '2', 9)] + \
[('35', '2', 10)] + \
[('35', '2', 11)] + \
[('35', '2', 13)] + \
[('35', '2', 14)] + \
[('36', '2', 2)] + \
[('36', '2', 3)] + \
[('36', '2', 17)] + \
[('36', '2', 4)] + \
[('36', '2', 5)] + \
[('36', '2', 6)] + \
[('36', '2', 8)] + \
[('36', '2', 9)] + \
[('36', '2', 10)] + \
[('36', '2', 11)] + \
[('36', '2', 13)] + \
[('36', '2', 14)] + \
[('38', '3', 1)] + \
[('38', '3', 2)] + \
[('38', '3', 3)] + \
[('38', '3', 4)] + \
[('38', '3', 5)] + \
[('38', '3', 6)] + \
[('38', '3', 7)] + \
[('38', '3', 8)] + \
[('38', '3', 9)] + \
[('38', '3', 10)] + \
[('38', '3', 11)] + \
[('38', '3', 12)] + \
[('38', '3', 13)] + \
[('38', '3', 21)] + \
[('38', '3', 14)] + \
[('38', '3', 15)] + \
[('38', '3', 16)] + \
[('38', '6', 1)] + \
[('38', '6', 2)] + \
[('38', '6', 3)] + \
[('38', '6', 4)] + \
[('38', '6', 5)] + \
[('38', '6', 6)] + \
[('38', '6', 7)] + \
[('38', '6', 8)] + \
[('38', '6', 9)] + \
[('38', '6', 10)] + \
[('38', '6', 11)] + \
[('38', '6', 12)] + \
[('38', '6', 13)] + \
[('38', '6', 14)] + \
[('38', '6', 15)] + \
[('38', '6', 16)] + \
[('38', '1', 1)] + \
[('38', '1', 2)] + \
[('38', '1', 3)] + \
[('38', '1', 4)] + \
[('38', '1', 5)] + \
[('38', '1', 6)] + \
[('38', '1', 7)] + \
[('38', '1', 8)] + \
[('38', '1', 9)] + \
[('38', '1', 10)] + \
[('38', '1', 11)] + \
[('38', '1', 12)] + \
[('38', '1', 13)] + \
[('38', '1', 14)] + \
[('38', '1', 15)] + \
[('38', '1', 16)] + \
[('52', '1', 1)] + \
[('52', '1', 2)] + \
[('52', '1', 3)] + \
[('52', '1', 4)] + \
[('52', '1', 5)] + \
[('52', '1', 6)] + \
[('52', '1', 7)] + \
[('52', '1', 8)] + \
[('52', '1', 9)] + \
[('52', '1', 10)] + \
[('52', '1', 11)] + \
[('52', '1', 12)] + \
[('52', '1', 13)] + \
[('52', '1', 14)] + \
[('52', '1', 15)] + \
[('52', '1', 16)] + \
[('52', '2', 1)] + \
[('52', '2', 2)] + \
[('52', '2', 3)] + \
[('52', '2', 17)] + \
[('52', '2', 4)] + \
[('52', '2', 5)] + \
[('52', '2', 6)] + \
[('52', '2', 7)] + \
[('52', '2', 8)] + \
[('52', '2', 9)] + \
[('52', '2', 10)] + \
[('52', '2', 11)] + \
[('52', '2', 12)] + \
[('52', '2', 13)] + \
[('52', '2', 14)] + \
[('52', '2', 15)] + \
[('52', '2', 16)] + \
[('53', '1', 3)] + \
[('54', '3', 1)] + \
[('54', '3', 2)] + \
[('54', '3', 3)] + \
[('54', '3', 4)] + \
[('54', '3', 5)] + \
[('54', '3', 6)] + \
[('54', '3', 7)] + \
[('54', '3', 8)] + \
[('54', '3', 9)] + \
[('54', '3', 10)] + \
[('54', '3', 11)] + \
[('54', '3', 12)] + \
[('54', '3', 13)] + \
[('54', '3', 14)] + \
[('54', '3', 15)] + \
[('54', '3', 16)] + \
[('55', '1', 1)] + \
[('55', '1', 2)] + \
[('55', '1', 3)] + \
[('55', '1', 4)] + \
[('55', '1', 5)] + \
[('55', '1', 6)] + \
[('55', '1', 7)] + \
[('55', '1', 8)] + \
[('55', '1', 9)] + \
[('55', '1', 10)] + \
[('55', '1', 11)] + \
[('55', '1', 12)] + \
[('55', '1', 13)] + \
[('55', '1', 14)] + \
[('55', '1', 15)] + \
[('55', '1', 16)] + \
[('56', '1', 1)] + \
[('56', '1', 2)] + \
[('56', '1', 3)] + \
[('56', '1', 4)] + \
[('56', '1', 5)] + \
[('56', '1', 6)] + \
[('56', '1', 7)] + \
[('56', '1', 8)] + \
[('56', '1', 9)] + \
[('56', '1', 10)] + \
[('56', '1', 11)] + \
[('56', '1', 12)] + \
[('56', '1', 13)] + \
[('56', '1', 14)] + \
[('56', '1', 15)] + \
[('56', '1', 16)] + \
[('57', '7', 1)] + \
[('57', '7', 22)] + \
[('57', '7', 2)] + \
[('57', '7', 3)] + \
[('57', '7', 4)] + \
[('57', '7', 5)] + \
[('57', '7', 6)] + \
[('57', '7', 7)] + \
[('57', '7', 8)] + \
[('57', '7', 9)] + \
[('57', '7', 10)] + \
[('57', '7', 11)] + \
[('57', '7', 12)] + \
[('57', '7', 13)] + \
[('57', '7', 14)] + \
[('57', '7', 15)] + \
[('57', '7', 16)] + \
[('58', '11', 1)] + \
[('58', '11', 22)] + \
[('58', '11', 2)] + \
[('58', '11', 3)] + \
[('58', '11', 4)] + \
[('58', '11', 5)] + \
[('58', '11', 6)] + \
[('58', '11', 7)] + \
[('58', '11', 8)] + \
[('58', '11', 9)] + \
[('58', '11', 10)] + \
[('58', '11', 11)] + \
[('58', '11', 12)] + \
[('58', '11', 13)] + \
[('58', '11', 14)] + \
[('58', '11', 15)] + \
[('58', '11', 16)] + \
[('59', '1', 3)] + \
[('59', '1', 17)] + \
[('59', '1', 4)] + \
[('59', '1', 5)] + \
[('59', '1', 6)] + \
[('59', '1', 8)] + \
[('59', '1', 9)] + \
[('59', '1', 10)] + \
[('60', '1', 3)] + \
[('60', '1', 17)] + \
[('60', '1', 4)] + \
[('60', '1', 5)] + \
[('60', '1', 6)] + \
[('60', '1', 8)] + \
[('60', '1', 10)] + \
[('61', '3', 1)] + \
[('61', '3', 2)] + \
[('61', '3', 3)] + \
[('61', '3', 4)] + \
[('61', '3', 5)] + \
[('61', '3', 6)] + \
[('61', '3', 7)] + \
[('61', '3', 8)] + \
[('61', '3', 9)] + \
[('61', '3', 10)] + \
[('61', '3', 11)] + \
[('61', '3', 12)] + \
[('61', '3', 13)] + \
[('61', '3', 14)] + \
[('61', '3', 15)] + \
[('61', '3', 16)] + \
[('62', '3', 1)] + \
[('62', '3', 2)] + \
[('62', '3', 3)] + \
[('62', '3', 4)] + \
[('62', '3', 5)] + \
[('62', '3', 6)] + \
[('62', '3', 7)] + \
[('62', '3', 8)] + \
[('62', '3', 9)] + \
[('62', '3', 10)] + \
[('62', '3', 11)] + \
[('62', '3', 12)] + \
[('62', '3', 13)] + \
[('62', '3', 21)] + \
[('62', '3', 14)] + \
[('62', '3', 15)] + \
[('62', '3', 16)] + \
[('63', '3', 1)] + \
[('63', '3', 2)] + \
[('63', '3', 3)] + \
[('63', '3', 4)] + \
[('63', '3', 5)] + \
[('63', '3', 6)] + \
[('63', '3', 7)] + \
[('63', '3', 8)] + \
[('63', '3', 9)] + \
[('63', '3', 10)] + \
[('63', '3', 11)] + \
[('63', '3', 12)] + \
[('63', '3', 13)] + \
[('63', '3', 14)] + \
[('63', '3', 15)] + \
[('63', '3', 16)] + \
[('64', '2', 1)] + \
[('64', '2', 2)] + \
[('64', '2', 3)] + \
[('64', '2', 17)] + \
[('64', '2', 4)] + \
[('64', '2', 5)] + \
[('64', '2', 6)] + \
[('64', '2', 7)] + \
[('64', '2', 8)] + \
[('64', '2', 9)] + \
[('64', '2', 10)] + \
[('64', '2', 11)] + \
[('64', '2', 12)] + \
[('64', '2', 13)] + \
[('64', '2', 21)] + \
[('64', '2', 14)] + \
[('64', '2', 15)] + \
[('64', '2', 16)] + \
[('65', '13', 1)] + \
[('65', '13', 22)] + \
[('65', '13', 2)] + \
[('65', '13', 3)] + \
[('65', '13', 4)] + \
[('65', '13', 5)] + \
[('65', '13', 6)] + \
[('65', '13', 7)] + \
[('65', '13', 8)] + \
[('65', '13', 9)] + \
[('65', '13', 10)] + \
[('65', '13', 11)] + \
[('65', '13', 12)] + \
[('65', '13', 13)] + \
[('65', '13', 14)] + \
[('65', '13', 15)] + \
[('65', '13', 16)] + \
[('66', '18', 1)] + \
[('66', '18', 2)] + \
[('66', '18', 3)] + \
[('66', '18', 4)] + \
[('66', '18', 5)] + \
[('66', '18', 6)] + \
[('66', '18', 7)] + \
[('66', '18', 8)] + \
[('67', '19', 1)] + \
[('67', '19', 22)] + \
[('67', '19', 2)] + \
[('67', '19', 3)] + \
[('67', '19', 4)] + \
[('67', '19', 5)] + \
[('67', '19', 6)] + \
[('67', '19', 7)] + \
[('67', '19', 8)] + \
[('67', '19', 9)] + \
[('67', '19', 10)] + \
[('67', '19', 11)] + \
[('67', '19', 12)] + \
[('67', '19', 13)] + \
[('67', '19', 14)] + \
[('67', '19', 15)] + \
[('67', '19', 16)] + \
[('68', '2', 1)] + \
[('68', '2', 2)] + \
[('68', '2', 3)] + \
[('68', '2', 4)] + \
[('68', '2', 5)] + \
[('68', '2', 6)] + \
[('68', '2', 7)] + \
[('68', '2', 8)] + \
[('68', '2', 9)] + \
[('68', '2', 10)] + \
[('68', '2', 11)] + \
[('68', '2', 12)] + \
[('68', '2', 13)] + \
[('68', '2', 14)] + \
[('68', '2', 15)] + \
[('68', '2', 16)] + \
[('69', '2', 1)] + \
[('69', '2', 2)] + \
[('69', '2', 3)] + \
[('69', '2', 4)] + \
[('69', '2', 5)] + \
[('69', '2', 6)] + \
[('69', '2', 7)] + \
[('69', '2', 8)] + \
[('69', '2', 9)] + \
[('69', '2', 10)] + \
[('69', '2', 11)] + \
[('69', '2', 12)] + \
[('69', '2', 13)] + \
[('69', '2', 14)] + \
[('69', '2', 15)] + \
[('69', '2', 16)] + \
[('70', '15', 1)] + \
[('70', '15', 22)] + \
[('70', '15', 2)] + \
[('70', '15', 3)] + \
[('70', '15', 4)] + \
[('70', '15', 5)] + \
[('70', '15', 6)] + \
[('70', '15', 7)] + \
[('70', '15', 8)] + \
[('70', '15', 9)] + \
[('70', '15', 10)] + \
[('70', '15', 11)] + \
[('70', '15', 12)] + \
[('70', '15', 13)] + \
[('70', '15', 14)] + \
[('70', '15', 15)] + \
[('70', '15', 16)] + \
[('71', '6', 1)] + \
[('71', '6', 2)] + \
[('71', '6', 3)] + \
[('71', '6', 4)] + \
[('71', '6', 5)] + \
[('71', '6', 6)] + \
[('71', '6', 7)] + \
[('71', '6', 8)] + \
[('71', '6', 9)] + \
[('71', '6', 10)] + \
[('71', '6', 11)] + \
[('71', '6', 12)] + \
[('71', '6', 13)] + \
[('71', '6', 14)] + \
[('71', '6', 15)] + \
[('71', '6', 16)] + \
[('72', '6', 1)] + \
[('72', '6', 2)] + \
[('72', '6', 3)] + \
[('72', '6', 4)] + \
[('73', '5', 1)] + \
[('73', '5', 2)] + \
[('73', '5', 3)] + \
[('73', '5', 6)] + \
[('73', '5', 7)] + \
[('73', '5', 8)] + \
[('73', '5', 9)] + \
[('73', '5', 10)] + \
[('73', '5', 11)] + \
[('73', '5', 12)] + \
[('73', '5', 13)] + \
[('73', '5', 14)] + \
[('73', '5', 15)] + \
[('73', '5', 16)] + \
[('74', '1', 1)] + \
[('74', '1', 2)] + \
[('74', '1', 3)] + \
[('74', '1', 6)] + \
[('74', '1', 7)] + \
[('74', '1', 8)] + \
[('74', '1', 9)] + \
[('74', '1', 10)] + \
[('74', '1', 11)] + \
[('74', '1', 12)] + \
[('74', '1', 13)] + \
[('74', '1', 14)] + \
[('74', '1', 15)] + \
[('74', '1', 16)] + \
[('75', '1', 1)] + \
[('75', '1', 2)] + \
[('75', '1', 3)] + \
[('75', '1', 4)] + \
[('75', '1', 5)] + \
[('75', '1', 6)] + \
[('75', '1', 7)] + \
[('75', '1', 8)] + \
[('75', '1', 9)] + \
[('75', '1', 10)] + \
[('75', '1', 11)] + \
[('75', '1', 12)] + \
[('75', '1', 13)] + \
[('75', '1', 14)] + \
[('75', '1', 15)] + \
[('75', '1', 16)] + \
[('76', '1', 1)] + \
[('76', '1', 2)] + \
[('76', '1', 3)] + \
[('76', '1', 4)] + \
[('76', '1', 5)] + \
[('76', '1', 6)] + \
[('76', '1', 7)] + \
[('76', '1', 8)] + \
[('76', '1', 9)] + \
[('76', '1', 11)] + \
[('76', '1', 12)] + \
[('76', '1', 13)] + \
[('76', '1', 14)] + \
[('76', '1', 15)] + \
[('76', '1', 16)] + \
[('77', '1', 1)] + \
[('77', '1', 2)] + \
[('77', '1', 3)] + \
[('77', '1', 4)] + \
[('77', '1', 5)] + \
[('77', '1', 6)] + \
[('77', '1', 7)] + \
[('77', '1', 8)] + \
[('77', '1', 9)] + \
[('77', '1', 10)] + \
[('77', '1', 11)] + \
[('77', '1', 12)] + \
[('77', '1', 13)] + \
[('77', '1', 14)] + \
[('77', '1', 15)] + \
[('77', '1', 16)] + \
[('78', '20', 1)] + \
[('78', '20', 22)] + \
[('78', '20', 2)] + \
[('78', '20', 3)] + \
[('78', '20', 4)] + \
[('78', '20', 5)] + \
[('78', '20', 6)] + \
[('78', '20', 7)] + \
[('78', '20', 8)] + \
[('78', '20', 9)] + \
[('78', '20', 10)] + \
[('78', '20', 11)] + \
[('78', '20', 12)] + \
[('78', '20', 13)] + \
[('78', '20', 14)] + \
[('78', '20', 15)] + \
[('78', '20', 16)] + \
[('79', '1', 1)] + \
[('79', '1', 2)] + \
[('79', '1', 3)] + \
[('79', '1', 4)] + \
[('79', '1', 5)] + \
[('79', '1', 6)] + \
[('79', '1', 7)] + \
[('79', '1', 8)] + \
[('79', '1', 9)] + \
[('79', '1', 10)] + \
[('79', '1', 11)] + \
[('79', '1', 12)] + \
[('79', '1', 13)] + \
[('79', '1', 14)] + \
[('79', '1', 15)] + \
[('79', '1', 16)] + \
[('80', '1', 1)] + \
[('80', '1', 2)] + \
[('80', '1', 3)] + \
[('80', '1', 4)] + \
[('80', '1', 5)] + \
[('80', '1', 6)] + \
[('80', '1', 7)] + \
[('80', '1', 8)] + \
[('80', '1', 9)] + \
[('80', '1', 10)] + \
[('80', '1', 11)] + \
[('80', '1', 12)] + \
[('80', '1', 13)] + \
[('80', '1', 14)] + \
[('80', '1', 15)] + \
[('80', '1', 16)] + \
[('81', '1', 1)] + \
[('81', '1', 2)] + \
[('81', '1', 3)] + \
[('81', '1', 4)] + \
[('81', '1', 5)] + \
[('81', '1', 6)] + \
[('81', '1', 7)] + \
[('81', '1', 8)] + \
[('81', '1', 9)] + \
[('81', '1', 10)] + \
[('81', '1', 11)] + \
[('81', '1', 12)] + \
[('81', '1', 13)] + \
[('81', '1', 14)] + \
[('81', '1', 15)] + \
[('81', '1', 16)] + \
[('82', '1', 3)] + \
[('82', '1', 4)] + \
[('82', '1', 5)] + \
[('82', '1', 6)] + \
[('82', '1', 7)] + \
[('82', '1', 8)] + \
[('82', '1', 9)] + \
[('82', '1', 10)] + \
[('83', '1', 1)] + \
[('83', '1', 2)] + \
[('83', '1', 3)] + \
[('83', '1', 4)] + \
[('83', '1', 5)] + \
[('83', '1', 6)] + \
[('83', '1', 7)] + \
[('83', '1', 8)] + \
[('83', '1', 9)] + \
[('83', '1', 10)] + \
[('83', '1', 11)] + \
[('83', '1', 12)] + \
[('83', '1', 13)] + \
[('83', '1', 14)] + \
[('83', '1', 15)] + \
[('83', '1', 16)] + \
[('84', '1', 1)] + \
[('84', '1', 2)] + \
[('84', '1', 3)] + \
[('84', '1', 4)] + \
[('84', '1', 5)] + \
[('84', '1', 6)] + \
[('84', '1', 7)] + \
[('84', '1', 8)] + \
[('84', '1', 9)] + \
[('84', '1', 10)] + \
[('84', '1', 11)] + \
[('84', '1', 12)] + \
[('84', '1', 13)] + \
[('84', '1', 14)] + \
[('84', '1', 15)] + \
[('84', '1', 16)] + \
[('85', '1', 1)] + \
[('85', '1', 2)] + \
[('85', '1', 3)] + \
[('85', '1', 4)] + \
[('85', '1', 5)] + \
[('85', '1', 6)] + \
[('85', '1', 7)] + \
[('85', '1', 8)] + \
[('85', '1', 9)] + \
[('85', '1', 10)] + \
[('85', '1', 11)] + \
[('85', '1', 12)] + \
[('85', '1', 13)] + \
[('85', '1', 14)] + \
[('85', '1', 15)] + \
[('85', '1', 16)] + \
[('86', '1', 1)] + \
[('86', '1', 2)] + \
[('86', '1', 3)] + \
[('86', '1', 4)] + \
[('86', '1', 5)] + \
[('86', '1', 6)] + \
[('86', '1', 7)] + \
[('86', '1', 8)] + \
[('86', '1', 9)] + \
[('86', '1', 10)] + \
[('86', '1', 11)] + \
[('86', '1', 12)] + \
[('86', '1', 13)] + \
[('86', '1', 14)] + \
[('86', '1', 15)] + \
[('86', '1', 16)] + \
[('87', '15', 3)]
right_merge_df = pd.DataFrame(right_merge_tuples, columns=right_on)
for col in right_on:
right_merge_df[col] = right_merge_df[col].astype(column_types[col])
def create_sample_column_data(name, np_type, size):
if np_type in ('int32', 'int64'):
return np.full(size, 1, dtype=np_type)
elif np_type in ('float32', 'float64'):
return np.full(size, 1.0, dtype=np_type)
elif np_type == 'object':
return ['A'] * size
how = 'left'
indicator = False
left_len = 346893
left_data = {}
for col in (set(left_cols) - set(left_on)):
left_data[col] = create_sample_column_data(col, column_types[col], left_len)
left = pd.DataFrame(left_data)
left[left_on] = left_merge_df
right_len = 1544
right_data = {}
for col in (set(right_cols) - set(right_on)):
right_data[col] = create_sample_column_data(col, column_types[col], right_len)
right = pd.DataFrame(right_data)
right[right_on] = right_merge_df
profile = LineProfiler(
pd.core.reshape.merge._MergeOperation.get_result,
pd.core.internals.concatenate_block_managers,
pd.core.internals.concatenate_join_units,
pd.core.internals.JoinUnit.get_reindexed_values,
pd.core.algorithms.take_nd,
)
profile.run('pd.merge(left=left, right=right, left_on=left_on, right_on=right_on, how=how, indicator=indicator)', )
profile.print_stats()
Problem description
Pandas DataFrame merge is much slower on Python 3.6 and 3.7 as compared to Python 3.5 for some merges. I tried drilling through the merge method to find the lines where line_profiler was showing a lot of time being spent and I got to pd.core.algorithms.take_nd. I tried running some of the slow calls to the method from Python 3.6 in Python 3.5 and got the same timing so I suspect something higher in the call hierarchy may be causing this. It does seem like take_nd is being called many more times in Python > 3.5. The number of hits on take_nd varies every time I run Python >3.5 as well. See results below.
| Python Version | Merge Duration | take_nd Hits |
|---|---|---|
| 3.5 | 0.28786 s | 13 |
| 3.6 | 2.97011 s | 88 |
| 3.7 | 2.85604 s | 89 |
Expected Output
Python 3.5 Profile
Timer unit: 3.11047e-07 s
Total time: 0.185652 s
File: C:\ProgramData\Continuum\Anaconda\envs\py35\lib\site-packages\pandas\core\algorithms.py
Function: take_nd at line 1548
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1548 def take_nd(arr, indexer, axis=0, out=None, fill_value=np.nan, mask_info=None,
1549 allow_fill=True):
1550 """
1551 Specialized Cython take which sets NaN values in one pass
1552
1553 This dispatches to ``take`` defined on ExtensionArrays. It does not
1554 currently dispatch to ``SparseArray.take`` for sparse ``arr``.
1555
1556 Parameters
1557 ----------
1558 arr : array-like
1559 Input array.
1560 indexer : ndarray
1561 1-D array of indices to take, subarrays corresponding to -1 value
1562 indices are filed with fill_value
1563 axis : int, default 0
1564 Axis to take from
1565 out : ndarray or None, default None
1566 Optional output array, must be appropriate type to hold input and
1567 fill_value together, if indexer has any -1 value entries; call
1568 _maybe_promote to determine this type for any fill_value
1569 fill_value : any, default np.nan
1570 Fill value to replace -1 values with
1571 mask_info : tuple of (ndarray, boolean)
1572 If provided, value should correspond to:
1573 (indexer != -1, (indexer != -1).any())
1574 If not provided, it will be computed internally if necessary
1575 allow_fill : boolean, default True
1576 If False, indexer is assumed to contain no -1 values so no filling
1577 will be done. This short-circuits computation of a mask. Result is
1578 undefined if allow_fill == False and -1 is present in indexer.
1579
1580 Returns
1581 -------
1582 subarray : array-like
1583 May be the same type as the input, or cast to an ndarray.
1584 """
1585
1586 # TODO(EA): Remove these if / elifs as datetimeTZ, interval, become EAs
1587 # dispatch to internal type takes
1588 13 981.0 75.5 0.2 if is_extension_array_dtype(arr):
1589 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1590 13 343.0 26.4 0.1 elif is_datetimetz(arr):
1591 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1592 13 432.0 33.2 0.1 elif is_interval_dtype(arr):
1593 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1594
1595 13 261.0 20.1 0.0 if is_sparse(arr):
1596 arr = arr.get_values()
1597 13 160.0 12.3 0.0 elif isinstance(arr, (ABCIndexClass, ABCSeries)):
1598 arr = arr.values
1599
1600 13 217.0 16.7 0.0 arr = np.asarray(arr)
1601
1602 13 41.0 3.2 0.0 if indexer is None:
1603 indexer = np.arange(arr.shape[axis], dtype=np.int64)
1604 dtype, fill_value = arr.dtype, arr.dtype.type()
1605 else:
1606 13 123.0 9.5 0.0 indexer = _ensure_int64(indexer, copy=False)
1607 13 40.0 3.1 0.0 if not allow_fill:
1608 1 11.0 11.0 0.0 dtype, fill_value = arr.dtype, arr.dtype.type()
1609 1 3.0 3.0 0.0 mask_info = None, False
1610 else:
1611 # check for promotion based on types only (do this first because
1612 # it's faster than computing a mask)
1613 12 2284.0 190.3 0.4 dtype, fill_value = maybe_promote(arr.dtype, fill_value)
1614 12 82.0 6.8 0.0 if dtype != arr.dtype and (out is None or out.dtype != dtype):
1615 # check if promotion is actually required based on indexer
1616 4 12.0 3.0 0.0 if mask_info is not None:
1617 mask, needs_masking = mask_info
1618 else:
1619 4 3498.0 874.5 0.6 mask = indexer == -1
1620 4 378.0 94.5 0.1 needs_masking = mask.any()
1621 4 18.0 4.5 0.0 mask_info = mask, needs_masking
1622 4 15.0 3.8 0.0 if needs_masking:
1623 1 3.0 3.0 0.0 if out is not None and out.dtype != dtype:
1624 raise TypeError('Incompatible type for fill_value')
1625 else:
1626 # if not, then depromote, set fill_value to dummy
1627 # (it won't be used but we don't want the cython code
1628 # to crash when trying to cast it to dtype)
1629 3 36.0 12.0 0.0 dtype, fill_value = arr.dtype, arr.dtype.type()
1630
1631 13 44.0 3.4 0.0 flip_order = False
1632 13 54.0 4.2 0.0 if arr.ndim == 2:
1633 9 58.0 6.4 0.0 if arr.flags.f_contiguous:
1634 4 12.0 3.0 0.0 flip_order = True
1635
1636 13 39.0 3.0 0.0 if flip_order:
1637 4 29.0 7.2 0.0 arr = arr.T
1638 4 20.0 5.0 0.0 axis = arr.ndim - axis - 1
1639 4 12.0 3.0 0.0 if out is not None:
1640 out = out.T
1641
1642 # at this point, it's guaranteed that dtype can hold both the arr values
1643 # and the fill_value
1644 13 42.0 3.2 0.0 if out is None:
1645 13 117.0 9.0 0.0 out_shape = list(arr.shape)
1646 13 84.0 6.5 0.0 out_shape[axis] = len(indexer)
1647 13 83.0 6.4 0.0 out_shape = tuple(out_shape)
1648 13 69.0 5.3 0.0 if arr.flags.f_contiguous and axis == arr.ndim - 1:
1649 # minor tweak that can make an order-of-magnitude difference
1650 # for dataframes initialized directly from 2-d ndarrays
1651 # (s.t. df.values is c-contiguous and df._data.blocks[0] is its
1652 # f-contiguous transpose)
1653 4 133.0 33.2 0.0 out = np.empty(out_shape, dtype=dtype, order='F')
1654 else:
1655 9 166609.0 18512.1 27.9 out = np.empty(out_shape, dtype=dtype)
1656
1657 13 123.0 9.5 0.0 func = _get_take_nd_function(arr.ndim, arr.dtype, out.dtype, axis=axis,
1658 13 499.0 38.4 0.1 mask_info=mask_info)
1659 13 419613.0 32277.9 70.3 func(arr, indexer, out, fill_value)
1660
1661 13 166.0 12.8 0.0 if flip_order:
1662 4 70.0 17.5 0.0 out = out.T
1663 13 48.0 3.7 0.0 return out
...
Total time: 0.28786 s
File: C:\ProgramData\Continuum\Anaconda\envs\py35\lib\site-packages\pandas\core\reshape\merge.py
Function: get_result at line 563
Line # Hits Time Per Hit % Time Line Contents
==============================================================
563 def get_result(self):
564 1 5.0 5.0 0.0 if self.indicator:
565 self.left, self.right = self._indicator_pre_merge(
566 self.left, self.right)
567
568 1 240962.0 240962.0 26.0 join_index, left_indexer, right_indexer = self._get_join_info()
569
570 1 20.0 20.0 0.0 ldata, rdata = self.left._data, self.right._data
571 1 4.0 4.0 0.0 lsuf, rsuf = self.suffixes
572
573 1 24.0 24.0 0.0 llabels, rlabels = items_overlap_with_suffix(ldata.items, lsuf,
574 1 2488.0 2488.0 0.3 rdata.items, rsuf)
575
576 1 5.0 5.0 0.0 lindexers = {1: left_indexer} if left_indexer is not None else {}
577 1 3.0 3.0 0.0 rindexers = {1: right_indexer} if right_indexer is not None else {}
578
579 1 10.0 10.0 0.0 result_data = concatenate_block_managers(
580 1 4.0 4.0 0.0 [(ldata, lindexers), (rdata, rindexers)],
581 1 848.0 848.0 0.1 axes=[llabels.append(rlabels), join_index],
582 1 678985.0 678985.0 73.4 concat_axis=0, copy=self.copy)
583
584 1 20.0 20.0 0.0 typ = self.left._constructor
585 1 112.0 112.0 0.0 result = typ(result_data).__finalize__(self, method=self._merge_type)
586
587 1 3.0 3.0 0.0 if self.indicator:
588 result = self._indicator_post_merge(result)
589
590 1 1806.0 1806.0 0.2 self._maybe_add_join_keys(result, left_indexer, right_indexer)
591
592 1 153.0 153.0 0.0 self._maybe_restore_index_levels(result)
593
594 1 3.0 3.0 0.0 return result
Python 3.6 Profile
Timer unit: 3.11047e-07 s
Total time: 2.11554 s
File: C:\ProgramData\Continuum\Anaconda\envs\py36\lib\site-packages\pandas\core\algorithms.py
Function: take_nd at line 1548
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1548 def take_nd(arr, indexer, axis=0, out=None, fill_value=np.nan, mask_info=None,
1549 allow_fill=True):
1550 """
1551 Specialized Cython take which sets NaN values in one pass
1552
1553 This dispatches to ``take`` defined on ExtensionArrays. It does not
1554 currently dispatch to ``SparseArray.take`` for sparse ``arr``.
1555
1556 Parameters
1557 ----------
1558 arr : array-like
1559 Input array.
1560 indexer : ndarray
1561 1-D array of indices to take, subarrays corresponding to -1 value
1562 indices are filed with fill_value
1563 axis : int, default 0
1564 Axis to take from
1565 out : ndarray or None, default None
1566 Optional output array, must be appropriate type to hold input and
1567 fill_value together, if indexer has any -1 value entries; call
1568 _maybe_promote to determine this type for any fill_value
1569 fill_value : any, default np.nan
1570 Fill value to replace -1 values with
1571 mask_info : tuple of (ndarray, boolean)
1572 If provided, value should correspond to:
1573 (indexer != -1, (indexer != -1).any())
1574 If not provided, it will be computed internally if necessary
1575 allow_fill : boolean, default True
1576 If False, indexer is assumed to contain no -1 values so no filling
1577 will be done. This short-circuits computation of a mask. Result is
1578 undefined if allow_fill == False and -1 is present in indexer.
1579
1580 Returns
1581 -------
1582 subarray : array-like
1583 May be the same type as the input, or cast to an ndarray.
1584 """
1585
1586 # TODO(EA): Remove these if / elifs as datetimeTZ, interval, become EAs
1587 # dispatch to internal type takes
1588 88 8930.0 101.5 0.1 if is_extension_array_dtype(arr):
1589 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1590 88 3047.0 34.6 0.0 elif is_datetimetz(arr):
1591 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1592 88 3553.0 40.4 0.1 elif is_interval_dtype(arr):
1593 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1594
1595 88 2296.0 26.1 0.0 if is_sparse(arr):
1596 arr = arr.get_values()
1597 88 1180.0 13.4 0.0 elif isinstance(arr, (ABCIndexClass, ABCSeries)):
1598 arr = arr.values
1599
1600 88 2013.0 22.9 0.0 arr = np.asarray(arr)
1601
1602 88 319.0 3.6 0.0 if indexer is None:
1603 indexer = np.arange(arr.shape[axis], dtype=np.int64)
1604 dtype, fill_value = arr.dtype, arr.dtype.type()
1605 else:
1606 88 930.0 10.6 0.0 indexer = _ensure_int64(indexer, copy=False)
1607 88 297.0 3.4 0.0 if not allow_fill:
1608 1 9.0 9.0 0.0 dtype, fill_value = arr.dtype, arr.dtype.type()
1609 1 4.0 4.0 0.0 mask_info = None, False
1610 else:
1611 # check for promotion based on types only (do this first because
1612 # it's faster than computing a mask)
1613 87 17825.0 204.9 0.3 dtype, fill_value = maybe_promote(arr.dtype, fill_value)
1614 87 834.0 9.6 0.0 if dtype != arr.dtype and (out is None or out.dtype != dtype):
1615 # check if promotion is actually required based on indexer
1616 54 209.0 3.9 0.0 if mask_info is not None:
1617 mask, needs_masking = mask_info
1618 else:
1619 54 35668.0 660.5 0.5 mask = indexer == -1
1620 54 6300.0 116.7 0.1 needs_masking = mask.any()
1621 54 274.0 5.1 0.0 mask_info = mask, needs_masking
1622 54 249.0 4.6 0.0 if needs_masking:
1623 1 4.0 4.0 0.0 if out is not None and out.dtype != dtype:
1624 raise TypeError('Incompatible type for fill_value')
1625 else:
1626 # if not, then depromote, set fill_value to dummy
1627 # (it won't be used but we don't want the cython code
1628 # to crash when trying to cast it to dtype)
1629 53 698.0 13.2 0.0 dtype, fill_value = arr.dtype, arr.dtype.type()
1630
1631 88 304.0 3.5 0.0 flip_order = False
1632 88 461.0 5.2 0.0 if arr.ndim == 2:
1633 84 651.0 7.8 0.0 if arr.flags.f_contiguous:
1634 4 13.0 3.2 0.0 flip_order = True
1635
1636 88 304.0 3.5 0.0 if flip_order:
1637 4 42.0 10.5 0.0 arr = arr.T
1638 4 22.0 5.5 0.0 axis = arr.ndim - axis - 1
1639 4 14.0 3.5 0.0 if out is not None:
1640 out = out.T
1641
1642 # at this point, it's guaranteed that dtype can hold both the arr values
1643 # and the fill_value
1644 88 339.0 3.9 0.0 if out is None:
1645 88 1023.0 11.6 0.0 out_shape = list(arr.shape)
1646 88 784.0 8.9 0.0 out_shape[axis] = len(indexer)
1647 88 646.0 7.3 0.0 out_shape = tuple(out_shape)
1648 88 380.0 4.3 0.0 if arr.flags.f_contiguous and axis == arr.ndim - 1:
1649 # minor tweak that can make an order-of-magnitude difference
1650 # for dataframes initialized directly from 2-d ndarrays
1651 # (s.t. df.values is c-contiguous and df._data.blocks[0] is its
1652 # f-contiguous transpose)
1653 4 126.0 31.5 0.0 out = np.empty(out_shape, dtype=dtype, order='F')
1654 else:
1655 84 1910204.0 22740.5 28.1 out = np.empty(out_shape, dtype=dtype)
1656
1657 88 1244.0 14.1 0.0 func = _get_take_nd_function(arr.ndim, arr.dtype, out.dtype, axis=axis,
1658 88 4880.0 55.5 0.1 mask_info=mask_info)
1659 88 4793020.0 54466.1 70.5 func(arr, indexer, out, fill_value)
1660
1661 88 1715.0 19.5 0.0 if flip_order:
1662 4 145.0 36.2 0.0 out = out.T
1663 88 397.0 4.5 0.0 return out
...
Total time: 2.97011 s
File: C:\ProgramData\Continuum\Anaconda\envs\py36\lib\site-packages\pandas\core\reshape\merge.py
Function: get_result at line 563
Line # Hits Time Per Hit % Time Line Contents
==============================================================
563 def get_result(self):
564 1 6.0 6.0 0.0 if self.indicator:
565 self.left, self.right = self._indicator_pre_merge(
566 self.left, self.right)
567
568 1 222965.0 222965.0 2.3 join_index, left_indexer, right_indexer = self._get_join_info()
569
570 1 15.0 15.0 0.0 ldata, rdata = self.left._data, self.right._data
571 1 7.0 7.0 0.0 lsuf, rsuf = self.suffixes
572
573 1 18.0 18.0 0.0 llabels, rlabels = items_overlap_with_suffix(ldata.items, lsuf,
574 1 2756.0 2756.0 0.0 rdata.items, rsuf)
575
576 1 5.0 5.0 0.0 lindexers = {1: left_indexer} if left_indexer is not None else {}
577 1 3.0 3.0 0.0 rindexers = {1: right_indexer} if right_indexer is not None else {}
578
579 1 3.0 3.0 0.0 result_data = concatenate_block_managers(
580 1 4.0 4.0 0.0 [(ldata, lindexers), (rdata, rindexers)],
581 1 805.0 805.0 0.0 axes=[llabels.append(rlabels), join_index],
582 1 9319169.0 9319169.0 97.6 concat_axis=0, copy=self.copy)
583
584 1 25.0 25.0 0.0 typ = self.left._constructor
585 1 135.0 135.0 0.0 result = typ(result_data).__finalize__(self, method=self._merge_type)
586
587 1 4.0 4.0 0.0 if self.indicator:
588 result = self._indicator_post_merge(result)
589
590 1 2641.0 2641.0 0.0 self._maybe_add_join_keys(result, left_indexer, right_indexer)
591
592 1 186.0 186.0 0.0 self._maybe_restore_index_levels(result)
593
594 1 2.0 2.0 0.0 return result
Python 3.7 Profile
Timer unit: 3.11047e-07 s
Total time: 2.05158 s
File: C:\ProgramData\Continuum\Anaconda\envs\py37\lib\site-packages\pandas\core\algorithms.py
Function: take_nd at line 1548
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1548 def take_nd(arr, indexer, axis=0, out=None, fill_value=np.nan, mask_info=None,
1549 allow_fill=True):
1550 """
1551 Specialized Cython take which sets NaN values in one pass
1552
1553 This dispatches to ``take`` defined on ExtensionArrays. It does not
1554 currently dispatch to ``SparseArray.take`` for sparse ``arr``.
1555
1556 Parameters
1557 ----------
1558 arr : array-like
1559 Input array.
1560 indexer : ndarray
1561 1-D array of indices to take, subarrays corresponding to -1 value
1562 indices are filed with fill_value
1563 axis : int, default 0
1564 Axis to take from
1565 out : ndarray or None, default None
1566 Optional output array, must be appropriate type to hold input and
1567 fill_value together, if indexer has any -1 value entries; call
1568 _maybe_promote to determine this type for any fill_value
1569 fill_value : any, default np.nan
1570 Fill value to replace -1 values with
1571 mask_info : tuple of (ndarray, boolean)
1572 If provided, value should correspond to:
1573 (indexer != -1, (indexer != -1).any())
1574 If not provided, it will be computed internally if necessary
1575 allow_fill : boolean, default True
1576 If False, indexer is assumed to contain no -1 values so no filling
1577 will be done. This short-circuits computation of a mask. Result is
1578 undefined if allow_fill == False and -1 is present in indexer.
1579
1580 Returns
1581 -------
1582 subarray : array-like
1583 May be the same type as the input, or cast to an ndarray.
1584 """
1585
1586 # TODO(EA): Remove these if / elifs as datetimeTZ, interval, become EAs
1587 # dispatch to internal type takes
1588 89 7400.0 83.1 0.1 if is_extension_array_dtype(arr):
1589 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1590 89 2994.0 33.6 0.0 elif is_datetimetz(arr):
1591 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1592 89 3451.0 38.8 0.1 elif is_interval_dtype(arr):
1593 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
1594
1595 89 1707.0 19.2 0.0 if is_sparse(arr):
1596 arr = arr.get_values()
1597 89 940.0 10.6 0.0 elif isinstance(arr, (ABCIndexClass, ABCSeries)):
1598 arr = arr.values
1599
1600 89 2688.0 30.2 0.0 arr = np.asarray(arr)
1601
1602 89 332.0 3.7 0.0 if indexer is None:
1603 indexer = np.arange(arr.shape[axis], dtype=np.int64)
1604 dtype, fill_value = arr.dtype, arr.dtype.type()
1605 else:
1606 89 1025.0 11.5 0.0 indexer = _ensure_int64(indexer, copy=False)
1607 89 310.0 3.5 0.0 if not allow_fill:
1608 1 8.0 8.0 0.0 dtype, fill_value = arr.dtype, arr.dtype.type()
1609 1 3.0 3.0 0.0 mask_info = None, False
1610 else:
1611 # check for promotion based on types only (do this first because
1612 # it's faster than computing a mask)
1613 88 16582.0 188.4 0.3 dtype, fill_value = maybe_promote(arr.dtype, fill_value)
1614 88 680.0 7.7 0.0 if dtype != arr.dtype and (out is None or out.dtype != dtype):
1615 # check if promotion is actually required based on indexer
1616 52 196.0 3.8 0.0 if mask_info is not None:
1617 mask, needs_masking = mask_info
1618 else:
1619 52 25628.0 492.8 0.4 mask = indexer == -1
1620 52 5112.0 98.3 0.1 needs_masking = mask.any()
1621 52 254.0 4.9 0.0 mask_info = mask, needs_masking
1622 52 248.0 4.8 0.0 if needs_masking:
1623 1 3.0 3.0 0.0 if out is not None and out.dtype != dtype:
1624 raise TypeError('Incompatible type for fill_value')
1625 else:
1626 # if not, then depromote, set fill_value to dummy
1627 # (it won't be used but we don't want the cython code
1628 # to crash when trying to cast it to dtype)
1629 51 656.0 12.9 0.0 dtype, fill_value = arr.dtype, arr.dtype.type()
1630
1631 89 302.0 3.4 0.0 flip_order = False
1632 89 461.0 5.2 0.0 if arr.ndim == 2:
1633 85 653.0 7.7 0.0 if arr.flags.f_contiguous:
1634 4 12.0 3.0 0.0 flip_order = True
1635
1636 89 315.0 3.5 0.0 if flip_order:
1637 4 33.0 8.2 0.0 arr = arr.T
1638 4 22.0 5.5 0.0 axis = arr.ndim - axis - 1
1639 4 13.0 3.2 0.0 if out is not None:
1640 out = out.T
1641
1642 # at this point, it's guaranteed that dtype can hold both the arr values
1643 # and the fill_value
1644 89 333.0 3.7 0.0 if out is None:
1645 89 853.0 9.6 0.0 out_shape = list(arr.shape)
1646 89 701.0 7.9 0.0 out_shape[axis] = len(indexer)
1647 89 651.0 7.3 0.0 out_shape = tuple(out_shape)
1648 89 393.0 4.4 0.0 if arr.flags.f_contiguous and axis == arr.ndim - 1:
1649 # minor tweak that can make an order-of-magnitude difference
1650 # for dataframes initialized directly from 2-d ndarrays
1651 # (s.t. df.values is c-contiguous and df._data.blocks[0] is its
1652 # f-contiguous transpose)
1653 4 116.0 29.0 0.0 out = np.empty(out_shape, dtype=dtype, order='F')
1654 else:
1655 85 1881815.0 22139.0 28.5 out = np.empty(out_shape, dtype=dtype)
1656
1657 89 1210.0 13.6 0.0 func = _get_take_nd_function(arr.ndim, arr.dtype, out.dtype, axis=axis,
1658 89 3814.0 42.9 0.1 mask_info=mask_info)
1659 89 4631805.0 52042.8 70.2 func(arr, indexer, out, fill_value)
1660
1661 89 1579.0 17.7 0.0 if flip_order:
1662 4 74.0 18.5 0.0 out = out.T
1663 89 358.0 4.0 0.0 return out
...
Total time: 2.85604 s
File: C:\ProgramData\Continuum\Anaconda\envs\py37\lib\site-packages\pandas\core\reshape\merge.py
Function: get_result at line 563
Line # Hits Time Per Hit % Time Line Contents
==============================================================
563 def get_result(self):
564 1 4.0 4.0 0.0 if self.indicator:
565 self.left, self.right = self._indicator_pre_merge(
566 self.left, self.right)
567
568 1 224006.0 224006.0 2.4 join_index, left_indexer, right_indexer = self._get_join_info()
569
570 1 20.0 20.0 0.0 ldata, rdata = self.left._data, self.right._data
571 1 3.0 3.0 0.0 lsuf, rsuf = self.suffixes
572
573 1 21.0 21.0 0.0 llabels, rlabels = items_overlap_with_suffix(ldata.items, lsuf,
574 1 2305.0 2305.0 0.0 rdata.items, rsuf)
575
576 1 5.0 5.0 0.0 lindexers = {1: left_indexer} if left_indexer is not None else {}
577 1 2.0 2.0 0.0 rindexers = {1: right_indexer} if right_indexer is not None else {}
578
579 1 3.0 3.0 0.0 result_data = concatenate_block_managers(
580 1 4.0 4.0 0.0 [(ldata, lindexers), (rdata, rindexers)],
581 1 793.0 793.0 0.0 axes=[llabels.append(rlabels), join_index],
582 1 8952267.0 8952267.0 97.5 concat_axis=0, copy=self.copy)
583
584 1 24.0 24.0 0.0 typ = self.left._constructor
585 1 129.0 129.0 0.0 result = typ(result_data).__finalize__(self, method=self._merge_type)
586
587 1 5.0 5.0 0.0 if self.indicator:
588 result = self._indicator_post_merge(result)
589
590 1 2278.0 2278.0 0.0 self._maybe_add_join_keys(result, left_indexer, right_indexer)
591
592 1 154.0 154.0 0.0 self._maybe_restore_index_levels(result)
593
594 1 3.0 3.0 0.0 return result
Output of pd.show_versions() for Python 3.5
conda create -n py35 python=3.5.6 pandas=0.23.4 line_profiler=2.1.2
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.6.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 45 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.4
pytest: None
pip: 10.0.1
setuptools: 40.2.0
Cython: None
numpy: 1.15.2
scipy: None
pyarrow: None
xarray: None
IPython: 6.5.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
Output of pd.show_versions() for Python 3.6
conda create -n py36 python=3.6.6 pandas=0.23.4 line_profiler=2.1.2
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 45 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.4
pytest: None
pip: 10.0.1
setuptools: 40.4.3
Cython: None
numpy: 1.15.2
scipy: None
pyarrow: None
xarray: None
IPython: 7.0.1
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
Output of pd.show_versions() for Python 3.7
conda create -n py37 python=3.7.0 pandas=0.23.4 line_profiler=2.1.2
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.0.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 45 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.4
pytest: None
pip: 10.0.1
setuptools: 40.4.3
Cython: None
numpy: 1.15.2
scipy: None
pyarrow: None
xarray: None
IPython: 7.0.1
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
Issue Analytics
- State:
- Created 5 years ago
- Reactions:2
- Comments:12 (7 by maintainers)
Top Results From Across the Web
Pandas merge is very slow on large data set - Stack Overflow
I am looping through each column and creating pandas data frame and appending the column it.if column belong to time which is already...
Read more >What's New In Python 3.7 — Python 3.11.1 documentation
This article explains the new features in Python 3.7, compared to 3.6. Python 3.7 was released on June 27, 2018. For full details,...
Read more >Using Alpine can make Python Docker builds 50× slower
Let's build a Python image. We want to package a Python application that uses pandas and matplotlib . So one option is to...
Read more >What's New — pandas 0.23.4 documentation - PyData |
Instantiation from dicts respects order for Python 3.6+. Dependent column arguments for assign. Merging / sorting on a combination of columns and index ......
Read more >OrderedDict vs dict in Python: The Right Tool for the Job
It's a dictionary subclass specially designed to remember the order of items, which is defined by the insertion order of keys. This changed...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
pandas is all volunteer
would appreciate some analysis to see where the issue is
Any updates on this? I am still seeing similar performance issues with pandas 0.25.3, numpy 1.18, and python 3.6.