Iteration over DatetimeIndex stops at 10000

Code Sample

import pandas as pd

num_index = pd.Index(range(20000))
hour_index = pd.date_range('2000-01-01', periods=20000, freq='H')
minute_index = pd.date_range('2000-01-01', periods=20000, freq='min')

a = 0
for i in num_index: # OK
    a += 1

b = 0
for i in hour_index: # unexpectedly
    b += 1

c = 0
for i in minute_index: # unexpectedly
    c += 1

a, b, c # -> (20000, 10000, 10000)

Problem description

Iteration over DatetimeIndex stops unexpectedly when it’s evaluated 10000 times.

NOTE : this happened in 0.23.0rc2, not 0.22.0

Expected Output

(20000, 20000, 20000)

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None python: 3.6.5.final.0 python-bits: 64 OS: Linux OS-release: 3.10.0-693.21.1.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: ja_JP.UTF-8 LOCALE: ja_JP.UTF-8

pandas: 0.23.0rc2 pytest: None pip: 10.0.1 setuptools: 39.1.0 Cython: None numpy: 1.14.3 scipy: None pyarrow: None xarray: None IPython: None sphinx: None patsy: None dateutil: 2.7.2 pytz: 2018.4 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: None openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: None sqlalchemy: None pymysql: None psycopg2: None jinja2: None s3fs: None fastparquet: None pandas_gbq: None pandas_datareader: None

[cbertinato]

The issue is not so much about dimensionality as it is about the identity of the index as an iterator. This leads to a somewhat deeper question: does it matter whether the index itself is an iterator? Perhaps the answer is: if the tests pass, it doesn’t matter.

If it does matter, then another solution is to make a separate iterator for indexes, instead of returning the index itself from __iter__.

Anyhow, something along the lines of what you suggest would fix it, though something a bit more general, such as not isinstance(tuples, Index) might be better in case something like this crops up for other indexes.

[kittoku]

A separate iterator sounds good. All of the three test passed because DatetimeIndex itself is not a iterator. I’m going to post a PR later.