date_range / creating trading indices

Hi @Stryder-Git, great to see a new take on evaluating a trading index, especially if its quicker!

I’ve copied your date_range function into a Jupyter Notebook and this…

import pandas_market_calendars as pmc
cal = pmc.get_calendar('XHKG')
schedule = cal.schedule("2021-07-07", "2021-07-07")
date_range(schedule, frequency="25T", close="left", force_close=True)

returned…

DatetimeIndex(['2021-07-07 02:00:00+00:00', '2021-07-07 02:25:00+00:00',
               '2021-07-07 02:50:00+00:00', '2021-07-07 03:15:00+00:00',
               '2021-07-07 03:40:00+00:00', '2021-07-07 05:20:00+00:00',
               '2021-07-07 05:45:00+00:00', '2021-07-07 06:10:00+00:00',
               '2021-07-07 06:35:00+00:00', '2021-07-07 07:00:00+00:00',
               '2021-07-07 07:25:00+00:00', '2021-07-07 07:50:00+00:00',
               '2021-07-07 08:00:00+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)

…which is the same as the return from the existing function.

As you’re working on this now I thought I’d mention a couple of things that occured to me when I was looking at using date_range. You may or may not want to consider them…

For reference: schedule.iloc[0] ->

market_open    2021-07-07 02:00:00+00:00
market_close   2021-07-07 08:00:00+00:00
break_start    2021-07-07 04:00:00+00:00
break_end      2021-07-07 05:00:00+00:00
Name: 2021-07-07 00:00:00, dtype: datetime64[ns, UTC]

• if forcing close on the market close, wouldn’t you also force close on the break start? i.e. shouldn’t 04:00 also be an indice in the above example?
• indices following the break end are only accurate if the break end is ‘on-frequency’ based off the market open. Above this is shown by the first post-break indice being 05:20. I would have thought this first post-break indice should be 05:00 (as closed on “left”), or at a push 05:25 (on-frequency based on break end), although 05:20 seems rather meaningless in light of there having been a trading break? Personally, I’d consider this a bug in the existing function.

Maybe some food for thought if you hadn’t already considered these points.

_Originally posted by @maread99 in https://github.com/rsheftel/pandas_market_calendars/issues/136#issuecomment-876002673_

[rsheftel]

—> Do you agree that aligning the indices along break_start and break_end makes more sense than how it has been handled originally, and that I can change those tests?

Yes.

I have chosen not to add a warning for this yet, let me know if you think I should.

We should definitely have a warning, otherwise I am sure we will get the question over and over in the issues or on Reddit

Since I found that this only concerns the following three calendars [‘CMES’, ‘IEPA’, ‘us_futures’], the function is just going to raise a ValueError when such a schedule is passed to it, requesting the user to adjust the schedule, rather than trying to handle this inside the function.

The problem with the futures calendars is that to make them start properly on a Sunday, the open needs to be greater than the close. We will have to think about how to handle this.

[maread99]

@Stryder-Git, I’ve only just got round to having a look at your new date_range. Tremendous work on the vectorised implementation!! I’ve overhauled my attempt with a very simlar vectorised evaluation - one which I would never have got anywhere close to if you hadn’t shown the way!

If you’re interested, the PR’s here in a bit of a devlopement queue over at exchange_calendars. Hopefully it will be incorporated prior to release 3.4. You’ll see the internals work in nanos (sourced from ExchangeCalendar’s nano properties). This has made it quite considerably faster than the current date_range. Using a schedule for Hong Kong and with closed as “right” I got:

• 33min frequency over a year, 8x faster (1.8ms v 14.7ms)
• 1ms frequency over a single day, 10x faster (1.6s v 16.6s)

Worth noting that, if you were interested in doing something similar with the nanos, you could simplify what I’ve done a lot by not offering the separate force_break_close option.