seaborn heatmap's colorbar is generating random ticks

I am trying to make a heatmap with logarithmic colorbar. But it keeps generating it’s own ticks and ticklabels.

import numpy as np
import seaborn as sns
from matplotlib.colors import LogNorm
import matplotlib.ticker as tkr

matrix = np.random.rand(10, 10)/0.4
vmax=2
vmin=0.5

cbar_ticks = [0.5, 0.75, 1, 1.33, 2]
formatter = tkr.ScalarFormatter(useMathText=True)
formatter.set_scientific(False)

log_norm = LogNorm(vmin=vmin, vmax=vmax)
ax = sns.heatmap(matrix, square=True, vmax=vmax, vmin=vmin, norm=log_norm, cbar_kws={"ticks": cbar_ticks, "format": formatter})

[mwaskom]

That ax.collections[0].colorbar.ax object is the same as the cax object in my code, fyi.

[rikdhar]

The following completely removes the ticks and the label.

ax.collections[0].colorbar.ax.yaxis.set_ticks([], minor=True)