sabre algorithm improvements in transpilation at level 3 not in effect

Environment

• Qiskit Terra version: 0.22.2 (vs. 0.21.1)
• Python version: 3.8.2
• Operating system: Windows

What is happening?

transpilation at optimization_level=3 does not (significantly) perform better in terra 0.22.2 than 0.21.1 despite improvements in sabre algorithm.

How can we reproduce the issue?

both in terra 0.22.2 and 0.21.1 run:

qc.x(5)
qc.h(range(6))
qc.cx(range(5),5)
qc.h(range(5))
qc.measure(range(5), range(5))
qc.draw('mpl')

# Transpile the circuit 'qc' 1000 times
trans_circ_list = transpile([qc]*1000, backend, optimization_level=3)

# get the number of cnot gates
cx_counts = np.asarray([circ.count_ops().get('cx', 0) for circ in trans_circ_list])

# create violin plot
import seaborn as sns
ax = sns.violinplot(data=[cx_counts])

What should happen?

Acc to level 3 preset , there should be 20 swap_trials performed for each iteration of transpile. So, the performance should be improved by these 20 iterations. Instead, there is no visible difference, but doing 20 iterations of transpile manually in qiskit 0.21.1 performs much better (will take about 60mins on a 4core-CPU):

global_circ_list=[]

for i in range(1000):
    # Transpile the circuit 'qc' 20 times
    trans_circ_list_loop = transpile([qc]*20, backend, optimization_level=3)

    # get the number of cnot gates
    cx_counts_loop = np.asarray([circ.count_ops().get('cx', 0) for circ in trans_circ_list_loop])
    #print('Number of cnots:', cx_counts_loop)

    # select the circuit with the lowest number
    best_idx_loop = np.argmin(cx_counts_loop)
    best_trans_qc_loop = trans_circ_list_loop[best_idx_loop]
    global_circ_list.append(best_trans_qc_loop)

cx_counts_global = np.asarray([circ.count_ops().get('cx', 0) for circ in global_circ_list]) 

combining all in the plot shows that manually pre-transpiling in the older qiskit version performs much better

Any suggestions?

there might be a minimization step missing, or a mixup of max_iterations and swap_trials in https://github.com/Qiskit/qiskit-terra/blob/90b158c7e02432db957762e58c4c2ed75d89a1ca/qiskit/transpiler/preset_passmanagers/level3.py#L140

[mtreinish]

Hmm, yeah I was seeing bad performance with that branch in other benchmarks too: https://github.com/Qiskit/qiskit-terra/pull/9116#issuecomment-1311955595 I’m still not super confident with it which is why it’s [WIP] still, I probably broke something or am doing something else wrong in the PR.

At least running the EfficientSU2 example on main and 0.22.2 returns 68 for all the circuits.

[mtreinish]

I pushed up https://github.com/Qiskit/qiskit-terra/pull/9116 to do multiple seed trials for the combination of layout and routing here. I still need to tune the PR and characterize the performance (both in quality and runtime). But if you’d like to give that a try and see how it works for your use case that would be useful feedback while it’s still under review/being developed.