Party activists could be forgiven for wondering if he would now rather have Starmer in Downing Street than Sunak.