The Allahabad HC on Friday refused to set aside the decision of a lower court in Shahjahanpur, which had declined to grant permission to the state government for the withdrawal of alleged rape case pending against ex-Union minister Swami Chinmayanand Saraswati.