Raj Bhawan rejected the allegations