devil with (it)

(colloquial) I, we, etc., do not care about (a person or thing).