Well, also, given Narcissa, there's "What Barty and I have is just different" as a plausible bit of logic, as well as...

... well, Lucius does not always have great taste in mistresses, I think? (Didn't he have to frame the last one for something to finally get her to stop stalking him or something?)