I can imagine that some learner's don't handle factors with flipped levels perfectly.
A proposal would be to add an autotest that checks whether predictions are equivalent irrespective of the underlying factor representation.
Then, there could be a learner property that indicates whether the learner can deal with flipped factor levels.
In the $predict_newdata() and the $predict() method we could then (depending on whether this property is present) throw an error.