You have to switch to magic when the physical barrier goes up.
You also have to dispel his reflect, which will occasionally cause him to put up his magic barrier. So you just wait then.