when you say "leave out gravity" that implies not using the gravitational force or any multiple of it

I wouldn't say necessarily that physicists are "skilled" at math; they just use math as a tool to explain physics. They don't go poking at the borders of geometry and number theory or anything, they just use regressions, probability, differential equations etc to model things they observe in the universe. They do take higher math courses than say your average art historian, but they don't usually dig any deeper into the subjects. Engineers do the same thing with physics, using physics (and by association, math) to describe things that happen in their specific fields.