"ɒfn" and "ɔːfən" are the most common ways British and American people pronounce "often", respectively. I'd say you're only actually pronouncing it wrong if you pronounced the "e" as "eh" rather than "uh".