The word you use for family members depends on the situation. If you're speaking to someone else about your own younger brother, you use otouto (弟). If you're speaking about someone else's younger brother, you use otoutosan (弟さん), or informally, otouto (弟). If you're speaking to your own younger brother, you just use his name. (When speaking to your own older brother, you may or may not use his name.)