Like many key items, they have no use but to add to your Treasure Hunter Ranking. If you achieve an "S" ranking and confront the Four-Armed Man, he'll say a nice little speech (which happens to reveal his true name), give you a special medal (which also turns out to be useless, darn it), and walk out of your life till his next cameo.