Economy: Supply and demand basically. And the value of the item and variety.
Actually the cheapest item goes first. Say you put something for 890 gil, the base price is 900 gil for the item in AH. Someone bids 900, even though someone else put it at 900, the 890 one sells first.If that price is met by a player making a bid then your item sells, depending on the reserves set by other players and the quantity of that item placed on auction with the same reserve price before yours