It is based on which IV is highest:
Attack - Hitmonlee
Defense - Hitmonchan
HP - Hitmontop
If two or three IVs are tied for highest, it is random between those two or three evolutions.
IIRC, initial studies when Tyrogue was first released suggested that two-way ties were 50:50 chance, but three-way ties were closer to 25:25:50 in favor of Hitmontop. (Probably accidentally splitting the chance to two steps: 50:50 is it Top? Then if not Top, 50:50 is it Lee or Chan.) I don't know if there were any further studies to find out if this observation was either true or fixed.