Unrounded damage is used. You can check breakpoints for moves where a move damage breakpoint -1 is divisible by 5, e.g. Machamp v Tyranitar, Counter at level 36 reaches 16 damage, meaning the damage value in the formula has just crossed 15, as floor(15) + 1 = 16.
Switch to cloudy weather, and a 1.2x increase would result in a value of exactly 18, resulting in 19 damage at the exact same breakpoints that were previously 16 damage for all attack IVs.
If the rounded damage was used, then Counter when weather boosted would reach the breakpoint later as it wouldn't be receiving a 1.2x increase.