To the best of my knowledge, time holding a gym doesn't give you any progress on gym badges. First time you spin a gym photo disc, feed a berry, attack it or place a defender you get an empty badge. There are actions that grant you points towards badge level (bronze, silver, gold).
Putting a defender gives +100 badge points. Feeding berries, fighting, etc give you a variable number of points (no idea what's the formula here).
There's another different medal which does count time holding a gym (the sum of all the gyms you defended). For the gold medal you need a total of 1000 hours.
Please, someone correct me if time holding a gym accounts for badge progress.