I don't know if there is an exact answer, but as a first time father of a 16 month old we've had a few experiences that has influenced what we allow.
In the earlier days, TV was something we occasionally used for sensory stimulation. Myles seemed to enjoy seeing the motion of various shapes and characters while listening to nursery rhymes. By "enjoy" I mean he would be attentive, crack an occasional smile, and just seem generally satisfied. If a video would stop, it wouldn't be a big deal as he would avert his attention elsewhere.
As he started to develop very basic communication skills, we introduced him to something more complex: Sesame Street. I really like that show because of the diversity of subjects they cover. Watching the show with him was added to our play time routine and didn't seem to be an issue, but that quickly changed.
It's amazing how fast children develop. Suddenly Myles started to remember the characters, would clap along to the songs, and eventually associate the TV and even the TV remote with watching Elmo and friends. He really enjoyed it with a much deeper connection. When combined with the complexities of the toddler phase of development, certain aspects of TV started to become a problem.
A 13 month old toddler doesn't understand "the why", so if an episode of Sesame Street would end and it was time to turn off the TV for bath time, we had a very sad boy on our hands. If he excitingly found the TV remote but had his request denied, sad Myles would be in full effect. It became a source of tension often enough for us to hide the TV remote and completely stop asking him if he wanted to watch Elmo.
Today, we only watch one or two episodes of Sesame Street on the weekends. We are working parents, so our limited weeknight time with him is strictly for eating dinner, bath time, play time, and reading. We've found that having TV be out of sight and out of mind has made everybody much happier and makes it extra special when we do watch some Elmo. I won't lie, there are times we have to break the rules and there are times we need to allow a bit of Sesame Street while at a restaurant, but in general we are pretty good about limiting this.
I can only imagine that this will get more difficult over time. Especially when he can start playing video games, since I want to be player-two. 😀