You're right, but it really depends. Sometimes, odd shaped brushes can cut visleaves in such a way that it causes you to see more visleaves than needed. At a minimum, it rapes your compile time and makes it hard to get the most optimal visleaves.
If you don't have enough vis leaf disruption, you will be drawing the entire map at all times. this is an issue with empires especially, and the main cause of empires lag, because there is no good way to make vis leafs in empires. When you have too many vis leafs, every vis leaf has it's own set of calculations for the lighting called Vrad. Vrad calculates lighting and uses the vis leafs to determine what lighting needs to be drawn at what point. In other words, if you have a stair case that cuts vis leafs to hell, then instead of having 1 solid vis leaf with 1 set of calculations for that area, you have that same area being calculated possibly hundreds of times over. This causes issues when it comes to in game lag when your comp is trying to load these precalculated areas and keep track of them, and it takes longer for it to load these areas because there are 10,000 more of them than there should be.