I think about this too, although at best you would be talking about a morale hit to the DE/LB the two or three times a game that individual player gets to you which probably isn't a huge factor.
Maybe a running QB would get better use out of it, but even then I'd rather be faster (if all we do is run) or throw better (if we play balanced).