Too Many Ads
If you display banner ads on the top of your blog and the bottom of your blog and the side of your blog and the middle of your blog...THAT'S TOO MANY ADS!
Haven't you ever been turned off of a site that had too many ads? Your readers feel the same way. You need to weigh the income benefits of your display ads against the number of readers you might be losing from too many ads.
If you're making a hefty profit from the ads, then by all means keep them, they're clearly serving their purpose and not alienating readers since they're clicking on the ads.
If the ads are only making you a few bucks a month and look a bit spammy, reduce how many are displayed for a cleaner looking blog. That's a tiny price to pay for a better first (and returning) impression.