Empty pipe taste?

Let me preface with this. Ive got 2 MM Country Gentleman pipes that I truly enjoy smoking. They are even starting to color a bit, so im NOT willing to just toss them for another set.

Usually as im preparing (drying) the tobacco for my next smoke ill have the pipe clenched. Inadvertantly ill take a draw and I get an "ashy" taste, even though I clean the stem with alcohol after every smoke. Ive tried cleaning the bowl as well but after every time I smoke the smell returns. I know that a full pipe cleaning isnt required or encouraged after every smoke, so is there something im missing here? Its not really affecting the smoking session itself.

I guess what im asking is...is this normal?