Castiel understands that and doesn't begrudge him the need to do so. "I need to return to heaven." He has a room full of angels to deal with. "When you're done checking on Sam ... well. It's been too long between us, and you know you are welcome anywhere I am."

It's probably redundant giving Dean an open invitation to heaven, but he isn't sure what else to say or do, and he knows that Dean and Sam will need time to themselves.