Castiel had had ample time to stew in the juices of his misdeeds long before he returned to the bunker, his groceries in hand. When Dean was not there he grew agitated. When a full day came and went without word from either of the Winchesters he grew worried.

Considering the present climate of chaos in their lives any number of unsavory scenarios had occurred to Castiel as possibilities for their absence - but Dean has forbidden Castiel to go to him, so he tries to comply with Dean's wishes for once and stays. That doesn't mean it's easy. It doesn't mean the combination of his own intense guilt, his feeling of helplessness, and general ill-ease about Castiel's fight with Dean has made it easy. By the time he hears the impala pull up outside he's worked himself back into irritation again, almost anger, and so when Sam and Dean finally make their way in the door the first thing that comes out of Castiel's mouth is a very gruff and confrontational "Where were you?"