Perhaps he's deliberately misinterpreting. "If I didn't say anything, you might have thought I hadn't cared to be there."

Perhaps if one left off those last three words they'd get to the heart of the matter.