Concerned? Hardly. [Yes. Yes, just a little.]

I'm merely concerned that you'll soon be problematic with your memory loss. If you don't remember the last time we interacted, you may return to your foolish behavior and I'll be forced to deal with it.

[And he really doesn't want to deal with something stupid. But that's just the secondary concern.]