From Longman Dictionary of Contemporary Englishbe somebody’s affairbe somebody’s affairPRIVATE/PERSONALif something is your affair, it only concerns you and you do not want anyone else to get involved in it What I do in my free time is my affair. → affair
Examples from the Corpus
be somebody’s affair• What I do in my free time is my affair and nobody else's. 