Why doesn't Lester defend Despereaux?

Desperaux has just too many differences from the "typical" mouse. He's small, his ears are too big, he was born with his eyes open, he hears music, fall in love with a human, allows one to touch him, and sits at the foot of a king. He evens speaks to humans, doesn't scurry about, or search for crumbs. Though his father doesn't speak up for him, he does love his son. Lester is outnumbered by the other mice, and he knows his son's behavior could threaten the welfare of all mice.