Why did Harper Lee choose the name Jem for Scout's brother?

I think for the most accurate answer on that one, you'd have to ask Harper Lee yourself. I've searched through 4 different background reads to find purpose for the name, and these are some of the best guesses I have compiled.

Jem is short for 'Jeremy' who had a full name of Jeremy Atticus Finch. There is much symbolism in names for authors. It would be interesting for you to do a word study on the bird, the finch. The characteristics of a finch and a mockingbird are something that she subtley put into the storyline.

Atticus as a middle name mirrors the fact that Jem looks up to his father dearly but has all the same power struggles as normal boys approaching puberty do. Jem, fortunately, maintains respect for his father.

The meaning of Jeremy is "God will raise up" or "God will set free." Not knowing if she for sure took care to cast the role of an older brother with these attributes, I certainly believe they fit. Jem struggles to see freedom from poverty and race for others and he is not yet fully adult. Thus, God still has time to raise him up to be this man following in his father's footsteps and furthering the cause.

Jem is a likely a variety of Harper herself as is Scout. But she could only embody so much in one person. If as a child she had these wonderings about race and people and society, it is possible that Jem is a reflection of her older self, the one who published a book like this near the dawn of the Civil Rights Movement.

Lastly, Jem as a character represents bravery. Every piece of criticism I read on him relates to that. If you look through the book, his activity all circles round being brave enough to touch the house, dealing with the issue of courage with Mrs. Dubose, and keeping calm when people do and say mean things. I think Scout as a character considered her older brother a Jem (gem). He was the one who in the end got in the way of one who would have killed them both. Although Boo finished the job, having another kid there for Ewell to go after meant her life was spared.