This problem has been well-studied, as you might imagine. The optimal lineup varies depending on the game situation. (For example, if the home team is down by one run in the bottom of the ninth inning, they are no longer interested in the strategy which gives them the largest expected number of runs over the long term, but in the strategy that gives them the greatest probability of scoring two runs in the rest of the inning.) But in general, and over the long term, the answer is simple: To maximize the expected number of runs scored, the batters should appear in decreasing order of their on-base average. The player with the highest on-base average should appear first. I will explain why this is is a little further down.

On-base average is like batting average, but it is calculated differently---it's simpler. When a batter comes to the plate, that is a 'plate appearance'. On-base average is the fraction of a batter's plate appearances in which he reaches base on a hit, a walk, or a hit-by-pitch. (Your brother's league probably doesn't play with hit-by-pitch, so you can ignore this; they may not play with walks either, in which case you can ignore that too.)

Batting average is different: It's hits divided by at-bats, and not every plate appearance is an at-bat. Certain plate appearances do not count as at-bats. In particular, if the batter walks, that is not an at-bat. The batter did not get a hit, but his batting average is unchanged, because he wasn't offered a fair chance to get a hit.

For on-base average, you count all plate appearances, and hits and walks both count positively.

The reason on-base average is important is this: The team gets only 27 outs in the game. The outs are like a clock that is ticking away. Once they use up their 27 outs, the game is over. The more batters your team can send to the plate in 27 outs, the more likely the team is to score and win. On-base average is precisely the chance that a batter will go to the plate without producing an out. If the batters on the team have on-base averages of .250, the team sends an average of 4 batters to the plate each inning. If the batters on the team have on-base averages of .350, the team sends an average of 4.6 batters to the plate each inning. Compared with the .250 team, they are getting a free inning!

If only two consecutive batters get on base in an inning, they will probably not score. If four consecutive batters get on base in an inning, at least one will certainly score. Clustering together the batters with a high on-base average maximizes the likelihood of a long inning and therefore a high-scoring inning.

By the way, you don't want to use the word 'rotation' here. That refers to pitching staffs. The word you want is either 'lineup' or 'batting order'.

--
Mark Dominus
Perl Paraphernalia