One way is to add one line "-width=>20" ( as I did below, when you create your buttons. I tested and looked good to me.

      $_->{Button} = $colors_pane->Button
      (-width => 20,#this is what I added
           -text => $_->{Name},
         -background => $_->{Name},
         -foreground => $fg,
       );
      }
[download]

BTW, you better make your pane to have less columns, and with no horizontal scroll bar. It is not convenient for the user to scroll both ways.