You're assuming that all leaves must be at the same, known depth (5).

If that's a valid assumption, natatime from List::MoreUtils can do the "matrification".