If you've defined two explicit cases for $inner (scalar or array ref), you can always differentiate it from a hash ref, so why not just make the \%attr optional and just omit the empty hashes {}?

[ family => { name => 'Kawasaki' },
  [
    [father => 'Yasushisa' ],
    [mother => 'Chizuko' ],
    [children =>
      [
        [girl => 'Shiori'],
        [boy  => 'Yasuke'],
        [boy  => 'Kairi']
      ]
    ]
  ]
]
[download]

I also think your example had a couple minor errors (extra array ref and no punctuation between 'children' and its children.)