Because there's only one space after the number, which is consumed by the character class, but there's no more spaces to be matched by the following \s+.

You can just say "optional dot" instead, i.e.

[.]?
[download]

Working example:

#!/usr/bin/perl
use warnings;
use strict;
use experimental qw( signatures );

sub match($entry) {
    $entry =~ m{^ (\d+) [.]? \s+ (.*?) \s+(-|\N{EN DASH}|\N{EM DASH})\
+s+ (.*?) $}x;
}

use Test::More tests => 3;

ok match('123 Festival - Sunday');
ok match('456. Pool Party - Tuesday');
ok ! match('789| Gig - Friday');
[download]

Moreover, the | matches itself in a character class which is not what you want (try the last test against the original regex).