Is this useful?

Maybe - at least it is lightweight and it is pure Perl. A quick skim through CPAN revealed no direct duplicate, but I guess these functions are already made available by some of the math-library-wrappers, e.g Math::GSL.
Ah, there they are: Math::GSL::Randist, Math::GSL::CDF.

Since implementation of numerical algorithms is quite challenging, I would rather rely on the libraries that have been around for many years. At least, your module comes with some test cases that provide some initial trust...

Is anything missing?

Documentation: Instead of

$p = pdf( $x, $m, $s );
[download]

I would rather write

$prob_density = pdf( $x, $mean, $std_dev );
[download]

Maybe an additional alternative interface could be made available:

$prob_density = pdf( { x => $x, mean => $mean, std_dev => $std_dev } )
+;
$prob_density = pdf( { z => $z } ) / $std_dev;
[download]

Using a hashref allows to extend the current interface easily and backward compatible. Iff you are willing to always use named parameters, the potential pitfall to forget using an anonymous hash could be eliminated. Performance might suffer a little, though.

Import: use Math::Gauss qw(:all); makes IMHO only sense if you also support an OO-interface - which you currently don't. But maybe an OO-extension is already planned?

OO-Interface: An OO-extension could be useful since $mean and $std_dev are usually constant (parameters of the distribution) for a given computation-run ($pdf1 = Math::Gauss->new( mean => $mean, std_dev => $std_dev, err => 1e-5 );.
But then, a conflict occurs when $pdf1->pdf( $x ) becomes indistinguishable from $pdf1->pdf( $z ) which support the argument for named parameters ($pdf1->( x => $x ), $pdf1->( z => $z) ).

As can be seen from the example above, it would be nice if the error limit could be set individually, e.g. for speed/accuracy trade-offs.

Is the name well-chosen?

Maybe not. Taking into account that C. F. Gauss has covered a wide area in the field of mathematics, Math::Gauss appears too general. Something like Math::Gauss::Distributions might be a better choice, leaving room for Math::Gauss::Primes, Math::Gauss::Elimination, Math::Gauss::EasterDate, etc. - but most of these algorithms are already available under different name spaces.

The Math::Gauss::* name space also appears to Gauss-centric (same would apply to Newton, Leibnitz, Taylor, Fibonacci, Euler, etc.) - so Math::Stat::Gauss could fit? However Math::Stat already exists and there could be a conflict in expectations concerning API usage.

So I would plead for Math::Distributions::Gauss, leaving name-space for Math::Distributions::[FisherZ|Rayleigh|Laplace|StudentT|...]...

I hope, these remarks were helpful and not too discouraging. The module covers your design criteria quite well. A lightweight well tested pure Perl implementation could be useful in some situations while the hard-core number crunching Monk would resort to the established libraries.