I've tried to navigate to the folder and delete the perl 5.28.0 folder but that doesn't make a difference.

To complete the deinstallation, remove the perlbrew-related lines from $HOME/.bashrc, and for good measure check the directory paths contained in the environment variables PERLBREW_HOME and PERLBREW_ROOT and delete those if existing.

Then log out. Afterwards type perlbrew should show neither a filename path nor a bash function.