Done some more tests and it might be nginx causing problems after all.
sudo -u nginx /usr/local/bin/plackup -E production -s Starman --workers=2 --port 3000 --preload-app -a bin/app.pl
Having starman listen on port 3000, accessing it directly without going through nginx works fine.