The nginx config had no cache-specific setting.
I've tried setting the header, also setting proxy_no_cache and proxy_cache_bypass in the nginx config with same results.
I guess it can't only be nginx caching alone or I would have the same behavior with 1 starman worker.
It's the combination of whatever nginx is doing and multiple starman workers.