Well, in the begining I did a restart, but then I stopped it completely ... No change.
We patched the code to force the cleanup, because setting up the cleanup delay wouldn't work. It's very strange.
But I have 2 webservers ( listenning different ports ) on the same box. Would there be a possibility that it is because the cache is stored beetween to independant copys of apache? I don't believe so, but who knows?