Active ports may well respond more quickly than that, but for inactive ports NMap will have to wait a certain amount of time before it times the port out and concludes it is not active.

Closed ports are also reported immediately, unless they're blocked by a firewall. Stop portscanning people's firewalls! What's wrong with you?