This is what happens:

notepad.exe wants to create a window, but Windows blocks that because services don't have a desktop to create windows on. So notepad.exe sits there, idling, waiting for an API call to return that will never return. Your CGI has started notepad.exe via system, so it sits there and waits for notepad.exe to terminate. It won't, so your CGI won't, either. After some time, the webserver decides that your CGI is stuck, cuts all possible connections to that CGI and returns the error page to your browser.

This is what you are doing wrong:

You are starting a program designed to run on a desktop, showing a graphical user interface, from a service.

Alexander