in reply to Re^9: Should I just print my own HTTP headers?
in thread Should I just print my own HTTP headers?
Just for grins, I thought I'd have a go a seeing how the benchmark figures, as well as the extrapolations from them, stood up to a "real world" test. The following shows a the results of hitting the same, lightweight preforking server serving similar dynamic pages via CGI.pm, or a simplistic, 'raw cgi' script:
10 concurrent users making 100 requests each:
c:\test>608420-b.plt -REQUESTS=100 -THREADS=10 -MODE=raw 10 threads running 100 trials of Mode:raw thread: 2 (23.386s total), 233.863ms/trial 100 trials of Mode:raw thread: 3 (23.495s total), 234.949ms/trial 100 trials of Mode:raw thread: 1 (23.670s total), 236.701ms/trial 7 threads running 100 trials of Mode:raw thread: 7 (23.491s total), 234.911ms/trial 100 trials of Mode:raw thread: 6 (23.697s total), 236.970ms/trial 100 trials of Mode:raw thread: 8 (23.482s total), 234.823ms/trial 100 trials of Mode:raw thread: 10 (23.323s total), 233.227ms/trial 100 trials of Mode:raw thread: 9 (23.424s total), 234.242ms/trial 100 trials of Mode:raw thread: 5 (23.901s total), 239.014ms/trial 100 trials of Mode:raw thread: 4 (23.953s total), 239.527ms/trial c:\test>608420-b.plt -REQUESTS=100 -THREADS=10 -MODE=cgi 10 threads running 100 trials of Mode:cgi thread: 6 (80.614s total), 806.141ms/trial 9 threads running 100 trials of Mode:cgi thread: 1 (82.629s total), 826.292ms/trial 100 trials of Mode:cgi thread: 10 (81.855s total), 818.548ms/trial 100 trials of Mode:cgi thread: 2 (82.535s total), 825.349ms/trial 6 threads running 100 trials of Mode:cgi thread: 8 (83.137s total), 831.375ms/trial 100 trials of Mode:cgi thread: 9 (83.092s total), 830.920ms/trial 4 threads running 100 trials of Mode:cgi thread: 4 (84.023s total), 840.234ms/trial 100 trials of Mode:cgi thread: 7 (83.910s total), 839.102ms/trial 100 trials of Mode:cgi thread: 5 (84.161s total), 841.614ms/trial 100 trials of Mode:cgi thread: 3 (84.292s total), 842.917ms/trial
100 concurrent users making 10 requests each
c:\test>608420-b.plt -REQUESTS=10 -THREADS=100 -MODE=raw 100 threads running 10 trials of Mode:raw thread: 6 (34.110s total), 3.411s/trial 10 trials of Mode:raw thread: 7 (33.922s total), 3.392s/trial 10 trials of Mode:raw thread: 9 (33.703s total), 3.370s/trial 10 trials of Mode:raw thread: 4 (34.219s total), 3.422s/trial 96 threads running 10 trials of Mode:raw thread: 33 (32.078s total), 3.208s/trial 10 trials of Mode:raw thread: 28 (32.500s total), 3.250s/trial 10 trials of Mode:raw thread: 89 (27.578s total), 2.758s/trial 10 trials of Mode:raw thread: 94 (27.203s total), 2.720s/trial 10 trials of Mode:raw thread: 20 (33.031s total), 3.303s/trial 10 trials of Mode:raw thread: 5 (34.634s total), 3.463s/trial 10 trials of Mode:raw thread: 90 (27.484s total), 2.748s/trial 10 trials of Mode:raw thread: 14 (33.718s total), 3.372s/trial 10 trials of Mode:raw thread: 88 (27.625s total), 2.762s/trial 10 trials of Mode:raw thread: 76 (28.656s total), 2.866s/trial 10 trials of Mode:raw thread: 11 (34.000s total), 3.400s/trial 10 trials of Mode:raw thread: 82 (28.156s total), 2.816s/trial 10 trials of Mode:raw thread: 17 (33.203s total), 3.320s/trial 10 trials of Mode:raw thread: 16 (33.390s total), 3.339s/trial 10 trials of Mode:raw thread: 19 (33.078s total), 3.308s/trial 10 trials of Mode:raw thread: 79 (28.406s total), 2.841s/trial 10 trials of Mode:raw thread: 58 (30.297s total), 3.030s/trial 10 trials of Mode:raw thread: 3 (34.757s total), 3.476s/trial 10 trials of Mode:raw thread: 64 (29.750s total), 2.975s/trial 10 trials of Mode:raw thread: 92 (27.281s total), 2.728s/trial 10 trials of Mode:raw thread: 77 (28.609s total), 2.861s/trial 10 trials of Mode:raw thread: 26 (32.640s total), 3.264s/trial 10 trials of Mode:raw thread: 96 (26.922s total), 2.692s/trial 10 trials of Mode:raw thread: 54 (30.562s total), 3.056s/trial 10 trials of Mode:raw thread: 71 (29.109s total), 2.911s/trial 10 trials of Mode:raw thread: 31 (32.203s total), 3.220s/trial 10 trials of Mode:raw thread: 52 (30.750s total), 3.075s/trial 10 trials of Mode:raw thread: 35 (31.921s total), 3.192s/trial 10 trials of Mode:raw thread: 60 (30.140s total), 3.014s/trial 10 trials of Mode:raw thread: 85 (27.859s total), 2.786s/trial 10 trials of Mode:raw thread: 27 (32.547s total), 3.255s/trial 10 trials of Mode:raw thread: 8 (34.390s total), 3.439s/trial 10 trials of Mode:raw thread: 99 (26.750s total), 2.675s/trial 10 trials of Mode:raw thread: 50 (30.859s total), 3.086s/trial 10 trials of Mode:raw thread: 72 (28.953s total), 2.895s/trial 10 trials of Mode:raw thread: 97 (27.093s total), 2.709s/trial 10 trials of Mode:raw thread: 65 (29.656s total), 2.966s/trial 10 trials of Mode:raw thread: 63 (29.812s total), 2.981s/trial 10 trials of Mode:raw thread: 10 (34.047s total), 3.405s/trial 10 trials of Mode:raw thread: 73 (28.906s total), 2.891s/trial 10 trials of Mode:raw thread: 74 (28.765s total), 2.877s/trial 10 trials of Mode:raw thread: 44 (31.343s total), 3.134s/trial 10 trials of Mode:raw thread: 49 (30.890s total), 3.089s/trial 10 trials of Mode:raw thread: 98 (26.797s total), 2.680s/trial 10 trials of Mode:raw thread: 2 (34.737s total), 3.474s/trial 10 trials of Mode:raw thread: 42 (31.468s total), 3.147s/trial 10 trials of Mode:raw thread: 45 (31.156s total), 3.116s/trial 10 trials of Mode:raw thread: 48 (31.000s total), 3.100s/trial 10 trials of Mode:raw thread: 87 (27.734s total), 2.773s/trial 10 trials of Mode:raw thread: 15 (33.609s total), 3.361s/trial 10 trials of Mode:raw thread: 21 (32.953s total), 3.295s/trial 10 trials of Mode:raw thread: 78 (28.515s total), 2.852s/trial 10 trials of Mode:raw thread: 91 (27.453s total), 2.745s/trial 10 trials of Mode:raw thread: 62 (29.922s total), 2.992s/trial 10 trials of Mode:raw thread: 69 (29.328s total), 2.933s/trial 10 trials of Mode:raw thread: 1 (34.877s total), 3.488s/trial 10 trials of Mode:raw thread: 95 (27.187s total), 2.719s/trial 10 trials of Mode:raw thread: 51 (30.734s total), 3.073s/trial 10 trials of Mode:raw thread: 38 (31.750s total), 3.175s/trial 10 trials of Mode:raw thread: 61 (29.953s total), 2.995s/trial 10 trials of Mode:raw thread: 37 (31.797s total), 3.180s/trial 10 trials of Mode:raw thread: 66 (29.609s total), 2.961s/trial 10 trials of Mode:raw thread: 84 (27.890s total), 2.789s/trial 10 trials of Mode:raw thread: 12 (33.906s total), 3.391s/trial 10 trials of Mode:raw thread: 29 (32.406s total), 3.241s/trial 10 trials of Mode:raw thread: 53 (30.625s total), 3.062s/trial 10 trials of Mode:raw thread: 34 (32.047s total), 3.205s/trial 10 trials of Mode:raw thread: 32 (32.187s total), 3.219s/trial 10 trials of Mode:raw thread: 18 (33.172s total), 3.317s/trial 10 trials of Mode:raw thread: 36 (31.890s total), 3.189s/trial 10 trials of Mode:raw thread: 81 (28.281s total), 2.828s/trial 10 trials of Mode:raw thread: 70 (29.093s total), 2.909s/trial 10 trials of Mode:raw thread: 30 (32.328s total), 3.233s/trial 10 trials of Mode:raw thread: 22 (32.922s total), 3.292s/trial 10 trials of Mode:raw thread: 59 (30.093s total), 3.009s/trial 10 trials of Mode:raw thread: 25 (32.687s total), 3.269s/trial 10 trials of Mode:raw thread: 86 (27.765s total), 2.777s/trial 10 trials of Mode:raw thread: 24 (32.781s total), 3.278s/trial 10 trials of Mode:raw thread: 41 (31.500s total), 3.150s/trial 10 trials of Mode:raw thread: 43 (31.375s total), 3.137s/trial 10 trials of Mode:raw thread: 39 (31.625s total), 3.162s/trial 10 trials of Mode:raw thread: 23 (32.828s total), 3.283s/trial 10 trials of Mode:raw thread: 80 (28.484s total), 2.848s/trial 10 trials of Mode:raw thread: 40 (31.828s total), 3.183s/trial 10 trials of Mode:raw thread: 56 (30.640s total), 3.064s/trial 10 trials of Mode:raw thread: 55 (30.734s total), 3.073s/trial 10 trials of Mode:raw thread: 46 (31.437s total), 3.144s/trial 10 trials of Mode:raw thread: 13 (34.031s total), 3.403s/trial 10 trials of Mode:raw thread: 47 (31.297s total), 3.130s/trial 10 trials of Mode:raw thread: 67 (29.703s total), 2.970s/trial 10 trials of Mode:raw thread: 93 (27.500s total), 2.750s/trial 10 trials of Mode:raw thread:100 (26.890s total), 2.689s/trial 10 trials of Mode:raw thread: 57 (30.625s total), 3.062s/trial 10 trials of Mode:raw thread: 75 (29.099s total), 2.910s/trial 10 trials of Mode:raw thread: 83 (28.264s total), 2.826s/trial 10 trials of Mode:raw thread: 68 (29.709s total), 2.971s/trial c:\test>608420-b.plt -REQUESTS=10 -THREADS=100 -MODE=cgi 100 threads running 10 trials of Mode:cgi thread: 77 (86.047s total), 8.605s/trial 10 trials of Mode:cgi thread: 28 (89.797s total), 8.980s/trial 10 trials of Mode:cgi thread: 64 (87.093s total), 8.709s/trial 10 trials of Mode:cgi thread: 49 (88.297s total), 8.830s/trial 96 threads running 10 trials of Mode:cgi thread: 48 (88.984s total), 8.898s/trial 10 trials of Mode:cgi thread: 46 (89.156s total), 8.916s/trial 10 trials of Mode:cgi thread: 79 (86.547s total), 8.655s/trial 93 threads running 10 trials of Mode:cgi thread: 35 (92.234s total), 9.223s/trial 10 trials of Mode:cgi thread: 34 (92.328s total), 9.233s/trial 10 trials of Mode:cgi thread: 96 (87.093s total), 8.709s/trial 10 trials of Mode:cgi thread: 99 (86.922s total), 8.692s/trial 10 trials of Mode:cgi thread: 60 (90.375s total), 9.037s/trial 10 trials of Mode:cgi thread: 61 (90.316s total), 9.032s/trial 10 trials of Mode:cgi thread: 23 (93.093s total), 9.309s/trial 10 trials of Mode:cgi thread: 76 (89.015s total), 8.902s/trial 10 trials of Mode:cgi thread: 36 (92.093s total), 9.209s/trial 10 trials of Mode:cgi thread: 39 (92.000s total), 9.200s/trial 10 trials of Mode:cgi thread: 13 (93.781s total), 9.378s/trial 10 trials of Mode:cgi thread: 89 (87.687s total), 8.769s/trial 10 trials of Mode:cgi thread: 84 (88.172s total), 8.817s/trial 10 trials of Mode:cgi thread: 45 (91.515s total), 9.152s/trial 10 trials of Mode:cgi thread: 78 (89.078s total), 8.908s/trial 10 trials of Mode:cgi thread: 47 (91.390s total), 9.139s/trial 10 trials of Mode:cgi thread: 69 (89.656s total), 8.966s/trial 10 trials of Mode:cgi thread: 86 (87.906s total), 8.791s/trial 10 trials of Mode:cgi thread: 90 (87.484s total), 8.748s/trial 10 trials of Mode:cgi thread: 98 (86.953s total), 8.695s/trial 10 trials of Mode:cgi thread: 51 (91.156s total), 9.116s/trial 10 trials of Mode:cgi thread: 50 (91.125s total), 9.112s/trial 10 trials of Mode:cgi thread: 15 (93.625s total), 9.362s/trial 10 trials of Mode:cgi thread: 38 (91.968s total), 9.197s/trial 10 trials of Mode:cgi thread: 5 (94.618s total), 9.462s/trial 10 trials of Mode:cgi thread: 37 (92.156s total), 9.216s/trial 10 trials of Mode:cgi thread: 4 (94.776s total), 9.478s/trial 10 trials of Mode:cgi thread: 75 (88.953s total), 8.895s/trial 10 trials of Mode:cgi thread: 24 (93.078s total), 9.308s/trial 10 trials of Mode:cgi thread: 21 (93.234s total), 9.323s/trial 10 trials of Mode:cgi thread: 72 (89.359s total), 8.936s/trial 10 trials of Mode:cgi thread: 85 (88.156s total), 8.816s/trial 10 trials of Mode:cgi thread: 65 (90.078s total), 9.008s/trial 10 trials of Mode:cgi thread: 2 (94.913s total), 9.491s/trial 10 trials of Mode:cgi thread: 73 (89.390s total), 8.939s/trial 10 trials of Mode:cgi thread: 56 (90.672s total), 9.067s/trial 10 trials of Mode:cgi thread: 26 (92.937s total), 9.294s/trial 10 trials of Mode:cgi thread: 3 (94.895s total), 9.490s/trial 10 trials of Mode:cgi thread: 40 (91.682s total), 9.168s/trial 10 trials of Mode:cgi thread: 25 (92.984s total), 9.298s/trial 10 trials of Mode:cgi thread: 19 (93.343s total), 9.334s/trial 10 trials of Mode:cgi thread: 74 (88.984s total), 8.898s/trial 10 trials of Mode:cgi thread: 88 (87.734s total), 8.773s/trial 10 trials of Mode:cgi thread: 71 (89.515s total), 8.952s/trial 10 trials of Mode:cgi thread: 1 (95.035s total), 9.504s/trial 10 trials of Mode:cgi thread: 62 (90.234s total), 9.023s/trial 10 trials of Mode:cgi thread: 92 (87.375s total), 8.737s/trial 10 trials of Mode:cgi thread: 42 (91.687s total), 9.169s/trial 10 trials of Mode:cgi thread: 44 (91.562s total), 9.156s/trial 10 trials of Mode:cgi thread: 53 (91.031s total), 9.103s/trial 10 trials of Mode:cgi thread: 27 (92.781s total), 9.278s/trial 10 trials of Mode:cgi thread: 58 (90.531s total), 9.053s/trial 41 threads running 10 trials of Mode:cgi thread: 32 (93.203s total), 9.320s/trial 10 trials of Mode:cgi thread: 31 (93.172s total), 9.317s/trial 10 trials of Mode:cgi thread: 33 (93.047s total), 9.305s/trial 10 trials of Mode:cgi thread: 68 (90.530s total), 9.053s/trial 10 trials of Mode:cgi thread:100 (87.547s total), 8.755s/trial 10 trials of Mode:cgi thread: 6 (95.359s total), 9.536s/trial 10 trials of Mode:cgi thread: 93 (88.015s total), 8.802s/trial 10 trials of Mode:cgi thread: 54 (91.531s total), 9.153s/trial 10 trials of Mode:cgi thread: 9 (95.093s total), 9.509s/trial 10 trials of Mode:cgi thread: 57 (91.484s total), 9.148s/trial 10 trials of Mode:cgi thread: 17 (94.265s total), 9.427s/trial 10 trials of Mode:cgi thread: 66 (90.765s total), 9.077s/trial 10 trials of Mode:cgi thread: 11 (94.890s total), 9.489s/trial 10 trials of Mode:cgi thread: 10 (94.984s total), 9.498s/trial 10 trials of Mode:cgi thread: 16 (94.375s total), 9.437s/trial 10 trials of Mode:cgi thread: 94 (88.000s total), 8.800s/trial 10 trials of Mode:cgi thread: 14 (94.672s total), 9.467s/trial 10 trials of Mode:cgi thread: 95 (88.031s total), 8.803s/trial 10 trials of Mode:cgi thread: 8 (95.250s total), 9.525s/trial 10 trials of Mode:cgi thread: 22 (93.968s total), 9.397s/trial 10 trials of Mode:cgi thread: 18 (94.234s total), 9.423s/trial 10 trials of Mode:cgi thread: 91 (88.312s total), 8.831s/trial 10 trials of Mode:cgi thread: 63 (91.062s total), 9.106s/trial 10 trials of Mode:cgi thread: 7 (95.297s total), 9.530s/trial 10 trials of Mode:cgi thread: 55 (91.687s total), 9.169s/trial 10 trials of Mode:cgi thread: 82 (89.218s total), 8.922s/trial 10 trials of Mode:cgi thread: 43 (92.437s total), 9.244s/trial 10 trials of Mode:cgi thread: 12 (94.828s total), 9.483s/trial 10 trials of Mode:cgi thread: 52 (91.781s total), 9.178s/trial 10 trials of Mode:cgi thread: 30 (93.453s total), 9.345s/trial 10 trials of Mode:cgi thread: 59 (91.303s total), 9.130s/trial 10 trials of Mode:cgi thread: 83 (89.250s total), 8.925s/trial 10 trials of Mode:cgi thread: 81 (89.390s total), 8.939s/trial 10 trials of Mode:cgi thread: 29 (93.437s total), 9.344s/trial 10 trials of Mode:cgi thread: 20 (94.125s total), 9.412s/trial 10 trials of Mode:cgi thread: 70 (90.437s total), 9.044s/trial 10 trials of Mode:cgi thread: 87 (88.687s total), 8.869s/trial 10 trials of Mode:cgi thread: 80 (89.593s total), 8.959s/trial 10 trials of Mode:cgi thread: 41 (92.640s total), 9.264s/trial 10 trials of Mode:cgi thread: 97 (87.902s total), 8.790s/trial 10 trials of Mode:cgi thread: 67 (90.651s total), 9.065s/trial
The signifciance of these numbers? Probably very little as the users were running in the same box as the server and networking effects are eliminated, but it may serve to show that whilst the relativistic headline percentages produced by Benchmark serve to inflate differences, so extrapolations of the wallclock timings are equally flawed.
The only difference between the raw and cgi tests is the loading and use of the CGI.pm module. Whilst with 10 concurrent users the 3x difference in the delivery time, from ~250ms to ~850ms (excluding network latency), is neither here nor there. Once you get 100 users concurrent, the increase from ~3 seconds to ~9 seconds (exclusive) is likely to begin to impact the users perceptions of the website.
Is this significant enough to consider hand rolling CGIs with the risks that entails? Probably not. But it might be enough to consider looking elsewhere than CGI.pm, say CGI::Simple?
I'll make my test scripts available to anyone who's set up to run the tests under more real world conditions
|
|---|