decebel has asked for the wisdom of the Perl Monks concerning the following question:

Hi All,

I am not sure if this is right place to post this question as I posted one already on another section. Giving myself benefit of doubt, and a first timer on this excellent website for perl folks, I will ask again !!

I am trying to perform some functional testing on a particular website.

Basically, users can visit the website from different countries. And based upon the location of the user, the site displays relevant information. The webserver uses an external software that does the client IP to country mapping.

My question is, how do I simulate the GET requests to this server from the varied locations/country ? One solution that comes to my mind is to use free http proxies in various countries and write custom code using say LWP module for making the requests routed from the proxies.

The difficulty I am facing with this approach is that the current list of web proxies I could find from google do not work with a simple GET/POST form for submitting the URL to the proxy. So automation is not easy. Instead they work with some additional headers such as the cookies and it gets quite cumbersome figuring out the logic for each of the free proxy's URL form logic. To add to the problem, the free proxies are online only for a few hours.

Did people have had any similar experiences? Any advice/feedback would be very helpful.

Gracias, Decebel

  • Comment on Effective request simulation from various location

Replies are listed 'Best First'.
Re: Effective request simulation from various location
by smiffy (Pilgrim) on Nov 03, 2008 at 01:26 UTC

    Basically, users can visit the website from different countries. And based upon the location of the user, the site displays relevant information.

    That should be "based on the assumed location of the user."

    I would suggest that you consider giving a little more control to the user regarding what is relevant. By all means provide content based on the location of the IP address, but also provide a means to provide alternative content if this is not relevant.

    You need to consider that the IP address that you are seeing may be nowhere near the user - they may be connected through a VPN (for instance, bypassing 'the Great Firewall of China',) running through an anonymising service like tor or through a gateway connecting a private corporate network with the Internet at large. Assume away but let the user have the choice to correct your assumption if they so wish ;-)

    As regards the logic of proxies, this is something with which you should probably familiarise yourself anyway. The environment variable $ENV{'REMOTE_ADDR'} is very often showing you a proxy (especially with ISPs that use transparent proxies, corporate networks again, etc.) rather than the IP address of the user. Which may be in a quite different geographic location from the user, if not in a different country.

    I would suggest that you Google for X-Forwarded-For, the non-RFC-standard header used by many (but by all means not all) proxy servers to identify the 'real' IP of the client. This should be available to you - if set - as $ENV{'X-FORWARDED-FOR'}; note that this might come out as an array, as I believe that the header can have up to three parameters (if I remember correctly). There is another, similar, header that is sometimes used to perform the same function, but I can't remember it off the top of my head. You may come across it reading up on X-Forwarded-For.

    And welcome to Perl Monks :-)

Re: Effective request simulation from various location
by BerntB (Deacon) on Nov 03, 2008 at 00:44 UTC

    I'd personally settle for a mock object for the external code finding the country from the IP. Or script the web side to set the IP address of the GET request.

    But then you can't test it easily from the web side exclusively, of course.

Re: Effective request simulation from various location
by Perlbotics (Archbishop) on Nov 03, 2008 at 21:44 UTC

    Personally, I wouldn't want to make myself dependent from some proxies that are out of my control. The Internet is not a very reproducible test environment for this sort of tests. So the question is: At which level do you want to (or: can you) control your test environment, e.g. application-level or network-level?

    I assume, that you already have at least control over the test generators and the application? Several approaches are conceivable, depending on the degree of influence that you have upon your test environment (HW+SW, SW only) - and budget:

    • HW1: use a test-PC and configure it's network card with the IPs you want to test / arrange proper routing
    • HW2: configure a NAT-capable (network address translation) device in your network (e.g. a load balancer), mapping your current test-PCs IP to the ones desired / if I remember correct, there are NAT devices available that can map a source IP+port to a destination IP+port, so your test client can control the IP address by selecting a destination port
    • SW1: enhance your application to accept an additional (CGI?) attribute, e.g. (__TESTIP=12.34.56.78) that overrides the current IP address value / you might want to disable this feature in a production environment / risk to introduce new errors
    • SW2: Don't know it this really works: Setup some virtual machines with their individual IPs using the same physical NIC (e.g. Solaris Zones)
    HW1+2 have the advantage not to influence your SUT. However, HW1 requires access to the test clients hardware and HW2 is costly in terms of hardware itself and configuration overhead. SW1 should be the cheapest solution but comes with the price of modifying the software under test (SUT). On the other hand, SW1 would allow you to simulate all conceivable IPs (countries) with your client ... a nice set-up if you have to do regression-tests often. Personally, I would vote for SW1.

    For what its worth, many browsers allow to define a preferred language, e.g. my browsers sends an Accept-Language: de,en;q=0.9 with each request... maybe it gives you further clues to present your localised content? IF you want to decide what language to use for display, then I would even consider country detection based on the IP as a fallback when no language preference was given by the client.