Re^3: Imploding URLs

So though I have a large collection of URLs (from my logs), I need to "implode" URLs on a one-by-one basis.

Why? Is the space savings that significant?

If all you have is a handful of substitutions, you can probably hand-pick the strings:

http   www.   .com  .org  .net
://    index  .htm  .jpg  .gif
google yahoo  mail  news  ebay
[download]

Comment on Re^3: Imploding URLs Download Code

Replies are listed 'Best First'.
Re^4: Imploding URLs by mobiGeek (Beadle) on Jun 11, 2005 at 03:54 UTC
Yes, the savings is quite significant. From a hand-selected list of one user's habits, I was able to reduce some pages by more than 20%. The other thing is that this is not a collection of URLs from across the entire web. The URLs being crawled vary, but the proxy is part of a kind of "portal". So there are potentially thousands of URLs, but they come from a select list of sites. Thus the reason I am looking for the weighting of substrings. If one URL or one particular site (i.e. a particular substring) is crawled extremely frequently, then imploding that string might be much more bandwidth saving than simply imploding "http://" on all URLs. mG.	[reply]

Replies are listed 'Best First'.

Re^4: Imploding URLs
by mobiGeek (Beadle) on Jun 11, 2005 at 03:54 UTC

The other thing is that this is not a collection of URLs from across the entire web. The URLs being crawled vary, but the proxy is part of a kind of "portal". So there are potentially thousands of URLs, but they come from a select list of sites. Thus the reason I am looking for the weighting of substrings.

If one URL or one particular site (i.e. a particular substring) is crawled extremely frequently, then imploding that string might be much more bandwidth saving than simply imploding "http://" on all URLs.

mG.

[reply]