in reply to Algorithom to find overlaping subnets (Internet IPv4)

An interesting question is what exactly do you mean by overlap?

For example: for some applications, a subnet that is entirely contained by another:

subnet 1:         S..........E
subnet 2:            S.....E

[download]

Can easily be done away with entirely.

But subnets that overlaps but not completely:

subnet 1:         S..........E
subnet 2:                  S.....E
subnet 3:         S.............E
subnet 4:    S......................E

[download]

Will rarely be able to be coalesced directly into a single subnet (#3), as the 'nearest' subnet that would contain both (#4) will usually also contain addresses not contained in the original set.

And given 50_000 inputs, the likely scenario -- in the absence of more specificity regarding the distribution of the subnets -- is that they will form a tree with a few large, 'root' level subnets each containing a hierarchy of smaller subnets:

   s...................................e s......e         s...........
+....e
     s...............e      s........e     s.e s...........e   s.e s..
+...e
    s......e s....e 
        s......e

[download]

That suggests a strategy whereby instead of sorting the subnets by start/end address, you should sort them by subnet size. The first (largest) therefore will not be contained by any of the others, so can be removed from the list, and used as the root of a tree. It may of course, overlap with one or more of the next few largest, but except for the rare event where the two can be combined into a single, unextended subnet, they will still be roots of their own subtrees.

So my suggestion would be to pick off the biggest ones and remove them from the list very quickly. You can then distribute the rest as subordinate to one (or more) of the roots you picked out. You can then (recursively) process each of those lists, to further divide their lists into smaller third level lists below a few second-level subroots. Rinse and repeat.

Subnets entirely contained within a higher level can be easily discarded.

The initial sorting by subnet size is very fast. And the first level of recursion very quickly splits the dataset into several or many small subsets that are quickly processed at each new level of recursion.

I might have posted code, but I found that testing such is very hard in the absence of a real dataset. Randomly generated datasets are just too random to give meaningful results.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: Algorithom to find overlaping subnets (Internet IPv4)
by chrestomanci (Priest) on Sep 20, 2011 at 08:17 UTC

    Thank you for your input, however I think you misunderstand the nature of the problem.

    By 'Overlapping subnets' I actually mean one subnet that is entirely within another. It is not possible for a subnet to partially overlap another because we are expressing them in CDIR notation, rather than arbitrary ranges. This makes the problem simpler than you allowed for in your analysis.

    For example, consider the subnet x.y.0.0/16 There are exactly 2 /17 subnets that might fit inside it (x.y.0.0/17 and x.y.128.0/17) and a greater number of smaller subnets. There might also be larger subnets that contain it, but it is impossible to define a subnet (in CDIR notation) that includes some of the address space covered by x.y.0.0/16, and some address space that is not covered.

    You also talk about merging overlapping subnets. This is not what I am trying to do. The end purpose is to produce a reports of all the overlaps which will be used by the network infrastructure people to reconfigure the routers and DNS/DHCP servers so that the subnets no longer overlap.

[reply]
      There might also be larger subnets that contain it, but it is impossible to define a subnet (in CDIR notation) that includes some of the address space covered by x.y.0.0/16, and some address space that is not covered.

      Indeed. I wasn't aware of that property of CIDRs, though I now see it is obvious.

      You also talk about merging overlapping subnets. This is not what I am trying to do. The end purpose is to produce a reports of all the overlaps which will be used by the network infrastructure people to reconfigure the routers and DNS/DHCP servers so that the subnets no longer overlap.

      When the infrastructure people get their hands on your report, won't one of the things they might do be to consolidate (say) 0.0.0.0/30 & 0.0.0.4/30 into 0.0.0.0/29 thus reducing router table sizes?

      Or dropping this lot:

      0.40.0.0/15 
0.40.0.0/16 
0.47.0.0/16 
0.42.0.0/18 
0.44.128.0/18 
0.47.192.0/18 
0.43.0.0/19 
0.47.64.0/21 
0.47.192.0/21

      [download]

      Because they are all already covered by: 0.40.0.0/13? (ie."coalescing" them.)

      Anyway, thanks for posting an interesting question. I guess if you don't find Re: Algorithom to find overlaping subnets (Internet IPv4) useful, someone else might :)

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom