Just a try: when going sequentially through the refreshed list of groups, check your objects for each group you go through, updating the info on existing objects or adding new objects for new groups; "tag" every group added or updated so you know you have visited it for this refresh.

When you reach the end of the refresh list, now go through all your objects and drop those which have no "visited" tag (meaning they are no longer in the refreshed list and presumably were removed). At the same time "reset" the visited tag for next time around.

You thus go once through the refreshed list and once through your object list: seems not too "expensive" to me.