Keeping the addresses in a list makes it optimal for round robin, which is something I like to have.
Indexing the set ids in a tree is useful when one has lots of groups, but for addresses might add penalties to some very used algorithms. Also, I don't know if anyone has lots of addresses inside same group to make it worth on indexing with a different structure.