Sorry to reply to myself,
OK, I've started looking at this but there seems to be a basic, pretty much unsovelable problem. In the case of an invite we don't actually know for sure where the other end of the conversation is going to be, since we havent seen the sdp in the OK by that stage.
The only thing I can think of, is to listen on the interface that the INVITE request itself will go out on, this isn't the same thing obviously. It will work for the setup I have, infact I already make similar assumptions when deciding whether or not to rtp proxy in the first place.
Any thaughts?