That is already one of the measures we use to protect the access to the equipments.
However, as we are a spread out organization, we also allow access to a limited "test" ip subnet in every remote location, for when we need to perform on-site interventions. And of course any of our old team members know what that ip subrange is...
Of course you could call us all paranoid, but we are facing some close attention from our security department and wouldn't want to let any loose ends.
As we are now implementing COM, I think one of the ways could be to restrict snmp access to the COM server, and every team member has to access the equipments through COM/EDM. I am still testing how that works with older equipment.