Restrict ability to add users to a project Follow

Is it possible to restrict the ability to add users to a project so that, for example, the project admin could add users only from a specific group or organization? Or give rules to edit a project without the ability to add users to this project?