Usual there is no need to physically access a server, except for exceptional reasons like, for example, hardware maintenance. Software configuration can be and should be performed remotely.
However, this remote access must be safe and secure.