So what's so "desktop-ish" about it? Well as it turns out the default configuration accepts connections from localhost only which is completely unusable in a server-like environment with no graphical environment and no web browser whatsoever.
The default behavior also manifests itself in a very non-verbose manner. At first the log entry
"GET /pgphpadmin HTTP/1.1" 403 268
is not saying very much. It'd be better if the error message said something more verbose and direct the user to change the access rules or better yet to have a configuration step during installation that asks for this rather than to have poke around some configuration files.
To change it's default behavior one has to edit the file /etc/apache2/conf.d/phppgadmin (which is a symbolic link to /etc/phppgadmin/apache.conf) and change the default access policy from 127.0.0.1/255.0.0.0 ::1/128 to all.
To do that simply comment out the line that reads
allow from 127.0.0.1/255.0.0.0 ::1/128
and remove the comment from one line below that reads
allow from all
That does the trick. Of course that opens the ability to connect to this server from the outside world, but since this is a server installation it's not uncommon to access it from far far away, is it?