x11vnc won't start up from an ssh session

x11vnc can't open a display

I've been connecting to a machine in my practise for years now using x11vnc, using an ssh tunnel. All of a sudden, it would not start up anymore, and said XOpenDisplay(":0") failed. caused by Invalid MIT-MAGIC-COOKIE-1 key.

After going through quite some headache, I now know why this happens and how to work around it.

My original command to remotely start the x11vnc server was:

ssh -p 50022 -t -L 5900:localhost:5900 user@host \
    'x11vnc -rfbauth ~/.vnc/passwd -localhost -display :0 -nomodtweak'

and then, to start the session:

vncviewer 127.0.0.1:0

SDDM doesn't use standard locations anymore

The problem is that x11vnc can't find the "MIT magic cookie" anymore, which contains auth information needed to talk to the X server. This information has been stored in ~/.Xauthority since ever. And I suppose this is where x11vnc looks for the auth info if $XAUTHORITY isn't set and/or xauth info doesn't reveal the location of the auth info.

This is caused by SDDM not storing the auth info in ~/.Xauthority anymore (for whatever reason). Actually, it can be found in /tmp/xauth_....

The problem that arises is that, on a local session when starting x11vnc, we both have $XAUTHORITY set, and xauth info also contains the location of the cookie. But not on a remote session. Thus, we're lost with our ssh call.

Giving x11vnc the explicit auth info

My workaround for this is to find the cookie and provide it explicitely to x11vnc. The cookie is a file starting with xauth in /tmp, owned by the user to start the x11vnc server. We can find it like this:

find /tmp -name xauth\* -user $(whoami) -type f

So, we can extend the above x11vnc call with the explicit auth info like this to make it work again:

ssh -p 50022 -t -L 5900:localhost:5900 user@host \
    'x11vnc -rfbauth ~/.vnc/passwd -localhost -display :0 -nomodtweak \
     -auth $(find /tmp -name xauth\* -user $(whoami) -type f)'

Maybe, this will help somebody ;-)