If you have access to the web-mgmt UI, you can select the message-VPN, then “Try Me!” from the left-column toolbar. The Try Me! tab allows you to test basic connectivity for Javascript clients in the browser, and you can specify WS or WSS protocol as well as username/password and msg-VPN.
In your case, it’s sounding like there is an issue with the browser accepting the server certificate.