Forward Proxy Strikes Again – Surface Hub (1703)

 

Here’s a screen shot of the Welcome Screen on a Surface Hub, and description of the items presented.

 

Surface Hub Welcome Screen   Surface Hub Welcome Screen Descriptions

 

Issue:

When attempting to select the “Sign in to see your meetings and files” option, a white dialog box appears with no content in it.  What’s expected is a dialog box requesting an O365 sign-in address.

 

Surface Hub Sign In missing

 

If you drag the empty sign in box to the side, you still have the ability to navigate the Start screen which can be seen in the background.

 

The Sign-In option allows you to sign in to O365, to provide full access to personal meetings and your most recently used documents.  NOTE: This requires your content to be hosted in O365 and you to be invited to a scheduled SfB meeting on the Surface Hub.

 

Troubleshooting:

Troubleshooting issues on a Surface Hub has proven to be a challenging process.  Unlike a Windows 10 device, the Shell or UI is not exposed to the user or administrator.  Administrative features, such as the Microsoft Management Console, Run, Command Prompt, PowerShell, registry editor, event viewer, and task manager are not supported on Surface Hub

 

Our first test was simple, let’s try and sign into the O365 portal via the Surface Hub.  So we fired up the Edge browser and went to https://login.microsoftonline.com.  What we expected was the typical Office 365 sign in page.  What we received was a stripped down version without any images and only a few dialog boxes.  Sorry, but no image at this time as I forgot to take a screen shot.

 

The Surface Hub was cabled to the local LAN.  So we decided to the same test from the guest Wi-Fi to see if it could be something network or firewall related.  With the network cable disconnected, and Wi-Fi fired up and connected to the guest network, we re-ran the same tests.  The first test was to try the Sign in button.  Success!  We were prompted with the expected sign in page asking for credentials.

 

Surface Hub O365 Sign In

 

Next we launched the Edge browser and went back to https://login.microsoftonline.com.  Success again!  We were greeted by the expected sign in page.

 

O365 Sign in page

 

Shocking I know, but it appears to be something on the internal network.

 

Our next step was to capture a network trace to see what was going on at the network layer.  We reconnected the Surface Hub to the local LAN via the wired connection.  Then we setup a capture using a mirrored port on the switch.  The capture was started, we clicked the “Sign in” button, and waited for the blank white screen to appear.  We stopped the capture and saved it.  The network trace was saved as a .pcap which we opened and viewed via WhireShark.

 

After some time analyzing the file, a specific conversation started to stand out.  Specifically, a conversation between the Surface Hub and secure.aadcdn.microsoftonline-p.com.  In this conversation, a TLS handshake was trying to be established.  Only the handshake was failing, and it continued to loop through the attempt.  What we saw in the WireShark payload was interesting.  In the Server-Hello message that was being sent from Microsoft, there was a certificate presented.

 

OK, I suppose seeing a certificate presented really isn’t that interesting, as this it what is expected to happen during the handshake.  The server presents the certificate it’s using to encrypt the data, the client accepts it, they negotiate Cipher Suites…bam, we have an encrypted session.  Now what was really interesting was the fact that the certificate that Microsoft presented, appeared to include a local Certificate Authority in the CA chain.  Local as in an Enterprise CA on the clients network, and not a trusted 3rd Party CA.  Now we were getting somewhere.

 

As a side test, I pulled out my laptop and connected it to their internal network.   As expected, I got the same results.  Not only was I presented with the imageless login page, but several other HTTPS sites also presented similar results, such as https://linkedin.com.  We turned to fiddler, another handy tool to keep in your toolbox, to inspect the HTTPS traffic using a different lens .  When I launched this on my local laptop and went to either https://login.microsoftonline.com or https://linkedin.com, fiddler popped an error message similar to the one below.  Sorry to redact so much from the screenshot, but the innocent must be protected.

 

Fiddler Capture 1

 

The SUBJECT contained the expected LinkedIn or Microsoft information; however, the ISSUER contained the local Enterprise CA name.  Which you can imagine is not what is expected.  The ISSUER of the certificate should not be an internal Enterprise CA on the clients network.  The ISSUER should contain the CA information from the 3rd party CA that actually issued the public cert.

Cause:

Classic forward proxy issue.  A forward proxy will decrypt and inspect SSL/TLS traffic from internal users to the web (Man in the Middle). 

 

 

pan-os-555

 

In this case it was a Palo Alto firewall.  Not the first time I’ve seen Palo’s get in the way. 

 

When the Palo Alto Networks device is configured to decrypt SSL traffic going to external sites, it functions as a forward proxy.  In this scenario, when the client (Surface Hub) initiates an SSL session with the server (secure.aadcdn.microsoftonline-p.com), the firewall intercepts the client SSL request and forwards the SSL request to the server.

 

The server (Microsoft) sends a certificate intended for the client (Surface Hub) that is intercepted by the firewall (Palo).  If the server certificate is signed by a CA that the firewall trusts, the firewall creates a copy of the server certificate signed by the Forward Trust certificate (Internal Enterprise CA) and sends the certificate to the client to authenticate.

 

When the client authenticates the certificate, the SSL session is established with the firewall functioning as a trusted forward proxy to the site that the client is accessing.  As the firewall continues to receive SSL traffic from the server that is destined for the client, it decrypts the SSL traffic into clear text traffic and applies security policies to the traffic. The traffic is then re-encrypted on the firewall and the firewall forwards the encrypted traffic to the client.

 

Resolution:

 

The Surface Hub was joined to the internal Active Directory Domain.  Since it was able to sign in successfully to the On-Prem SfB deployment, we knew it was pulling down the Enterprise CA certificates from Active Directory.  Unfortunately, there were multiple Enterprise CA’s deployed internally.  Further, the SfB Servers were signed using different internal CA’s than the Palo’s were. 

 

In this scenario, the Surface Hub did not have the Root/Int CA certs installed that were used to issue the Palo cert, causing the TLS handshake to fail.  Technically it should be pulling them from AD, but for one reason or another it wasn’t.  In order for the Surface Hub to trust the certificate coming from the Palo, we had to get the Root and Intermediate CA certificates into the local Computer Certificate Store on the Surface Hub.

 

As I mentioned earlier, the Surface Hub doesn’t expose the Shell, so this isn’t as easy as opening an MMC console and importing the certs. 

 

Instead, we use the Windows Configuration Designer to create a Package, which can be used to import the certificates to the Surface Hub. 

 

https://docs.microsoft.com/en-us/surface-hub/provisioning-packages-for-surface-hub 

 

Windows Configuration DesignerWCD Advanced Editor

WCD Certificate

 

Once the package is deployed and the Surface Hub is rebooted, the certs are in place.

 

Now that the Surface Hub trusts the certificate chain used by the Palo, the TLS handshake can be completed and the Sign in works as expected.

 

Surface Hub O365 Sign In  Surface Hub O365 Sign In Registering

Surface Hub O365 Sign In All Set

 

 

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *