Please note that, generally, fully quitting Teams (Systray icon / right click / quit) and restarting it clears all the components involved and can fix whatever issue you were experiencing.

1. Citrix Optimization For Ms Teams
2. Ms Teams On Citrix
3. Teams With Citrix
4. Disable Citrix Hdx Optimization For Microsoft Teams

The Citrix optimization for Microsoft Teams on MacOS,contains VDA-side HDX services and API to interface with the Microsoft Teams hosted app to receive commands. These components open a control virtual channel (CTXMTOP) to the Citrix Workspace app-side media engine. The endpoint decodes and renders the multimedia locally. In some scenarios, even though 'Citrix HDX Not Connected' is displayed, attempting to make a call will trigger Teams to connect to the Citrix VDA services and optimization will succeed. This is caused by a delay in the Teams UI refresh. If Teams is configured to Auto-start, the issue in the bullet above might also manifest. You can, of course, park your Citrix environment in the upcoming LTSR 1912 release and enjoy the HDX optimization for Microsoft Teams. Virtual Apps and Desktops 7 1912 and Citrix Workspace app 1912 will include optimization for Microsoft Teams and will be packed with a wide set of features.

[Warning: If your VDA is 7.15 (any CU) and you still want to use Teams in unoptimized mode (the only mode supported in these VDA versions), scroll here. Teams HDX optimization will not be backported to the LTSR 7.15].

Step-by-step:

Microsoft Teams optimization mechanisms can be divided into 5 main areas

1. Media Engine instantiation (HdxTeams.exe or HdxRtcEngine.exe, depending on CWA version)

2.Media Engine peripheral acquisition

3.Media Engine transport layer establishment

4. Media Engine multimedia stream processing

5. Monitoring
More information on how to leverage HDX Monitor to monitor Teams optimization can be found here.

Proper troubleshooting requires the Admin to verify each area progressively, moving down the list.

Please note that the presence of a Citrix Gateway as an HDX proxy should have no impact, so you can discard that from your troubleshooting. If you are also using the Gateway for VPN, make sure you are allowing the client machine to reach the O365 Teams servers.

#1: Media Engine instantiation

The quickest way to determine if the user is running in optimized mode is to look for HdxTeams.exe (or HdxRtcEngine.exe) running on the user’s client machine:

If the process is not found, then point #1.1 should be investigated first.

Click on the avatar / About / Version. “Citrix HDX Optimized” should be displayed. This means Teams is ready for an optimized call.

If the legend is not there, and instead you see “Citrix HDX Not Connected” then Teams did not load in VDI mode correctly.

Citrix HDX Not Connected means Teams loaded the Citrix API correctly, but it could not complete an end-to-end check because there was an error in subsequent parts of the stack. Most likely, VDA Services (Citrix HDX HTML5 Video Redirection, Citrix HDX Teams Redirection) or in Workspace app.

If there is no legend with the word Citrix, then Teams did not load in VDI mode at all:

The VDA regkey MsTeamsRedir is set by a Citrix service (CtxSvcHost.exe) when the user logs on or reconnects to the virtual desktop. Teams will read this key when starting, and if the value is 1, it will load in VDI mode.

The value is set to 1 automatically if and only if:

1. Microsoft Teams Redirection policy is enabled in Studio (and by default is Allowed)
2. CWA has loaded the Teams Virtual Channel (VDTeamsn.dll) and the CWA version is 1907 or higher

If = 0 or missing – trouble!

Action: Quit Teams from the systray itself (right click on the icon / Quit).
Sometimes, if Teams auto-updated while the session was in disconnected state, it can fail to read the key properly upon reconnections.

Lastly, if the regkey value is 1 but Teams still fails to display “Citrix HDX Optimized” or “Citrix HDX Not Connected”, there is likely a problem with Teams itself. Common issues could be erroneous cached items (settings.json and storage.json) in AppDataRoamingMicrosoftTeams , or even dual Teams installations (under %appdata% and Program Files). Customers are advised to contact Microsoft support in this case.

#1.2: WebSocket phase

The VDA Service “Citrix HDX HTML5 Video Redirection” [WebSocketService.exe] is the point of entry for Teams into the VDA. In other words, Teams must open a secure websocket connection to this service, which listens at 127.0.0.1:9002 in the VDA.
If WebSocketService.exe is not actively listening, optimization will fail. In some cases, the Service will show as “Running” in the services.msc console, but when you run netstat you will not see the socket.
Therefore, you must make sure it is running and LISTENING :

`

After a successful connection, you will see the state change to “ESTABLISHED”:

[Note: WebSocketService.exe listens also at 9001 – this socket is used for other HDX features (Browser Content Redirection and HTML5 Video Redirection, and they are independent of Teams)]

–Proxies: If an explicit proxy server is configured in the VDA, make sure connections to localhost are not routed through it. Otherwise, redirection will fail. You must “bypass proxy servers for local address” 127.0.0.1:9002

–PAC files: make sure your VDA pac file script is returning DIRECT for wss://127.0.0.1:9002. If not, optimization will fail. One way to do this:

shExpMatch(url, “wss://127.0.0.1:9002/*”)

–Anti-Virus: Some security software products interfere with the proper functioning of WebSocketService.exe and its certificates. While the actual service (Citrix HDX HTML5 Video Redirection) might be running in the services.msc console, the localhost 127.0.0.1:9002 TCP socket is never in listening mode as seen in netstat. Trying to restart the service causes it to hang (“Stopping…”). Ensure you apply the proper exclusions for the WebScoketService.exe process.

Once there is a successful secure WebSocket (wss) connection, WebSocketService.exe will create a new process in the user’s session called WebSocketAgent.exe:

This process can also be spotted in Director:

[Note: WebSocketAgent.exe is also used for BCR and HTML5 Video Redirection]

Action: if WebSocketAgent.exe is not being created while 127.0.0.1:9002 is ESTABLISHED, restart the WebSocketService.exe service. Relaunching Teams might be required. Also, in some rare cases, you might see WebSocketAgent.exe coming up but closing within a second, and in this case this could be related to a crash in HdxTeams.exe in the client machine (hence make sure to collect EventViewer entries on the client).

The last part of this second phase is the establishment of the Teams virtual channel (CTXMTOP), and this process is ultimately handled by the “Citrix HDX Teams Redirection” Service (CtxSvcHost.exe in the VDA).
CTXMTOP is a Command & Control virtual channel only – no media is exchanged between CWA and VDA. All the media is handled by CWA in a Client-Fetch-Client-Render manner.

Action: If WebSocketAgent.exe is running on the VDA but HdxTeams.exe is not running on the client, restart “Citrix HDX Teams Redirection”.

Notes on HdxTeams.exe / HdxRtcEngine.exe :

If you are locking down the endpoint in Kiosk-mode, or restricting users to interact only with whitelisted apps by using GPOs for “Assigned Access Configuration”, then make sure HdxTeams.exe is allowed to run. HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsAssignedAccessConfigurationProfiles

Some Anti-Virus programs might interfere with HdxTeams.exe (which will open and close instantly) . Make sure you are applying proper exclusions.

Once the virtual channel is established, the Citrix HDX Engine in CWA (wfica32.exe) will spawn the media engine on the client:

At this point, Teams is ready to handle calls.

If HdxTeams.exe / HdxRtcEngine.exe launches, then the respective webrpc log will be created.

Windows Logs:
They can be found on the user’s machine in %userprofile%AppDataLocalTempHDXTeams folder.
(Newer versions of CWA, e.g. 2009.6 or higher, store the logs in AppDataLocalTempHdxRtcEngine)
Each session will create its own separate folder.

Mac Logs:
1. vdwebrtc log — records the execution of Virtual Channel vdwebrtc
Location： /Users/<User Name>/Library/Logs/Citrix Workspace/CitrixViewer_<Y_M_D_H_M_S>.txt

2. HdxRtcEngine log — records the execution of process HdxRtcEngine
Location: $TMPDIR/hdxrtcengine/<W_M_D_H_M_S_Y>/hdxrtcengine.log
Note that Hdxrtcengine log is enabled by default.

Linux Logs:
The /tmp/webrpc/<current date>/ and /tmp/hdxrtcengine/<current date>/ directory will have log files.

#2:Media Engine peripheral acquisition

In Teams, go to Settings à Devices. All the peripherals should now be mapped, and the video self-preview visible. You should be able to toggle between the cameras too.
The “Make a test call” button is a very handy tool to assess peripheral acquisition and performance. Go to Settings/Devices/Audio devices.

Note: when Teams enumerates Playback/Recording peripherals, it will select the ones that are defaults at the endpoint. Open mmsys.cpl on the endpoint, and you can see what the default devices are in the user’s client machine. These are the ones that will be auto-selected by Teams.
You can toggle these devices in the middle of a call – in that case, a 1 sec audio glitch might be experienced while the new devices are remapped.

The peripheral acquisition process is captured in the webrpc.txt logs.
All device enumeration entries will have “webrtcapi.RPCStubs” in the respective lines.

Thu Nov 28 15:08:30 2019 Nov 28 15:08:30.947 webrtcapi.DeviceManager Info: Audio device IMMNotificationClient registration successful. Nov 28 15:08:30.948 webrtcapi.VideoOverlay Info: setProvider. set overlay provider (current = 0, new = 10168640)! Nov 28 15:08:30.949 webrtcapi.DeviceManager Info: operator (). Device monitor message window created Nov 28 15:08:31.287 webrtcapi.RPCStubs Info: -> device id = {0.0.1.00000000}.{100c5b9e-8f9f-4bc4-8ff2-196c45c88f76} label = Headset Microphone (Jabra EVOLVE LINK MS) groupId = 1445567130 Nov 28 15:08:31.305 webrtcapi.RPCStubs Info: -> device id = {0.0.1.00000000}.{ee2e77ed-f871-4c65-8476-a40942ea913f} label = Microphone Array (Realtek High Definition Audio(SST)) groupId = 152748168 Nov 28 15:08:31.348 webrtcapi.RPCStubs Info: -> device id = {0.0.0.00000000}.{c11106bd-8a8e-441e-8bba-1740b9b9934b} label = Headset Earphone (Jabra EVOLVE LINK MS) groupId = 1445567130 Nov 28 15:08:31.351 webrtcapi.RPCStubs Info: -> device id = {0.0.0.00000000}.{d10485d5-ddbb-4568-a3ee-214d73dc6b09} label = Speakers (Realtek High Definition Audio(SST)) groupId = 152748168 Nov 28 15:08:31.385 webrtcapi.RPCStubs Info: -> device id = ?display#int3470#4&1835d135&0&uid13424#{65e8773d-8f56-11d0-a3b9-00a0c9223196}{bf89b5a5-61f7-4127-a279-e187013d7caf} label = Microsoft Camera Front groupId = Nov 28 15:08:31.397 webrtcapi.RPCStubs Info: -> device id = ?display#int3470#4&1835d135&0&uid13424#{65e8773d-8f56-11d0-a3b9-00a0c9223196}{7c9bbcea-909c-47b3-8cf9-2aa8237e1d4b} label = Microsoft Camera Rear groupId = […] Nov 28 15:15:01.197 webrtcapi.CameraCapture Info: start. capabilities: width = 1280 height = 720 fps = 30 fmt = 9 Nov 28 15:15:01.553 webrtcapi.DxVideoRenderer Info: configure. window = 203588 frameWidth =1280 frameHeight = 720 […]

#2.1 Errors
Any error in the acquisition process will be displayed in the webrpc logs.
E.g.:

Sun Mar 29 13:23:57 2020 Mar 29 13:23:57.232 webrtcapi.WebRTCEngine Info: init. initializing... Mar 29 13:23:57.520 webrtcapi.WebRTCEngine Error: init. couldn't create audio module!

HdxTeams.exe supports only these specific audio device formats (channels, bit depth, and sample rate): Playback Devices: {1 or 2 channels}, 16 bit, frequencies up to 96000 Hz (or equal) Recording Devices: {1,2 or 4 channels}, 16 bit, frequencies up to 96000 Hz (or equal) Even if one speaker or microphone does not match the expected settings, device enumeration in Teams fails and None displays under Settings > Devices. As a workaround, open the Sound Control Panel (mmsys.cpl), select the playback or recording device, go to Properties > Advanced and change the settings to a supported mode. Also, 'Restore Defaults' under Advanced can be used to recover from this error. Alternatively, disable the specific device.

Important: The error in the table above is fixed in Citrix Workspace app 2009.6 for Windows - so any peripheral (even Studio-Quality soundcards) is mapped.

If Teams loaded in unoptimized mode, it will try to acquire the endpoint’s peripherals leveraging the traditional HDX technologies for Audio, Microphone and Webcam redirection (see section on 7.15 for a description of unoptimized mode).
In this case, the camera name will show the “Citrix HDX Web Camera” legend – this is an unequivocal sign that Teams is unoptimized.

In this phase, errors are generally equivalent to call failures, or one-way audio/video only where one peer cannot hear/see the other.

The call setup process is handled primarily by Teams, not HDX WebRTC engine In other words, the hosted Teams app will utilize the signaling channel back to O365 Teams servers who will in turn contact the other peer.
Teams does rely on HDX WebRTC engine for the call negotiation by requesting an SDP file, which is a compilation of all the media and transport address characteristics of the endpoint.
HDX WebRTC engine relies on hole punching techniques (ICE/STUN/TURN) to eventually establish the connection to the other peer or conference server.

Enterprise network environments can be a challenge in this phase.
Proxies, Firewalls, symmetric NATs, VPNs, Deep packet inspectors, Session Border Controllers, etc etc can all affect the call establishment process; hence it is important you understand your network topology and consult with your networking team if necessary.

Teams conference servers are all hosted in O365 with the IP Blocks 52.114.0.0/12 52.120.0.0/12 and UDP ports 3478-3481. TCP 443 can be used as a fallback.
Peer to peer calls will be attempted via direct connections, but if that fails a TURN server (also hosted in O365, same IP:PORTS as above) might be used.

Important note for DNS : make sure the endpoint can do DNS resolutions. The Teams TURN Servers FQDN must be resolved by the endpoint (e.g. worldaz.turn.teams.microsoft.com and euaz.turn.teams.microsoft.com)

#3.1 When the user initiates or receives a call, the signaling channel between the Teams hosted app and the Teams servers in O365 will negotiate an SDP file, containing all the required information for HdxTeams.exe or HdxRtcEngine.exe to execute the call.
Please note that the SDP file can be updated multiple times after the initial single pass negotiation,
containing new info about ICE candidates or new call parameters (resolution, fps, etc).

Each peer maintains two simultaneous descriptions:

• a local one describing itself ('webrtcapi.RTCPeerConnection Info: setLocalDescription”)

• a remote one describing the other peer (setRemoteDescription)

The SDP files are logged in webrpc.txt. Look for the “>>> begin:sdp” entries.
The caller will ‘set’ his SDP offer first (without any ICE candidates), which can be seen in the webrpc.txt logs as setLocalDescription.

#3.2 Interactive Connectivity Establishment
The caller now will ask STUN servers to generate ICE candidates. It will then send -over the signaling channel- individual ICE candidates as soon as they are discovered (a.k.a “trickle ICE”) by the local ICE agent .

Example #1 : VDI caller (behind Firewall blocking UDP) joining a conference

TRICKLE ICE snippet [...] >>> begin:sdp candidate:780896439 1 udp 2122063615 10.0.0.140 52811 typ host generation 0 ufrag 9gep network-id 1 network-cost 10 candidate:1938109490 1 udp 24911871 52.114.188.57 51757 typ relay raddr 73.205.xxx.xxx rport 5039 generation 0 ufrag 9gep network-id 1 network-cost 10 <<< end:sdp [...]

SDP ANSWER: […] a=candidate:1 1 UDP 36044286 52.114.138.158 3480 typ relay raddr 10.0.136.233 rport 3480 a=candidate:2 1 TCP 18086910 52.114.138.158 3478 typ relay raddr 10.0.136.233 rport 3478 […]

The VDI user caller will register remote ICE candidate and begin connectivity checks (STUN Bind request/response). Sometimes, the calling peer can be behind a Firewall that blocks STUN/UDP packets, and in such a case there will no answer to the STUN Bind requests:

[Warning: STUN Bind requests are essential for a healthy call setup, as this allows candidate discovery with optimal call routing (especially in p2p calls). If your branch office Firewalls are blocking STUN and UDP destination ports 3478-81, then a transport relay in O365 must be contacted using TCP which adds additional latency and degrades call quality experience].

Since the Firewall is blocking UDP, the calling peer must attempt a TCP connection to the other candidate:2
If that also fails, it will have to use his transport relay (“relay raddr”) seen in the Trickle ice snippet above (52.114.188.57).

Note: HdxTeams.exe cannot read explicit proxy configurations on the endpoint.

#3.3 RTP media flows
If the connection was established using TCP through a TURN server, you will not be able to decode the traffic in Wireshark as RTP since it is encrypted using TLS.

In case of a peer-to-peer connection or conference via UDP, then Wireshark will show more detailed info that can be used to understand the traffic flow between the participants.

Wireshark recommendations:

RTP cannot be automatically analyzed by Wireshark. Applying the Display filter ‘rtp’ won’t result in any RTP packets displayed.
Instead, first set the Display Filter to ‘udp’ to spot UDP traffic. After the connection is established, there will be UDP traffic sent regularly between the peers. RTP over UDP traffic will begin with 90 hexa (or 80 hexa if no RTP header extensions are present) in Data, the first octet of the RTP header.

By selecting one of these UDP packets and selecting 'Analyze/Decode As' … then selecting RTP, Wireshark will process the packets as RTP.

The WebRTC media engine in Workspace app (HdxTeams.exe) uses the Secure RTP protocol for multimedia streams that are offloaded to the client. SRTP provides confidentiality and authentication to RTP, using symmetric keys to encrypt media and control messages, and uses the AES encryption cipher in Counter Mode.

4. Media Engine multimedia stream processing

Once the call set up is finalized and the ICE connectivity checks are completed, the media will be exchanged between the parties.
Currently we support video resolutions up to 720p@30fps, with 360p being common [H264 and VP9].

-The video streams are captured in the webrp.txt logs as 'DxVideoRenderer ':
webrtcapi.DxVideoRenderer Info: configure. window = 657170 frameWidth = 640 frameHeight = 360
-The webcam captures entries in webrp.txt are'CameraCapture':
webrtcapi.CameraCapture Info: start. capabilities: width = 640 height = 360 fps = 30 fmt = 9

Lip sync issues might be caused by high CPU usage on the endpoint. The encoding process (camera capture and subsequent encoding) is more expensive in CPU terms than decoding. Some endpoints might not have enough processing power (see here for system requirements), resulting in 100% CPU utilization and causing audio/video to be out of sync.

Important note:Workspace app for Windows 1912 LTSR and 2002 CR (and any posterior version) introduced a performance estimator that is designed to measure the encoding capabilities of the endpoint's CPU. As a result, the video encoding is limited to a certain resolution (240p, 360p, 720p, 1080p). This is particularly useful for endpoints that have low CPU speeds/amount of cores. The estimator is run on HdxTeams.exe launch time, transparent to the user. The estimator might return a value that is lower for VP9 vs H264, since VP9 consumes more CPU. If you would like to tweak the resolution values, or disable VP9 and use H264 exclusively, this can be controlled via regkeys on the endpoint. See here for more details.

The RTP protocol can be decoded in Wireshark to understand what type of codecs are being used and how much bandwidth is consumed.

Citrix Optimization For Ms Teams

#4.1 As an example we will investigate a p2p audio call between two VDI users. HdxTeams will select the Opus codec for this scenario, which can be seen clearly in the SDP Offer/Answer negotiation.
The calling party will state all his supported codecs, and the callee will then reply with his preferred choice in the SDP answer.

[Note: The notification audio (ringtone) is played by Teams out of the VDA’s default playout device.]

The call bandwidth usage can then be plotted in Wireshark using the Statisticsà IO Graph option.
Apply the proper filters to display only the traffic associated with the call.
We can see that the typical bandwidth for a voice call is ~40-70 kbps (each way).

Green: outgoing | Blue: incoming | Red: outgoing + incoming

Screensharing

In the Skype for Business optimization pack (RTOP), screensharing was not optimized. In other words, there was no offloading of such streams and therefore they were rendered server-side natively by Skype for Business.

But in terms of HDX optimization for Microsoft Teams, incoming screensharing is no different than incoming webcam video streams, since Teams relies on video-based-screen-sharing. Hence it is optimized by default.

However, a VDI user sharing his monitor (i.e. outgoing screensharing) will result in HdxTeams.exe / HdxRtcEngine.exe performing display captures at the endpoint (with H.264). Currently it is only possible to share the primary (default) monitor.
HdxTeams.exe / HdxRtcEngine.exe crops the sharing to the Desktop Viewer window only (if CWA's Desktop Viewer is in Windowed mode).

If Desktop Viewer (cdviewer.exe) is in full screen mode, then of course the entire monitor is shared.
In multi-monitor setups, only the primary monitor is shared, and the screen picker button in Teams will label the preview as 'Screen #1'.

Common issues with outgoing screensharing, as perceived by the other peer, are:
- low resolution
- black window instead of the actual screen
- screen freeze on the sender's side

The webrpc.txt logs are the best source for troubleshooting such issues.

HdxTeams.exe / HdxRtcEngine.exe contains a bandwidth estimator that will allow Teams to pick the best resolution that can be sustained by the network, and sometimes it can take a few seconds for it to converge to the right value. So screensharing might actually start with a low resolution before it jumps to a higher one.
In the logs, the parameter 'webrtcapi.DesktopCapture' can be used for quick finds, and if there are errors they would be logged as 'webrtcapi.DesktopCapture Error:'.
Black windows can be caused by issues on the sender's side (user capturing) or the receiver's side (user rendering the stream).
For example, if the error is on the sender's side the following entry could be seen:
webrtcapi.DesktopCapture Error: OnCaptureResult. invalid cropping rectangle (x = 0 y = 0 width = 0 height = 0)

Monitoring Teams with HDX Monitor

On 1912 VDAs (or higher), you can monitor an optimized active call using HDX Monitor (3.5.2 or higher).
Important : make sure you are matching the HDX monitor version with the right VDA version (HDX Monitor is packaged with the CVAD .iso)
To turn this on, create the following WebrtcDirectorIntegration regkey on the VDA:

HKLMSoftwareCitrixHDXMediaStream
Or
HKLMSOFTWAREWOW6432NodeCitrixHdxMediaStream
- reg key value:
name: WebrtcDirectorIntegration
type: DWORD
value: enable(1), disable(0)

The information is displayed under the WebRTC icon :

The following data content will be populated when an active Teams redirection is going on.
Please see below for the detailed field description:

• Call Device Used Audio: audio device used for the last active call
• Call Device Used Video: video device used for the last active call
• Call Direction: incoming or outgoing
• Call Duration Audio: last active audio call duration in seconds
• Call Duration Video: last active video call duration in seconds
• Call Duration Conference: last active conference call duration in seconds
• Call Duration Screenshare: last active desktop sharing call duration in seconds
• Call Establishment Duration Incoming: last active incoming call establishment in sec
• Call Establishment Duration Outgoing: last active outgoing call establishment in sec
• Call State: Active or Idle
• Call Type: Audio, Video, Conference, Screenshare
• Version - Endpoint OS: not filled currently
• Version - Receiver: CWA version
• Version - Typescript: API version
• Version - Vda: not filled currently
• Version - Webrpc: webrpc version
• Version - WebrtcCodec: webrtc codec version

Guidelines for 7.15 LTSR

If you are using 7.15 LTSR (with any CU), please note that the new HDX Optimization for Microsoft Teams is not (and will not be) available in this version and the only way to use Teams is through the legacy HDX technologies:

• Client Audio Redirection (we recommend using the RTP/UDP policy, which requires the Audio quality to be set to Medium)
• Multimedia Conferencing (for Webcam redirection, a.k.a HDX RealTime Webcam Video Compression)

While these technologies do offload some of the CPU/RAM/BW from the VDA, they do not fully optimize the experience for Real Time traffic and will create additional loads on the VDA resources.
The best example is the rendering of a video feed from another peer / meeting. This is done using Thinwire graphics technologies and it renders on the VDA, which is expensive in CPU terms.
Admins concerned about Single Server Scalability might want to disable video conferencing policies on the Microsoft Teams admin center. See here for more info.

A secondary method for redirecting peripherals is available, and some customers might prefer to use Generic USB Redirection.
With HDX Generic USB Redirection technology, the peripherals (e.g webcam or headset) is virtually detached from the client device and attached to the XenApp/XenDesktop session. This provides all the native functionalities of the webcam/peripheral in the virtual session.
HDX Generic USB Redirection requires the device drivers for the webcam/peripheral to be available on both the client device as well as on the VDA.
Bandwidth usage for webcams using HDX Generic USB Redirection technology can vary based on the vendor and model of the device, but it is significantly higher compared to use it over HDX RealTime Webcam Video Compression. HDX Generic USB for webcams/peripherals is recommended to be used only under LAN conditions where bandwidth and specially latency are not constraints.
Refer the following link regarding more information on HDX Generic USB configuration: Configure USB Support.

Known Issues with Microsoft Teams and 7.15 VDA:

• Webcam redirection will not work with 64-bit Teams, only with 32-bit. This is known limitation in 7.15 (any CU) - see here for more details.
• In Windows Server, Teams.exe might crash, or the Teams logon screen might become blank and no input fields are presented to the user. Please add Teams.exe to the VDA registry under SfrHook, and make sure the Flag value REG_DWORD is set to 204 Hexadecimal
• After you add the SfrHook key, you must reboot your VDA.
• As a last resort, you can disable ALL hooking for Teams by following CTX107825

Related:

• No Related Posts

During the last couple of weeks I have been helping customers implement Microsoft Teams in their Citrix VAD setups. A common denominator for most of the Teams implementations was Teams consuming a lot of resources, different Teams versions were present in the environment and Teams generating a huge amount of temporary or cached data in the user’s profile.

In this article I’ll share my experiences with Teams in Citrix VAD. This is by no means a best-practices install or configuration guide it’s more of a guide on how to avoid a couple of different pitfalls and hopefully also provide a great user experience with Teams in a Citrix VAD setup.

If you are not familiar with Microsoft Teams, you might want to gather some information before installing or configuring anything with Teams in a Citrix VAD setup. Visit this site, if you want to know more about Microsoft Teams.

First of all I want us to be on common ground before going any further with this article, so we’ll have to cover the different ways of installing Microsoft Teams, as this is an area causing a bit of confusion. In this article I am using the 64-bit version of Teams and the 64-bit version of Office installed in Windows Server 2019 with using FSLogix Profile Container.

Installing Microsoft Teams Per-User:

Today there are 2 different ways of installing Microsoft Teams. You can install it either as a per-user install or a per-machine (machine-wide) install. Microsoft recommends to install Teams as a per-machine install in non-persistent setups.

The per-user install can be installed in a few different ways. Either via the Office 365 click-to-run installer, via an EXE file or via an MSI file, Microsoft isn’t making this easy! Both the EXE installer and MSI installer can be downloaded in either 32-bit or 64-bit, make sure to get to one matching the Windows architecture.

You can get the EXE file here:
https://products.office.com/en-us/microsoft-teams/download-app
You can get the MSI files here:
32-bit – https://teams.microsoft.com/downloads/desktopurl?env=production&plat=windows&managedInstaller=true&download=true
64-bit – https://teams.microsoft.com/downloads/desktopurl?env=production&plat=windows&arch=x64&managedInstaller=true&download=true

So, as you can there are 3 different ways of deploying Microsoft Teams as a per-user install, a bit of a mess if you ask me and I am not surprised if some finds it a bit confusing.

We’ll need to dive a bit deeper in how the per-user install actually works, even though it’s not the recommended way of deploying Microsoft Teams, there is some useful information for when we cover the migration from the per-user install to a per-machine later in this article.

Both the EXE file, MSI file and the Office 365 click-to-run “installs” a Teams.exe file and a setup.json file in C:Program Files (x86)Teams Installer:

In this case I have installed version 1.3.0.4461 of Teams:

The Teams.exe file is the actual installer, which installs Microsoft Teams in AppDataLocalMicrosoft the user’s profile. The installation is triggered by Teams.exe process via registry, which can be found here:

For copy/pasting:
HKEY_LOCAL_MACHINESOFTWAREWOW6432NodeMicrosoftWindowsCurrentVersionRun

So a plain old registry value in Run is used to kick off Teams, not necessarily the best way to start an app in a non-persistent shared environment, but then again this is the per-user install of Teams, which is meant to be installed on a physical Windows 10 machine, not a shared environment.

As mentioned, during logon Teams is installed in the user’s profile and when Teams is started up and the user has logged on, this is how the Teams install folder looks like:

Once this is completed, the Update.exe process, now in the user’s profile, is used to start Teams. This is, again, done via registry:

For copy/pasting:
HKEY_CURRENT_USERSoftwareMicrosoftWindowsCurrentVersionRun

As you can see the Update.exe is executed with a few parameters. I have not been able to find any information as to why this procedure is used to start Teams in a per-user install. My guess is that this Update.exe process checks for any new releases of Teams during startup of Teams, and then downloads the latest version at some point.

Microsoft has a very short article about the update process here:
https://docs.microsoft.com/en-us/microsoftteams/teams-client-update

According to the article Teams is updated every two weeks, no specific time of day is mentioned, so we’ll have to assume that the update process just kicks in at random. I have had a Teams running in a session for a couple of hours, no update kicked in. I have tried to log on and log off several times with Teams auto launching, nothing. At a customer I have seen 3 different versions of Teams being used at the same time, by different users. This might complicate things a bit in terms of troubleshooting because of the different versions. Some users might have issues that other users don’t have because they user another version of Teams.

For the sake of this article, I have done a manuel update via the “Check for Updates” feature:

This kicks off the update process, where the Teams.exe process and the Updates.exe process both consume a considerable amount of CPU resources, both processes have the priority of “normal” in Windows, which means that it might slow any other applications down for a couple of minutes, especially if you have multiple users where this update kicks in at the same time.

The update process goes out to Microsoft and downloads the latest version of Teams to the AppDataLocalMicrosoftTeamsstage folder in the user’s profile:

Once the source files for the new version of Teams are downloaded, the user will get a notification about a new version being available:

If the user clicks the “Please refresh now” text box, the updater kicks in and is again consuming a considerable amount of CPU resources, still at “normal” process priority, which may once again potentially slow other apps down for a period of time.
Interesting stuff is also going on in the user’s profile. The “stage” folder is now removed, and replaced with a “previous” folder:

So the user now has two versions of Teams in the profile, the current updated version, which is installed in the “current” folder and is the one being actively used in the current folder, and then the previous version of Teams, which is no longer used, essentially now doubling the amount of space used for the Teams install. Considering that I have found no information of how a user might be able to revert to a previous version of Teams, there is nothing in the Teams app that enables the user to roll back to a previously used Teams version, I am having a difficult time understanding why it’s necessary to store the previous version in the user’s profile, why isn’t just deleted?

To wrap this section up, there really isn’t any reason to use a Teams per-user install in a shared environment. In a shared environment we should have a degree of control of the apps installed and update process of the apps, to ensure stability and functionality. With a Teams per-user install, we don’t have any control, from the moment it’s installed it’s out of our control, because we don’t control the update process.

Migrate Teams per-user to Teams per-machine

Now you have come this far and you might have realized that Teams isn’t installed in the correct and recommended way, you can go a few different ways. Leave it be, and hope that Microsoft doesn’t change anything major or add additional features, which might demand even more resources or maybe break existing functionality. Or remove the current Teams per-user install and deploy the Teams per-machine install instead, which is also the recommendation from Microsoft.

If you decide to leave Teams alone in it’s current state, then there is no reason for you to read any further. However if you want to deploy the Teams per-machine instead, then stay with me.

To be honest this isn’t really a migration, it’s really “just” an uninstall of Teams, and an install of Teams suited for non-persistent shared environments.

Switching to a Teams per-machine install is fairly easy, you are probably not expecting that, considering we have to go out to every single user profile and remove a Teams per-user install, but Microsoft has actually done some clever thinking, when it comes to removing Teams per-user.

Uninstall Teams per-user

The first thing we’ll need to do is to remove the Teams per-user install. In Windows Server 2019 we’ll go to Apps and Features select the “Teams Machine-wide installer” and click uninstall. In this case the name is not entirely accurate, or it is, but the “Teams Machine-wide installer” is the machine-wide, or the per-machine installer, but it can also do a Teams per-user install. You might see “Teams” or “Teams Installer” instead, this is because you have used the EXE installer, mentioned earlier.

Back on track. The uninstall should be pretty uneventful, it’s an uninstall like any other uninstall, other than this uninstall only removes the C:Program Files (x86)Teams Installer folder, and not the Teams installed in the user’s profile. So, how to remove Teams from the users profiles? This is where Microsoft has done some clever thinking. During the uninstall of Teams per-user, two registry values are created here:

For copy/pasting:
HKEY_LOCAL_MACHINESOFTWAREWOW6432NodeMicrosoftWindowsCurrentVersionRun

We need the data in the value “TeamsMachineUninstallerLocalAppData”, this string will uninstall Teams per-user, in the user’s profile.
For copy/pasting:
%LOCALAPPDATA%MicrosoftTeamsUpdate.exe –uninstall –msiUninstall –source=default

You HAVE to use this uninstall string, it is not enough to just delete the Teams folder from the user’s profile, Teams will come back if you do and you could end up with a mix of Teams per-user and Teams per-machine, they are able to exist perfectly fine side by side, you don’t want that!.
If you leave both values where they are, Teams will be uninstall during the next logon. In some cases that might be OK, however if you want a more controlled process, let’s say you want to do the uninstall for a specific group of users or when user’s access a test-server, you can bring in something like Citrix Workspace Environment Management, to execute the uninstall string based on AD group membership or anything that would identify the server as a test-server or whether the Teams install is a per-user or per-machine.

If you are going with the WEM approach make sure that both the “TeamsMachineUninstallerLocalAppData” and “TeamsMachineUninstallerProgramData” values are deleted, before going any further.

In WEM we can use an external task to execute the uninstall string:

Instead of using an AD group membership as a filter for the Teams per-user uninstall, we can use a combination of two filter conditions doing File/Folder matches, making sure that Teams per-user is not uninstalled, unless there is a Teams per-machine installed on the Session Host/VDI. We will have to create a filter condition which is checking to see if “%LOCALAPPDATA%MicrosoftTeamscurrentTeams.exe” exists and another filter condition which is checking to see if “C:Program Files (x86)MicrosoftTeamscurrentTeams.exe” exists. The “C:Program Files (x86)MicrosoftTeams” folder is where the Teams per-machine is installed, we’ll cover that in a moment.

The filter conditions look like this:

With these conditions I can create a filter rule which can be assigned to the “Teams per-user uninstall” external task.

The filter rule looks like this:

For this filter rule to apply, both filter conditions have to me met.

The last thing we need is to assign the “Teams per-user uninstall” external task:

Go to Assignments and click the little arrow button

In the drop down box select the filter rule we just created

You should end up with an assignment looking like this.

To summarize – Via WEM we are now uninstalling Teams per-user if the user is logging on to a Session Host/VDI that has Teams per-machine installed and Teams per-user exists in the user’s profile. We now have a controlled way of getting rid of Teams per-user.

Install Teams per-machine (Machine-wide)

There are a lot of different articles and guides on how to install Teams in a non-persistent and/or shared environment, I recommend this article by fellow CTA Manuel Winkel:
https://www.deyda.net/index.php/en/2020/02/25/install-teams-onedrive-in-citrix-machine-based/

Going further, I am assuming that you are going with the WEM approach, if you are not there might be some slight differences in how Teams behaves.

Also be aware that Microsoft is not making things easy for us at the moment. Currently there are two different download links for the Teams per-machine MSI installer, make sure to get the version from the link i Manuels article, as this is the version currently supported by Citrix (CTX253754). Make sure to keep an eye on that CTX253754 article.

The most important thing to remember is to user the correct install parameters during setup, to make sure that Teams is deployed as a per-machine install. Either go to the article by Manuel, refer to the official “Teams for Virtualized Desktop Infrastructure” documentation or use this command:
msiexec /i Teams_windows_x64.msi ALLUSER=1 ALLUSERS=1

To verify that it is a Teams per-machine install, make sure that you have a “C:Program Files (x86)MicrosoftTeams” folder. The folder structure in here should look familiar to you:

Teams is launched from the “current” folder via the Teams.exe process and once again a registry value is used to do the launch.
The registry value can be found here:

For copy/pasting:
HKEY_LOCAL_MACHINESOFTWAREWOW6432NodeMicrosoftWindowsCurrentVersionRun

Personally I delete this registry value, because I don’t want Teams auto starting via registry. There might be situations where you want to have a bit more control over who is running Teams, maybe because of license enforce ment or maybe you are testing Teams, and only want a certain group of users to be able to access Teams. Or perhaps you just don’t want applications auto launching during logon.

To control the Teams startup, we’ll again turn to Citrix WEM. Create an action, in this case it’s just called “Teams”:

Assign the newly created Teams action:

Ms Teams On Citrix

In this case I have created filter rule with a filter condition with an AD group membership check, so my user will have to be a member of a specific AD group for the action to apply.

Configure Teams for automatic start up:

Make sure Auto Start has a green check mark.

This is it! Teams per-machine is now alive and kicking.

Profile Exclusions

Both Teams per-user and Teams per-machine downloads a huge amount of temporary/cache data during first launch just to immediately flush it again, and to be honest I am not entirely sure why or what kind of data is downloaded, especially not with the per-machine install. However if you are not configuring the correct exclusions, you might see your FSLogix Profile Container increase in size, as the temporary/cached Teams is written and flushed.
With a fresh FSLogix profile, I have seen the container expand to around 4-5GB in size when launching Teams, with writes going the the AppDataRoamingMicrosoftTeamsService WorkerCacheStorage folder. If you mount the profile container, when it’s not in use, you’ll find that there’s only around 400-800MB of data in the container, and nothing or very few small files in the AppDataRoamingMicrosoftTeamsService WorkerCacheStorage folder.

As with any other profile exclusions, you should of course do some testing, before implementing in a production environment

UPDATE – 14-07-2020 (july 14, 2020):
If you are using FSLogix Office Container, do not include Teams data in the Office Container, as the exclusions mentioned will no apply to the Office Container, they only apply to the Profile Container.
This means that you should either leave this policy at not configured or configured it as disabled:

UPDATE – 19-05-2020 (may 19, 2020):
The list of exclusions, below, has once again been updated. Via a Citrix discussions forum post, I have been made aware that certain exclusions are breaking things.
Excluding “AppDataLocalMicrosoftTeamscurrentresourceslocales” apparently breaks the system tray menu.
Excluding “AppDataLocalMicrosoftTeamsCurrentLocales” apparently breaks SSO to Teams.
Do not add the folders with a strikethrough. If you do, test, test, test!

Exclusions:
AppDataLocalMicrosoftTeamsPackagesSquirrelTemp
AppDataLocalMicrosoftTeamscurrentresourceslocales
AppDataLocalMicrosoftTeamsCurrentLocales
AppDataRoamingMicrosoftTeamsService WorkerCacheStorage
AppDataRoamingMicrosoftTeamsApplication Cache
AppDataRoamingMicrosoftTeamsCache
AppDataRoamingMicrosoft TeamsLogs
AppDataRoamingMicrosoftTeamsMedia-Stack
AppDataRoamingMicrosoftTeams*.txt (Cannot be implemented with FSLogix Profile Container, as it does not support file exclusion or exclusions based on wildcards)

UPDATE – 03-05-2020 (march 3, 2020):
The list of exclusions, below, has been updated. According to the Microsoft Teams documentation the AppDataRoamingMicrosoftTeamsMedia-Stack should be excluded and the same goes with AppDataRoamingMicrosoftTeams*.txt files

Teams Outlook Add-in

For some reason the Teams per-machine Outlook add-in is not loaded, so when a user launches Outlook and wants to arrange a new Teams meeting, the Teams add-in is simply not there, and it’s nowhere to be found in the list of available add-ins:

I would expect the add-in to be between the Skype add-in and the OneNote add-in, but it’s not. I am not entirely sure what is going on here, but I have found a workaround which should bring the Teams add-in back.

UPDATE – 03-05-2020 (march 3, 2020):
Teams has to be launched at least once to be able to access the Teams plugin. This means that even if you activate the plugin in Outlook,during first logon, it does not work until Teams is launched. For now I haven’t found any solution to that issue.

Teams With Citrix

The workaround is a minor registry change in HKCU, configuring the LoadBehavior value for Microsoft Outlook add-ins:
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USERSoftwareMicrosoftOfficeOutlookAddInsTeamsAddin.FastConnect]
“Description”=”Microsoft Teams Meeting Add-in for Microsoft Office”
“LoadBehavior”=dword:00000003
“FriendlyName”=”Microsoft Teams Meeting Add-in for Microsoft Office”

This should bring back the Teams outlook add-in. We can, once again, use our trusted Citrix WEM to do the import where we’ll create a nice little action group, with the Teams shortcut and the registry values like this:

Apply the Teams Auto Start filter rule we created earlier, in this way we have everything around Teams in one single group.

And here is the highly demanded Teams outlook add-in:

Citrix HDX Optimization

The last thing we need to do is to make sure that Citrix HDX Optimization has kicked in.

Disable Citrix Hdx Optimization For Microsoft Teams

The Teams HDX Optimization is supported in Citrix Virtual Apps and Desktops 1906.2 and later and you’ll also have to use Citrix Workspace App 1907, however Citrix strongly recommends using Citrix Workspace App 1912 or 2002. You will also need Teams version 1.2.00.31357, however Citrix recommends version 1.3.00 .4461 or later.
Refer to this article for additional information:
https://support.citrix.com/article/CTX253754

If all of the above mentioned criteria have been met, you should see a “Citrix HDX Optimized” notification in Teams (in about -> version):

The Teams HDX Optimization enables Teams video and audio calls to be offloaded to the local endpoint device, this feature offloads a considerable amount of CPU usage on the Session Host/VDI to the endpoint. Be aware that the Teams HDX Optimization feature is not available on Linux based devices, at the moment it’s only supported on Windows devices.

Thank you for reading. If you have any questions feel free to contact me via Twitter, LinkedIN or in the World of EUC Slack channel.