Skype for Business (S4B)/ Lync 2013 Resiliency Outline:  S4B and Lync 2013 resilience pools are similar to other highly available (HA) and fault tolerance solutions.  If one pool fails (i.e., server or network disruption) its clients automatically connect to the second pool. Even better, the Lync clients maintain their client-to-client sessions (i.e., VOIP and IM) after connecting to the backup pool :

• UDP Direct.
• UDP NAT.
• UDP Relay.
• TCP Relay.

When the primary server pool fails (e.g., server down) its clients experience a brief hiccup (e.g., 1 second delay) as they connect to the backup pool.  Although this near-HA solution is useful, the system is not without flaws.  The resiliency caveat is that clients enter a limited resilience mode upon connecting to the backup pool.  Clients in resilience mode operate with limited functionality:

• New users connect in resilience mode.
• Scheduling features are unavailable.
• Presence state displays as unknown.
• The Contact list and Address book is not available (searches for individuals work though).

What is the Lync CMS?  The central management store (CMS) is a SQL Express database (Standard Edition) that stores configuration data in XML format.  The CMS database is named RTS.  Lync server tools make changes to the CMS database:

• Lync PowerShell Console.
• Lync Server Control Panel (LsCP).
• Lync Topology builder.  

What is the replication database?  The CMS database uses a master-slave replication model.  All changes to the active RTS database are replicated to the RTSLocal databases.  Changes are replicated across pools.   Lync uses the RTCLocal instance -not the master RTC for its client services.  This distinction is important for resiliency purposes.   

What is the Lync Front End Pool?    The Lync Front End pool consists of a single (i.e., Lync Standard) or multiple (i.e., Lync Enterprise) Front End servers that connect to an associated back-end CMS database.  There is only one pool per CMS instance.  Users are assigned to a single pool; never both.

When users sign-on to Lync, their clients automatically connect to Front end servers which resides in the users' assigned Front End pool.  The Front End sever uses the replication database located in its local pool. 

What are Lync Resiliency Pools?  The Lync Resiliency pool consists of exactly two Front End pools (e.g., pool_A or pool_B).   This 1:1 pool ratio provides one primary pool and one backup pool.  If a single resiliency pool becomes unavailable, the second Front End server prevents any change to its CMS database.  However, the second server continues to operate using its replica's database which maintains a static configuration from the time off the loss.  Clients that re-connect use resilient mode because the back-end service is in a read-only state.  Consider:

• The resiliency pool is considered active/passive when individuals are all assigned to the same pool.  
• The resiliency pool is considered active/active when individual Lync users are assigned to separate Front End pools.  

The difference between the two models is significant during the fail-over process.  In the event one of the Lync servers fail there will be some clients that enter resiliency mode and others will remain in standard mode.


What is Lync High Availability?  Lync Enterprise separates the front-end and back-end roles for truly high-available services within a single pool.  Each pool allows up to twelve Front End servers and connects up to 80,000 simultaneous users.  In addition, Microsoft SQL 2012 hosts the CMS database by either mirroring or shared storage clustering.  Disruption to any single server within the same pool is negligible and are therefore considered highly available.  

Lync 2013 Standard edition differs because it consolidates roles; both the Front End and CMS installs on the same server.  Nonetheless, Lync Standard remains a robust solution.  It scales up to 5,000 users, is simple to deploy, and supports all workloads (i.e., VOIP, IM, etc...).  Microsoft provides an alternative to HA through the use of resiliency pools.

How do Lync resiliency pools work?

Back-End Server Perspective:

1. There are only two resiliency pools (e.g., pool_A or pool_B).
2. There is only one CMS master database per pool.  
3. The CMS instance replicates to the LocalRTS database.
4. Lync servers always use the LocalRTS -never the CMS.

Front-End Server Perspective:

1. Front End Servers belong to a single pool.  Each pool has at least one Front End Server.
2. Users are assigned to a single pool; never both.
3. Active/Active pools assign users to both pools.
4. Active/Passive pools assign users to a single pool.

1. Users connect to Front End servers (i.e., pools) based on administrative configuration.
2. Clients from the primary pool communicate seamlessly with clients from the secondary pool; and vice verse.
3. Pools are not important to the end users unless the client is forced into resiliency mode.

What happens when a Lync server goes down?  

• Changes cannot be made to the CMS database from any pool when an outage occurs.
• Clients access the data, rather than modify it.  The Lync FE server uses a second database 
• Users associated with the unavailable server enter resiliency mode.  

How do we fail-over Lync clients?

 Determine which Front End pool hosts the CMS:

1. 
   

   Get-CsService -CentralManagement
   

   The command will most likely error out because the primary pool, which normally hold the active CMS is unavailable. 
   
   However, the CMS can run on either pool, and we can at least determine if the secondary pool is the active CMS host.  Look for the active attribute in the host Identity field. 
2. Let's assume the primary pool hosted the active CMS and is no longer available.  Fail over the CMS to the secondary pool:
   
   Invoke-CsManagementServerFailover -BackupSqlServerFqdn FE02.lab.local –BackupSqlInstanceName RTC –Force
   
   Verify the secondary pool hosts the active CMS:
   
   

   Get-CsService -CentralManagement
   

   Look for the active attribute in the host Identity field for confirmation.
3. Once we've confirmed the secondary pool hosts the active CMS we can fail over the Replica to the secondary pool.
   
   Invoke-CsPoolFailOver –PoolFqdn lync01.domain.com –Disastermode –Verbose
      
   Lync clients seamlessly exit resiliency mode and enter standard mode.  The "Limited Functionality" warning disappears!

Fix Temporary Profiles on RDS Server.

Problem:  Users receive temporary profiles each time they log onto the Remote Desktop Server.  The problem is not uniform and may only occur for individual users.

Possible errors may read, "Windows cannot find the local profile and is logging you on with a temporary profile.  Changes you make to this profile will be lost when you log off."

Background:  RDS server is configured with User Profile Disks (UPD).  The problem usually occurs when the user session state is disconnected.  Although the user is not connected their sessions continue to run in the server background -therefore their UPD remains attached to the RDS server.  Users will receive temporary profiles if their UPD was attached to the RDS server when it was restarted.

How to fix temporary profile issues:

1. Have user log off the server.  Do not allow users to log on while troubleshooting.
2. Attempt to delete the temporary profile from Advanced System Properties:
   
   Go to Control Panel → System → Advanced System Properties → Advanced → User Profiles → Settings.
   
   Delete any profiles with Type set as TEMP.
3. Delete any temporary use profiles from the ProfileList in Regedit.
   
   Open Regedit:
   HKLM\SOFTWARE\Microsoft\Windows NT\Current Version\ProfileList
   
   Profiles are listed using objectSIDs, for example:
   S-1-5-82-3848493281-17249319953-1783664087-774473773-2625933694
   
   Identify and delete any objectSIDs that have a .bak suffix, for example:
   S-1-5-82-3848493281-17249319953-1783664087-774473773-2625933694.bak
   
   N.B., each objectSID profile has an expandable string value called ProfileImagePath.  Use this string value to easily identify individual users, for example:
   
   Value Name:  ProfileImagePath
   Value Data:  C:\Users\BGates


4. Finally, manually delete any temporary profiles located in c:\users\, for example:
   
   TEMP.Domain.000
   TEMP.Backup-0

How to prevent temporary profiles issues (i.e., mitigation):

Set GPO policy to log off disconnected sessions after X minutes.   This will reduce the likelihood that UPDs will remain connected if the server is restarted or loses power.

Additionally, ensure all user sessions are logged off and do not allow new connections to hosts before performing maintenance and/or restarts.

Integrate Two-Factor  Authentication with Windows .

Table 2 MFA Pricing per Technology
MFA	Licensing and Equipment	Fees
RSA SecureID	RSA VM servers included with licenses.
	One time purchase of Authentication Users licenses.  $81 per user.  Minimum 10 users:		$810
	3 Year Tokens.  $55 per token.  Minimum 10 users:		$550
	Annual Maintenance (SA). $28 per user.  10 users:		$280
	*Tokens and maintenance are reoccurring expenses.
	Total installation for 10 Users:		$1640 (one time).
	Total reoccurring (every 3 years) for 10 users:		$1390 (every 3 years).
MS Azure	On-premise server included with Azure licenses. ADFS and IIS proxy included with current 2012R2 Data Center licenses.
	$2 Per user per month subscription.  10 users per year:		$240 (annually).
Certificates	AD CA server included with 2012R2 Data Center licenses.  No additional out-of-pocket fees.		N/A
Smart Cards	Smart cards are dependent on AD CA infrastructure.
	One time $30 per smart card per user.  10 users:		$300
	One time $30 per smart card reader per user.  10 users:		$300
	Total one-time price for smart cards and readers; based on 10 users:		$600

Table 3.  MFA Pros and Cons Summary

MFA

Agent Required

Secure Remote

Secure Admin

Easy to  Replace

Token/ Card

Cost per License

Recurring Fees

Est.Total Cost

RSA

Yes

Yes

Yes

Yes

$56

$81

$46/y

High

MS Azure

No

Yes

No

Yes

N/A

N/A

$24/y

Medium

Certificates

No

Yes

No

Yes

N/A

N/A

N/A

Low

Smart Card

No

Yes

Yes

Yes

$60

NA

NA

Low

by Steven Jordan on 4/16/2014

Take Away:  Use Microsoft Windows and its MakeCert tool to create free SSL certificates.  These custom-made certificates can be added as trusted resources for individual users and computers.  Their primary use is generally for testing purposes, code-signing, and other unique situations. 

Create certificates in three easy steps:

1. Download MakeCert:  Use the Microsoft tool called MakeCert to generate the certificate.  Microsoft includes MakeCert with their Windows 8 Software and Development Toolkit.
   
   *Hint, choose the Windows App Certificate Kit to avoid downloading the entire SDK.
   
   
   
   MakeCert is also available for download on my OneDrive.
2. Create a Certificate and Private Key: 
   
   a.) Makecert.exe is located in the c:\Windows Kits\8.0\bin\x64 directory by default.  Feel free to copy it to a different location.
   
   b.) Run makecert.exe with the following extended options: 
   
   C:\scripts>makecert -n "CN=Self Sign Cert" -a sha256 -eku "1.3.6.1.5.5.7.3.3" -r -sv root.pvk root.cer -ss root -sr localmachine
   
   N.B., -n=Subject Name; -a=digest algorithm; -eku=Enhanced Key Usage, OID; -r=self signed; -sv= Subject PVK file; -ss=Certificate Store Name; -sr=Certificate store location.
   
   d.)  The above command requests a new private key password. 
   
   
   
   e.)  The new private key and certificate file are located in the same directory as the Makecert application.
3. Pair the Certificate and Private Key:
   
   a.) Run the following command:
   
   C:\scripts>makecert -pe -n "CN=PowerShell local certificate root" -ss MY -a sha1 -eku "1.3.6.1.5.5.7.3.3" -iv root.pvk - ic root.cer
   
   b.) The above command requests the private key password.
   
   
   
   c.) The certificate pair automatically imports into the Windows Certificate Store.  This is confirmed using the MMC Certificate Snap-In.
   
   Start → MMC → Add/Remove Snap-In → Certificates → Personal 
   
   
   
   

That's It!

References:

http://blogs.msdn.com/b/kaushal/archive/2013/06/13/working-with-wild-card-certificates.aspx
http://kreelbits.blogspot.com/2013/09/deploying-signed-powershell-scripts-in.html
http://technet.microsoft.com/en-us/magazine/2008.04.powershell.aspx

By Steven Jordan on 4/16/2014.

*Update:  This issue is related to Winodows Update KB2506143 and KB2506146 (i.e., WMF 3.0).  Uninstall these updates from DPM 2010!

Problem Statement: The DPM server takes a long time to logon. It can take 15 to 90 minutes to logon after the server restarts. Additionally, Windows updates fail and rolls back to its previous state after the server restarts. The integrity of system backups and restorations are at risk because DPM server has become unreliable.

Additional Symptoms:

  a.)  Expired recovery points are not removed per DPM policy goals. Roughly half of the protected members show excessive recovery points in the DPM console:

Figure 1. Example of a protected member with 20478 recovery points

  b.)  Protection groups show excessive volume size. For example, the Exchange protection group indicates the recovery point volume consumes nearly 2TB.

  c.)  PruneshadowcopiesDPM2010.ps1 is a DPM PowerShell script that removes expired recovery points. The script hangs and does not remove expired recovery points.

 d.)  DPM console hangs when deleting inactive protection group members. The GUI is unresposive and must be manually closed.

 e.)  The registry System file has bloated to over 220MB. System is located in c:\windows\system32\config\.

Figure 2. Bloated registry.

Root Cause:

   There were excessive disk based recovery points (i.e., VSS volumes). In our case, DPM (or Windows) had improbably kept tens of thousands of recovery points per proctection member. DPM, by design, is only supposed to store up to 64 recovery points for its file members, and up to 448 recovery points for its application (e.g., SQL database) members.

   The problem did not affect every protection group member. Some members (e.g., recent additions) that had less than 100 recvery points. However, nearly half of all the protection group members had excessive (e.g., over 20,000 ) recovery points (Figure 1).

   The excessive, or rather expired, recovery points had to be removed. Normally, DPM automatically removes expired recovery points with its PruneshadowcopiesDPM2010.ps1. The default script was not working so I turned to a custom PowerShell script named PruneVSS.ps1.

   PruneVSS.ps1 is a handy tool that removes disk based recovery points based on date. Its interactive session determines protection groups and recovery point date ranges. N.B. The script was originally written by the late, Ruud Baars.

  I had mixed success with Baars' script. It worked great on resources that had less than 8,000 recovery points. The script hung indefinetly for protection group members with more than 10,000 recovery points. The situation required extreme measures.

Inactive Protection Group Members

   The final option nixes the remaining protection group members that continue to retain expired recovery points. This DPM nuclear option removes all disk based recovery points by deleting their associated volumes. It is imperative to plan for continuity before committing. It's best to ensure the secondary DPM server has backups of the primary protection groups and to make a full tape backup before proceeding.

  The afflicted protection group members were transitioned as inactive protection group members. I then attempted to remove the disk based recovery points using the DPM console. Unfortunetly, I had limited success using the GUI. I was able to remove the disk based recovery points from a few of the inactive members. For the majority, however, the console simply froze. At this point, I turned to a second custom PowerShell script, named removeinactivedatasource.ps1. This script was a life saver -it removed all remaining disk based recovery points. I ran the script in verbose mode, so I could see its progress. It took about two hours to complete its job.

   I then moved the inactive protection group members back to their original protection groups. N.B., the recovery points must be deleted before re-adding them to their original protection group members; otherwise DPM will continue to use their originally assigned volumes.

   The next day recovery points looked great; less than 100 for each member in the DPM console. DPM's PruneshadowcopiesDPM2010.ps1 also ran without problems. I had high hopes that the problem was solved -except that DPM continued to hang after restarting it. Victory was short lived.

Secondary Cause

   I had won a battle but not the war. Efforts to fix the recovery point volumes were successful but its cure exposed a secondary sickness: phantom VSS volumes.

  I was fortunate to discover a handful of blogs that had somewhat similar DPM problems. Microsoft explains some of the symptoms in KB982210:

This issue occurs because there are a large amount of orphaned registry keys.

The Volume Shadow Copy Service (VSS) snapshots create many registry keys. However, they are not deleted after the VSS snapshot operations are completed. 

Indeed, the DPM's registry system was bloated with nearly 15,000 VSS volume registry keys.

Fig 3. Registry bloating from VSS Snapshots

   Scott Forsyth's Blog recommends applying the hotfix from KB982210. The hotfix however, cannot install on a DPM server unless it runs Hyper-V! In fact, most of the focus for this problem centers on Hyper-V backups -but my problem has nothing to do with Hyper-V. Even if I wanted to install Hyper-V, to allow the hotfix installation, the server was in no condition to install a new feature; all updates failed upon restarting the server.

   In our case, DPM uses iSCSI disks for the replica and shadow copy volumes. The alternate approach removes the phantom devices via script and then requires a second tool that shrinks the registry. Both Forsyth and Gary Fenton, recommend running the Microsoft tool called DevNodeClean to remove phantom devices from the registry.

DevNodeClean 

   DevNodeClean is available from Microsoft support or it can be compiled with Visual Studio per KB934234. Fenton also has a complete version available for download on his blog.

   I ran DevNodeClean and it indeed found orphaned devices -a grand total of 7. It was less than the 10,000 I had expected. The reason DevNodeClean did not work in this instance is because it only checks for orphaned devices on disks, partitions, and volumes; It does not check for phantom volume shadow copies.

I described the problem to a talented programmer, #SAK, who works at my office. He reviewed DevNodeClean and further developed it so it checks for orphaned VSS volumes. SAK explained his program lists all orphaned VSS volumes from the command prompt: c:\cleanup.exe.

The program removes can remove all orphaned VSS volumes by including a switch: c:\cleanup -r

Success! The SAK cleanup application found and deleted nearly 10,000 orphaned VSS volumes from the registry.  Download the SAK Cleanup tool from my OneDrive.

   N.B., David Candy's Blog has a good alternative to SAK's custom application. The modifed RmHidDev.bat also finds and deletes orphaned VSS shadow volumes.

Tertiary Problem (i.e., third time's a charm):

 The crazy slow logons remained; even after all the expired recovery point volumes were deleted; and all the orphaned VSS volume registry keys had been removed. Gambit's blog explains that DPM's problems persist because of its bloated registry. I confirmed the registry size had not changed:

Fig. 4. Bloated registry causes log on profile and update issues.

 Microsoft support provides a tool that shrinks the registry, called Chkreg.  N.B., Chkreg is only available by contacting their support team.  Chkreg is also available for download from my OneDrive.  The tool is easy to use; the process is somewhat tedious. Essentially, Chkreg cannot fix the system file while the server is operational.  The server must be turned off and the disk must be accessed using a separate method.

  I shut the server down and used the Windows 2008 installation media to boot into the recovery mode command line.  I then used the recovery command to navigate to c:\windows\System32\config, and copied the system file to a separate location. N.B., the drive letters in the recovery command were different from what Windows normally uses. FDISK provides current assignments with its list disk, list partition, and list volume commands.

  I removed the Windows CD and re-started the server (and waited an hour). When the server was back up I used the chkreg tool to repair the copy of the registry system. I issued the following commands:

   #Chkreg /F SYSTEM /R
   #Chkreg /F SYSTEM /C

  The new system file was significantly smaller than the original. The system file shrank from 219 MB to approximately 140 MB. I admit, I had hoped the new file size was closer to 10 MB, but at least there was some progress.

  Once more, I restarted the DPM server, and accessed the recovery command prompt with the installation media. I moved the original system file to a new location -as a precaution. I then copied the new (i.e., shrunken) system file back to it's original location, c:\windows\system32\config. I restarted the server and waited for DPM to come back online.

End result -it worked!  I can finally log onto the DPM server in less than 30 seconds.  Shortly thereafter I installed a year's worth of updates. Everything installed OK and the server remains trouble-free.



References

http://virtuallyaware.wordpress.com/2010/06/18/hotfix-kb982210-windows-server-2008-r1r2-hang-at-logon-bug-deep-dive/ 
http://garysgambit.blogspot.com/2010/05/windows-2008-hyper-v-vss-backup-bug.html
http://weblogs.asp.net/owscott/archive/2011/02/02/slow-boot-from-massive-registry-on-a-hyper-v-server-fix.aspx 
http://technet.microsoft.com/en-us/library/hh757783.aspx
http://blogs.technet.com/b/dpm/archive/2008/06/11/cli-script-to-remove-all-datasources-in-inactive-protection-state.aspx 
http://blogs.technet.com/b/dpm/archive/2011/03/21/rest-in-peace-ruud-baars.aspx
http://virtuallyaware.wordpress.com/2010/06/18/hotfix-kb982210-windows-server-2008-r1r2-hang-at-logon-bug-deep-dive/http://dcandy.wordpress.com/
http://blogs.technet.com/b/dpm/archive/2011/05/04/easy-dpm-2010-fix-disk-based-recovery-points-are-not-deleted-as-per-retention-goals.aspx

Problem:  BranchCache services do not work as expected.

Background:  BranchCache services are installed in a hub and spoke topology; data center serves multiple branch offices.

File servers at the data center are enabled as BranchCache content hosts.  BranchCache hosted cache servers set-up at each branch office. 

Troubleshooting: There is not a single pane BranchCahce GUI for management and monitoring -too bad!  The following steps confirm if BranchCache works:

1.  Confirm BranchCache is properly configured and operational on the servers and clients:

PowerShell:  Get-BCStatus

CMD:  netsh branchcache show status all

2.  Use Performance Monitor to check for real-time BranchCache activity (or lack of):

a)  Open PerfMon.



b)  Add the BranchCache counter.




c.)  Change graph type to "report".  The statistics will increase in real-time when any BranchCache file transfer takes place:



d)  Transfer data.  If BranchCache is working, the above highlighted numbers will increase in real-time.

Solution:

I missed the elephant in the room.

The problem was the specific version of Windows 8 used for the tests.  Windows 8 Enterprise must be used when working with BranchCache; not Professional.

This may seem obvious to anyone that has worked with the 2008R2 and Windows 7 BranchCache; however all the prerequisites from Microsoft simply read, "Windows 8" whereas Microsoft specifically indicated, "Windows 7 Enterprise".
References:

http://social.technet.microsoft.com/wiki/contents/articles/14309.branchcache-frequently-asked-questions.aspx

http://technet.microsoft.com/en-us/library/jj590747.aspx#bkmk_pre

http://technet.microsoft.com/library/jj572990

Last updated  March 30th, 2014 by Steven Jordan