Exchange 2010 Hybrid Wizard Fails

I have been working on a project recently and setting up an exchange hybrid deployment for 1400 users.I ran into a lot of issues with the wizard and have noticed a lot of people on the web have had some problems as well. So I thought I would post the most common issues and how I resolved them.

This is the default directory for the hybrid wizard log files
C:\Program Files\Microsoft\Exchange Server\V14\Logging\Update-HybridConfiguration

1. "" isn't a valid SMTP domain - This is when there is still a self signed certificate that has iis & smtp services applied to it Delete the certificate and ensure iis & smtp are assigned to a trusted certificate and rerun the wizard.
   



2. An unexpected result was received from Windows Live. Detailed information: "InvalidUri InvalidUri: Passed URI is not valid.".
   
   This is when the federation gateway services have blocked the domain you are trying to federate.Your DNS TXT confirmation record shows up fine when you query it on mxtoolbox.com txt:yourdomain.com. So you need to escalate it with Microsoft and get it white listed. The unfortunate part for me was that the particular domain that wasn't trusted was the external facing domain on my hybrid server and had the domain name linked to my wild card certificate and because of this the wizard would not complete.
   



3. ERROR:Updating hybrid configuration failed with error 'Subtask Configure execution failed: Creating Organization Relationships.

Now the last error is the most common one which I have seen people having problems with on the web and drove me crazy. So i just manually created the onpremise and online organisation trust.The hybrid wizard even though it fails , still manages to create the "On Premises to Exchange Online Organization Relationship" and it will populate the autodiscover endpoint of your office 365 tenant.

So now browse to the federation tab on your onpremise organisation and take note of the application URI

Now browse to the Online tenant and create a new organisation relationship and call it "Exchange Online to on premises Organization Relationship" Then manually enter the settings as per the image below. Adding in your already federated domains. You can simply add the domain that is blacklisted at a later stage. Enter the application uri that you previously took note of and finally enter https://autodiscover.yourdomain.com/autodiscover/autodiscover.svc/WSSecurity and this will be the domain that can access your hybrid server and the wild card certificate.

Enable free/busy on both the onprem and online organisation relationship.
Then add in a send connector and add the default onmicrosoft.com domain. If you were transitioned from bpos to office 365 add in that domain as well.Add in the smart hosts as per the image below and your source servers.

Then create a receive connector and only allow email from the following ip addresses and ranges. These addresses may change pending on what part of the world you are in.

Lastly you need to enable the mailbox replication proxy service which you can do so by running this command in the exchange management shell

Set-WebServicesVirtualDirectory -Identity “EWS (Default Web Site)” -MRSProxyEnabled $true –MRSProxyMaxConnections 100

This will enable the MRSProxy correctly in SP2. If you have changed the timeout values of the data move then you will again need to go to your web.config file and update the timeout value again. If you are not familiar with this it is the timeout value of the MRSProxy when performing a remove mailbox move. When you are performing bulk migrations of users to Office 365 it is a good idea to increase this so you don’t get failures during overnight data loads if you are using virtual machines for the Mailbox or CAS roles.Open the web.config file located in C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\exchweb\ewsGo to the bottom of the file and locate the new smaller MRSProxy section and make the change shown:

So this worked for me , I could see free busy between onprem and an online tenant, and move mailboxes between onprem and office 365 even though the hybrid wizard had been failing.

DPM 2010 Server Processor running at 100%

I recently fixed a problem for a customer whereby their DPM2010 server was very slow to do anything. Even opening up the GUI was taking a long time.

The server in question was a HP Proliant DL380G5.

So to fix it I needed to do 2 things.

1. Update the storage controller's firmware and driver.
   
   
2. As per the image above , login to the msdpm2010 instance and change the value in the red circle to 300 and reboot server.

Problem solved :)

Broken Hyper-V AVHD Chain recovery

I have been helping people around the world recover data from Hyper-V avhd chain disasters since 2009. I have helped people in a lot of countries and cities , some of which include.

Newyork

California

Seattle

New Hampshire

New Orleans

Germany

Paris

London

Sweden

Belgium

Netherlands

Australia

Singapore

Korea

Russia

People find me for help via the following forum posts.

Technet

Seanofarrelll.blogspot.com

seanofarrelll.blogspot.com

1. Snapshots are not for production servers. They are not backups.
2. Never delete a snapshot in a chain.
3. If you delete a snapshot via hyper-v console , wait for it to merge back into it's parent.
4. Use Microsoft DPM 2012 and hardware vss writers where possible.
5. Never increase the capacity of a base vhd when it has a broken avhd chain.
6. Some recommendations on vhd recovery are visible on my linkedin profile
7. Windows Server 2012 will make it easier to recover from broken snaphot chains.

Exchange 2010 SP2 RU1 & RU2 breaks OWA

Exchange 2010 Service Pack 2 Rollup 1 has broken some cas servers. I know it sounds crazy that an update direct from Microsoft would do this.

I have been working on a number of Exchange 2010 projects recently and both projects were using hardware load balancers. So i started to test the internal owa url on all of my cas servers and on some of the servers - the internal owa url wasn't working. So if that isn't working then OWA definitely wont work on that server. So to fix it is quite simple.

On the server that is having a problem. Delete the themes folder from both of these locations.

C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\14.2.283.3
C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\14.2.247.5

Once they are deleted then copy and paste themes folders from a healthy cas server back where they should go on the problematic server.

If the above steps dont work. Go to a server that is working ok and export the following regkeys and then add them to the server that had the problem and re-apply the rollup and problem will be solved.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.Exchange.Management.PowerShell.E2010]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.Exchange.Management.PowerShell.Setup]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.Exchange.Management.Powershell.Support]

Tier 1 Bpos to Office 365 Transition

I recently completed Ireland's first Tier 1 Bpos to Office 365 transition and would like to share the steps involved to achieve this.

My client was a global charity with their head quarters in Dublin and operating in 14 countries with 1400+ users spread over 90 locations and most with little or no it support.

As users were so dispersed throughout the globe the charity did not want to implement Single Sign On. One of the most critical requirements for the charity was the need to keep existing ost files on client machines. The reason for this is because some locations in Africa have poor Internet connectivity speeds and it could take weeks for the creation of a new Outlook profile to download and re-sync a local cached copy of a mailbox.So I confirmed with Microsoft that we could keep the existing OST's and they said we could. However the Office 365 Client Prereqs needed to be installed on each client machine prior to transition.

So I am going to bullet point in order the steps.

• Engage with a Microsoft Office 365 transition manager



• Remove Office Communicator prior to transition - this can be done running this command or a batch file MsiExec.exe /I{0F3AB690-1F39-40B8-9D4A-6E8DDA850FB0}/passive
  


• Once that has been done install Microsoft Lync



• Then install this UPDATE on all Client computers
  


• Send out a communication to all staff stating that after transition they can access their web mail via https://portal.microsoftonline.com and smart phones can access m.outlook.com



• One week before transition , setup Lync SRV Records for each domain as per Microsoft's GUIDE and internal firewall rules
  


• One week prior to transition , reset user's passwords so that they comply with Office 365's password policy I did this very quickly and easily via Messageops powershell gui for Bpos
  
  


• Then run the powershell below to ensure any mailboxes with delegated control preserve their custom permissions

Export Public Delegates

#$LiveCred = Get-Credential

#$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection

#Import-PSSession $Session
get-mailbox -filter {grantSendOnBehalfTo -ne $null} select userprincipalname, grantsendonbehalfto export-clixml delegates.xml

get-mailbox -filter {grantSendOnBehalfTo -ne $null} select userprincipalname, grantsendonbehalfto export-csv delegates.csv

Import Public Delegates
#$LiveCred = Get-Credential

#$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection

#Import-PSSession $Session
$logfile = "log-" + (get-date -uformat "%H%M-%Y%m%d") +".txt"
start-transcript $logfile
import-clixml delegates.xml foreach{
"User: " + $_.userprincipalname
foreach($i in $_.grantsendonbehalfto){
"GrantSendOnBehalfTo: " + $i
set-mailbox -identity $_.userprincipalname -grantsendonbehalfto @{Add=$i}
}

"-----------------------------------"

}

"number of mailboxes with grantSendOnBehalfTo : " + (get-mailbox -filter {grantSendOnBehalfTo -ne $null} ).count

stop-transcript

• During transition the external DNS records for autodiscover can be edited. So if you have a domain named contoso.ie you would create a CNAME record called autodiscover.contoso.ie and point it to autodiscover.outlook.com



• So once the transition has been complete, a user can still sign into the single sign in utility but Outlook will not be visible. The user can then open Outlook and will get a warning stating " An administrator has performed maintenance on your Outlook profile, Please restart outlook" So once the user restarts Outlook , the user will then be prompted for his/her user name and password.



• Some Outlook clients may not want to connect to Office 365 via autodiscover and if that happens simply configure the Outlook profile via this website config.365.com another great site from messageops.



• Once Outlook can open and close without password prompts you can remove the single sign in utility by running this command MsiExec.exe /X{A91E3887-5185-4091-AF33-AB0048444055} /passive
  


• I then wanted to chat to the charity's ICT manager so I enabled external federation by doing the following steps.
  
  Click "Manage" from Lync Online under Admin page
  If you are using E account, you will see current setting for Lync online for management page. If Domain federation or Public IM connectivity was disable under Current settings section, please enable them first.
  Click Domain federation: Select "Allow federation with all domains except those I block"
  Click Public IM: click "Enable" to active Public IM.
  After your enable federation, then you can see the External Access for particular user when editing setting

Ok so there are the technical steps and what is next for the charity. They were waiting for Office 365 before they did some customized development of Sharepoint online and now they can start this work. They absolutely love Lync and can easily communicate between 14 countries.Of the 1400+ users , there was an issue with 2 users , one in Dublin and one in Sudan!

My next blog post on Office 365 will be around the design and implementation of a ADFS 2.0 Farm which can tolerate one Active Directory site failure of a multi site Active Directory and still allow users to authenticate.

Implementing Kemp Load Balancers with Exchange 2010 Sp2

I recently implemented a 2 node Kemp Loadmaster 2200 array and I generally followed Henrik Walther articles on Msexchange.org. There are 2 articles

Load Balancing Exchange 2010 Client Access Servers using an Hardware Load Balancer Solution
Uncovering the new RPC Client Access service included with Exchange 2010

There are a couple of steps in the articles which have changed with Exchange 2010 SP2 and newer firmware on the Kemp Loadmasters which I can highlight below to help anyone out.

1. Instead of manually editing and adding registry entries for rpc static ports you can download and run a powershell script HERE from Bhargav.Some of the registry locations are different in Exchange 2010Sp2 and you dont need to edit Microsoft.exchange.addressbook.service.exe.config
   
   
2. The latest revisions of Kemploadmaster firmware do not include the persistence type 'active cookie or Source IP' the reason for this is because Kemp are phasing this persistence method out in favour of Super http. However if you would like to enable it you need to do the following.
   
   Go to logging options, debug options , enable l7traces then contact Kemp and they will give you a frame number which you can enter and it will allow 'active cookie or source ip persistence'
   
   
   
   
   
   

Data Protection Manager 2010 Overview Poster

I have had the pleasure to work Steve Buchanan and Yegor Startsev on compiling a Data Protection Manager 2010 Overview Poster.

Check it out on Technet Gallery