My journey to unified communications over 13 years

I have not commuted to a corporate office for about 13 years. This isn't because I've been self employed, unemployed, or working as a consultant or sales traveling to my customers full time. This is purely enabled because of the technology I've had available to me and the willingness of my employers (yes multiple companies) to not care where I live. During those 13 years I moved 7 times... sometimes to another state.

I started this whole notion while I was a network manager at a University. The technology at the time was largely enabled by modem. The first step was hooking up a modem to one of our Windows NT servers and then I was able to manage the servers and network from anywhere I had a phone line. For a while it was largely an IT only thing, but I was the one that mostly used it. This meant I could totally work from home some days, but unfortunately with the type of work I did, which was just about everything IT related, I needed to be at the users computer sometimes.

Fast forward a couple of years and I was offered a job I couldn't refuse from Nortel. I lived in Lakewood, CO at the time and the manager that hired me lived in Hartford, CT. I asked him during the interview about location and working arrangements and his expectations. His reply in 1999 was a dream come true... "As long as you have a phone line and an airport nearby, I don't care where you live". Think about that statement even today. That kind of attitude toward hiring employees is unheard of.

One of the things that annoyed me fairly quickly with this working arrangement is that I had no idea when my manager was available... and working for Nortel he spent most of his days on conference calls either finding new jobs or working through issues on current jobs. I came up with the idea to use AOL Instant Messenger to be able to ping him with questions or find out when he could talk. I knew when he was in the office or away from his desk. When we were on conference calls we could exchange critical information between each other out-of band (though IM) and become much more effective and professional. Little did I know... this idea would be so prevalent 10 years later with Unified Communications.

Even though I worked out of my home, I traveled to do work for Nortel because a lot of it still was hardware and wires.

Quickly after my manager recognized the benefits of asynchronous communications, the whole Professional Services organization under him was required to get AOL Instant Messenger accounts so they could communicate the same way. Yes they still used phones, pagers, email, and all the other tech available at the time. But this was the best way to have conversations with the guy at the top. When he had a technical question he would just go down his AOL friends list until he found an engineer with the skillset to answer it.

Fast forward a few years and I switched over to ISDN with a real Nortel PBX phone in my home office and when the phone wasn't being used I had 128k to burn! My communications were now unified... but only from a layer 1 perspective. Data and voice traveled down the same pipe... but they didn't know about each other.

In the middle of my years working for Nortel I moved to Duncan, OK (don't worry it didn't last long) But part of this move was getting access to DSL. I started to use VPN for access to Nortel, but Voice over IP was really not capable of going across the Internet at the time. So I was still stuck using the PSTN to make and receive calls. Nortel used technology from MCI at the time called VNET which allowed me to receive calls and make calls as if I was in a Nortel corporate office. VNET Worked pretty well at the time and I was able to move from Colorado to Oklahoma and continue working for the same employer and manager. I still had to travel to do work for Nortel and cell phones largely replaced pagers while we were away from our home office.

After about a year and a half I moved back to Colorado and switched to Cable modem, but for the most part the way I communicated when I worked didn't really change for about a year.

In the early 2000s (I don't remember when for sure). I moved into an apartment with a wireless ISP provided. Nortel at the time started to play around with a new technology called SIP. Up until this point when I worked with VoIP it usually involved H.323. The product for the enterprise at the time was called Succession MX. The enterprise product started out as a carrier IMS platform, but was quickly recognized to be something that corporations might pay for. I got up on the dog food trial inside Nortel and became instantly hooked with SIP. Instant Message, Presence, and Voice were built into the product and I realized immediately this was the future of communications. I showed it to my manager (which was still the same one from 1999) and he felt the same way. One of the key pieces of technology that made this solution viable was a packet loss concealment algorithm called Global IP Sound (eventually this was worked into most Nortel endpoints). Even though I frequently would lose packets on the wireless ISP I could hold conversations with others and rarely would notice the problem.

I managed to get some of the other engineers on the dog food trial, but could get them all so we unfortunately still did most of our communications through AOL instant messenger, email, cell phones, and the PSTN.

When the product finally shipped it became known as MCS 5100 and the Nortel IT dept deployed it for internal employees as well. My manager moved everyone over to it recognizing the benefit of having everyone on an internal system rather than using a 3rd party AOL to conduct company business. This is when I recognized that a great technology doesn't mean much until you have others you work with using the same technology.

Although Nortel did know how to build great technology, Nortel did not know how to market new ideas. The whole notion of Unified Communications was a foreign concept to every single business. When I was working on another product, the customer would always be amazed that I could start VPN and a single piece of software and start communicating as if I was in the office. When I told the sales people about the possible lead most of them had no idea what product I was talking about. When I would show the sales person the technology they were amazed... but rarely did I see any sales from these leads.

Although there were a handful of customer successes. The product never hit critical mass and if it was purchased it was mostly for its ability to handle large dial-in conferences (called MeetMe Conferencing) and the rest of the product was not understood or forgotten about.

Microsoft around the same time was working with Live Communications Server 2003 and 2005 and when the Innovative Communications Alliance was created in 2006 between Nortel and Microsoft I knew this was where I needed to be. I joined when I could find an opening and the first Microsoft UC product I was exposed to was Office Communications Server 2007 (R1). It wasn't hard for me to pick up this new product because the concepts were very similar to MCS 5100.

My world was still divided because for day to day communications I used MCS 5100, but when I did any deployments it was around Microsoft Unified Communications integrating with a Nortel PBX. This was the now defunct RCC with dual forking solution that was rarely understood how to deploy and feared by all. To this day, I know only three people in the world counting myself that actually fully understood how this worked. When setup correctly it really was the best of both worlds. But if it broke it was a complete nightmare to troubleshoot.

I did the RCC with Dual Forking deployments for a couple years and moved over to Avaya when they acquired Nortel Enterprise Solutions. I lasted three months before Avaya decided they didn't want any part of the Innovative Communications Alliance and laid off every Microsoft related employee I knew of. The vast majority of those people laid off from the Innovative Communications Alliance went to go work for competitors, mostly Microsoft Partners. These were engineers that had on average 10 years experience working with communications technologies and some of them were 20-30+ years working for Nortel. If I was the person at Avaya that made that call...looking back, I'd probably have regretted that decision.

So in 2009, I was kicked out the door at Avaya and I moved over to a Microsoft Gold Partner called Time2Market and started to deploy OCS and Lync to integrate with or replace PBXs. My first job was to replace 12 Avaya PBXs with OCS 2007 R2. Replace... not integrate.

Telecommuting wise I was now fully on Microsoft OCS 2007 R2 with Lync 2010 right around the corner. No more VPN. No more divided between a Nortel and Microsoft world. I expected I would travel at least 50% of the time for deployments and this was true for OCS 2007 R2, but as soon as Microsoft Lync 2010 became available, I rarely travel anymore. Most customers are content to work with me remotely. They don't have to baby-sit me... they don't have to pay for my travel expenses...when Federation is setup, working on a problem with them is a click away.

Without Federation and Remote User Access (with no VPN), you really don't have Unified Communications. You shouldn't have to be tied to a desk, or jump through hoops to get to where you can communicate. Unified Communications is about communicating with who you want, when you want, where you want, and how you want. I had all this in various different forms over the years... but they were not all in one single product and easy to use. Even my wife who hates technology uses Lync to communicate with me while I'm working (from her iPhone, with a mobile client).

I'm just waiting for the rest of the world to catch up to figure out they can move bits around instead of atoms.

Gotcha's for AT&T SIP Trunks

I've been involved in the deployment of AT&T SIP Trunks for a couple customers recently and a few Gotcha's have been consistent between them.

911
First off, do not expect 911 calls to work immediately. AT&T SIP Trunks take up to 24 hours to populate all the necessary databases with the numbers and addresses. Ask the account team to verify that all the billing addresses associated with your SIP Trunk are correct. AT&T made a mistake in this area and caused one of my customers to not have 911 service for 5 days while it was sorted out.

Referred-by Header
This is absolutely required for certain call forward and call transfer scenarios and is fairly typical requirement for SIP Trunks in general. The details for this configuration are contained in KB Article 2500421.

Transcoding
Calls to certain numbers fail with a 488 Not Acceptable Here and in the diagnostic header there is a "cause=65". These calls are completely fine when you dial from another PSTN device such as a cell phone. Most of the time I've found that this is another AT&T SIP Trunk customer that has only one codec they can negotiate and you have a different codec. AT&T does not do any transcoding (which makes zero sense to me). In order for the calls to work you need to make sure you have at least G.729 and G.711 codecs in your allowed list and order them for your preference.

Enjoy...

How to remotely view and test certificate, intermediate certs, and root certificate

There are a few instances during a Lync deployment (Edge, Mobility, EWS) when you need to see all the information about a public certificate, but you do not have easy access to the system that contains the certificate, intermediates, and root certificates.

I have found the help page at Digicert to give me all the information I need, even if the certificate is not issued by Digicert.

Normally you would just type in a server name and off it goes and queries port 443 by default. That works great if you are testing a cert on port 443. But if you have a cert of port 5061?

it is as easy as <Server Name>:5061

Digicert SSL Certificate Check returns the certificate with Common Name, Subject Alternative Names, and Issuer. Most importantly though, if you look further down you will see the entire certificate chain.

Having this information is especially handy if you are dealing with GoDaddy certs.

IM an Expert has been updated to version 1.5

Another Engineer at Time2Market was getting ready to deploy IM an Expert for a customer and mentioned to me that the Admin and End User docs were recently updated (March 8, 2012). So I pulled down the Admin doc and noticed it had version 1.5 on the title page. Also, on Page 13 is a section that highlighted the changes in 1.5.

Below I have highligted the changes on Page 13... having the bot running as a service is reason enough to upgrade. The other change that I found intriging was "Conversation Following". Enjoy!

Changes for Version 1.5

The latest version of IM an Expert replaces V1.0 of the system which shipped in March 2011. For the V1.5 release there are a number of feature additions and bug fixes.

New Features

1. Support of Resource Forests: Version 1.0 of IM an Expert did not work for organizations where Lync was setup with a resource forest configuration. Such a configuration has a set of users with information such as SIP address in a different domain/forest from the one that is their actual user account. So, when they visit the IM an Expert webpage, the site could not locate their information. To fix this, we improved the way we retrieved user SIP/Name/Email so that we search for the user in resource forests as well as the current forest. IM an Expert now offers a sophisticated highly-configurable mechanism to allow admins to specify how user identities are resolved from Active Directory. More information about deploying Lync Server 2010 in resource forests can be found here http://technet.microsoft.com/en-us/library/gg670909.aspx
2. Content-type Adaptation: If a message sent by IM an Expert cannot be received by the client (e.g., the message is sent as HTML and the client does not support HTML), then the system will automatically switch to plain text, resend the current message, and use plain text for the remainder of the conversation. The occurrence of the adaptation is governed by the SIP error code that IM an Expert receives when it tries to relay the first message in the conversation. The BotSipErrorCodesForPlainTextResend configuration parameter can be used to specify the list of SIP error codes for which content-type adaptation should be attempted.
3. Message Content-type Specification: Messages can be sent in plain text or HTML. This can be specified using the BotMessageContentType parameter in the imx.config file: “text/plain” for plain text, and “text/html” for HTML.
4. Lync Mobile and Outlook Web Access: IM an Expert will now work with Lync Mobile client (tested for iPad, iPhone, and Windows Phone), and Outlook Web Access (OWA).
5. Run as Service: IM an Expert can now be configured to be run as a service which can be configured to start automatically when the server hosting the Bot boots up. Note that the Bot can still be run as an executable.
6. Conversation Following: If a candidate answerer declines a question invitation they receive via IM, they now have the option of subscribing to the conversation by typing “follow” in response to the invite. At the conclusion of the dialog between the asker and another answerer (if one is found), all followers will be emailed a link to the transcript of the dialog on the IM an Expert website, where they can also add follow-up comments and additional answers.
7. Usage Statistics on Homepage: Administrators can now display usage statistics on the IM an Expert homepage, including a stacked bar chart of the ratings assigned to answered questions and other statistics such as the average time to answer and average number of users contacted to get an answerer.
8. Memory Management: The Bot should now consume less memory on the server on which it is running, especially in situations where the Bot is running for prolonged periods. To do this, we store less in memory, clear in-memory data structures more frequently, and periodically request garbage collection.
9. Support for More Sophisticated PageCrawler Regexes: The PageCrawler now supports multiple regular expressions for the same page and as multiple match groups for a page within the same regular expression. This allows administrators to extract multiple items from a web page via a single regular expression, multiple regular expressions, or some combination of the two.
10. Resetting Bot Status: Allows administrators to set the Bot’s Lync status to variable (“Online” or “Offline”, depending on whether the Bot is running) after being set to “Always Online”. In V1.0, once the Bot was set into an Always Online state, it could not be undone by IM an Expert. 

Bug Fixes

1. SIP Address Normalization: In V1.0, SIP addresses were not being normalized. In companies with uppercase domain names, this caused a non-match between user SIP addresses as far as we knew them (e.g., user@company.com and SIP addresses returned by Lync when we queried for presence status (it would give us user@company.com). This has been fixed so that IM an Expert normalizes all SIP addresses by lowercasing the sip: and the domain portions and leaving the username portion unchanged.
2. PageCrawler Configuration: Corrected errors with parsing of special fields such as EMAIL, SIP, and SIPALIAS, which are meant to be replaced by the current user’s email/SIP/SIP alias/etc.
3. Website Viewstate Validation: Occasionally, users would get a “validation of viewstate MAC failed” exception when using forms on the IM an Expert website. This has been fixed.

So.... You want to be a Lync Master?

I recently spent three weeks in Redmond, WA attending a Microsoft Certified Masters course for Lync Server 2010 and thought I would share some thoughts and tips based on my experience there.

LOST in Building 40
First of all, the class was amazing and better than I ever hoped it would be. 

Going to Lync Masters is like attending an IT version of LOST. When you attend training you are told to cut off communications (or at least limit them) and responsibilities with everyone in your work and personal life. Although necessary, this creates a island like effect. Once you arrive in Redmond... you are stuck on this island with other Masters attendees and slowly, over the course of three weeks, your previous life begins to seem more like a dream.

You now have this alternative life on the island (Microsoft Building 40 in our case) where you will form relationships and create alliances (study groups). If you are fortunate, like I was, you will find someone that is experienced in areas you are weak in.

After the first day you will feel like you have got this class nailed or at the very least can handle it. The second day you might still have the same feeling but not quite as confident. The third day the new information just keeps coming and coming non-stop and then the black cloud appears. You start to feel fatigue and if you are not careful... you will miss what the instructors are trying to teach you. Eating and sleeping well become very important at this point. There will be no break to fully recover until you get off the island.

When you start your second week there...you will not feel as on top of the game as you did the first day. The information and ah ha moments just keep coming... Things really start getting weird the second week when people start talking about dreaming of Call Admission Control. At the same time everyone in the class also is figuring out that it is really futile to remember "everything". This is when you should start to focus on what you and your study group thinks is important for the exam and qualification lab. Or for some people they start to think about how they are going to get off this island.

The third week now you are spending nearly all your waking moments taking in yet more information and when that is not happening, constantly going back over all the other content from previous episodes (uhh... I mean days) trying to keep all of it fresh in your head for the knowledge exam and then the qualification lab.

After being tested by the island(exam and qualification lab is done), you are now dealing with whether or not you are a Lync Master and you have to head back to your previous life.

For me, my real life seemed like a dream and the life back at Building 40 seemed like reality for at least a week.... probably longer. Life will not seem right... or normal for awhile...maybe not ever. The wife and kids will wonder why you are easily overwhelmed with all their attention. In some extreme cases, you might come home and be speaking of smashing the lab (this would be if you did really well on the qualification lab)




Tips for the class...

Taking Notes
First of all, do not even begin to think about attending a Lync Masters course until you have become familiar with every single workload in the product (yes, even Group Chat, which I have a new respect for). The instructors from day one expect you have done this and they will not be covering the basics. As an example, there is a significant amount of time dedicated to the voice workload and if you have limited expereience in this area I would not use this class as a time to learn that.

For our rotation all the slide decks were delivered to us in OneNote. This makes taking notes extremely handy. I also added audio recording to my notes using the Jabra Speak 410.

What is great about this setup is that as you are recording and typing, OneNote will keep track of both. This will allow you to go back later and click on the specific text to hear the audio at that point when you were typing. When I had slides that had no notes I still put a letter "A" as a place holder so I could come back and hear the audio at that point. You will want to work with this before you get to day one of the class because at that point you will not have time to be messing with setting up OneNote to do this.

Make sure before you spend a lot of money on a microphone that you check with whoever is running that rotation, to be sure it will be permitted. I strongly suggest a USB microphone of some sort because any analog mic I used had a hum in the background (probably something with my laptop).

Also, it should be expected that you will encounter a lot of content that you have not seen before. Do not come to class and expect that you know most of the content. Be preparred to learn lots of new content.

Study Groups
This was probably the single best reason why I passed. Find people you get along with that are strong in the areas you are weak...and hopefully you are strong in areas the others are weak. My study group consisted of 3 people to begin with, but mushroomed to 6 by the second weekend. There were times we were going over previous material in the classroom late at night or on the weekend and others would just join in... and make it that much better.

One thing that really seemed to help was that one person in the study group suggested we write our own exam questions based on the content we had learned and share them with each other. Each of us wrote questions in areas that we were really familiar with. There were some really tough questions that pushed us to understand the content even better. But even more benefitial was we decided to write some of the questions on content that we were weak in to really help cement the knowledge in our own heads...  By the end of the second weekend we had a pool of 47 questions. We had errors at first, but we found that was helpful too. If we had someone that didn't understand why an answer was right we tried to help them understand why we thought it was right...and that led to more discussions.

Also, as far as study groups go, make sure to give each other space as well... a few nights/days throughout the three weeks is what we did. Everyone needs some downtime.

Eating and Sleeping
Above all else... eating and sleeping should be a priority. Class is hard enough without feeling terrible from eating too much junk food and not sleeping because you want to study.

Sleeping will be a huge challenge for those coming from Central and Eastern time zones or from overseas. Several in our class found they were waking up at 4 or 5am. Adjust as much as you can to get as much sleep as possible.

There will be junk food all around you. Lots of candy and chips... and Microsoft provides a variety of drinks free for the taking. Resist it as long as you can.

As far as dining out... I really thought I would not have time to do this often. So did several others in the class based on the amount of food that was bought the Sunday before class started for each of our hotel rooms. I bought some fruit and some other things to snack on... and I did fair...although I had a huge bag of tortilla chips and jar of salsa I hardly touched. My point is that you will want to get away from studying and class and feel human for little while. Below are a list of places my study group enjoyed.

Food Suggestions
The cafeteria at Building 40 was being renovated while we were there so Microsoft arranged for some Food Trucks to come by at lunch time. I was extremely pleased with the quality food and some of them accepted credit cards. You can get an idea what is available here (Cafe 41).

Jimmy John's
I've always felt these guys have great subs... others in the class that never had them before became passionate about them. They have online ordering available and even though they won't deliver to Microsoft Buildings the drive to get this piece of heaven is fairly short.

17875 NE Redmond Way
Suite #124
Redmond, WA 98052

Five Guys Burgers and Fries
If you have never had a Five Guys Burger and Fries you are truly missing out on something special. Just go. Be warned that for most people their "Little Burger" is plenty big. Also a table of four people can share an "order of fries".

15011 Northeast 24th Street  
Redmond, WA 98052

Spazzo
Do not judge this place by the name. Great italian food. We did have one waiter that was borderline stalking us... finding us in the restaurant every time we came in and remember everything we ordered the first time we ate there.

Redmond Town Center
16499 NE 74th St
E255
Redmond, WA 98052

Red Robin
Red Robin is known for their Grourmet Burgers and sandwiches. They have bottomless fries that are great as well.

Near Redmond Town Center
7597 170th Ave NE
Redmond, WA 98052

Near Building 40
2390 148th Ave
Redmond, WA 98052

Earls
The food is awesome and if you like to people watch this is the place to come.

700 Bellevue Way NE
Suite 130
Bellevue, WA 98004

Joey's
Great food and this was the place that some of us got together the Sunday before class started.

800 Bellevue Way NE
Suite 118
Bellevue, WA 98004

Azteca
Great Mexican that is close to Microsoft

3040 148th Ave NE
Redmond, WA 98052

Neville's (The British Pantry)
Fish and chips were great....  and the Shepherds Pie was well liked as well.

8125 161st Ave NE
Redmond, WA 98052

Claim Jumper
This place will give you the most calories for your buck (the food tastes great too). One Masters student finished off a Widow Maker Burger (1149 calories) with Fries (346) and a mini I Declair (1075) in one sitting. One meal that shocked all of us was the Ore Cart (2724 calories).

7210 164th Ave NE
Redmond, WA 98052



This post brought to you by a Jimmy John's #9 with hot peppers and a Shamrock Shake