As VP of Engineering for a small company, it is not uncommon to have some sales guy ask if we can do something “custom” for a given prospect in order to close a deal. On this day, though, Tom, our VP of Sales, came to my door and in his brusque, New York style said: “Just got off the phone with my buddy at Citi. Whaddya want first, the good news or the bad news?”
When faced with this choice I always go for the bad news. He said: “Doesn’t make any difference what you want. The good news is the bad news. He wants to buy 1,000 seats. Have a nice day.” And he walked away, confident he had done his job and frustrated in the knowledge that this was a fool’s errand.
Tom knew our client-server product well enough to understand that, at the time, at about 700 simultaneous users performance became an issue. Like all our competitors, our clients polled the server for changes and between database queries and networks of the day, going beyond that scale was difficult. It had not yet proven to be a problem competitively because we were more efficient than our competitors, who stalled out even earlier. Most of our sales were in the 50-300 seat area, so this was a huge opportunity for us. But it was also unreachable.
A couple of days later, coincidentally, a friend of a former colleague asked for 30 minutes of my time for a sales demo, and, as a courtesy, I agreed. The product was sort of a combination messaging/task management tool. I had zero interest in it and was about to cut the demo short when he started to explain the architecture. He described a push-oriented architecture that was based on IP multicast. A light bulb went off. I interrupted the demo, andconfessed that I really had no need for the product, but wanted to know more about their underlying architecture and whether they had an SDK for the multicast component. They did not.
It was clear that we needed some way to reduce the both the number of database queries and network traffic to scale our product to the next level. The vast majority of our customer installs consisted of single networks with only a handful of remote clients. I spent the next couple of days researching publish/subscribe architectures and technologies including multicast and the available implementations. By two weeks later I had enlisted one of the senior engineers in my group to help me either invalidate the concept or demonstrate its feasibility in our context.
We did some back-of-the-envelope estimates of the potential scalability advantages that a switch to pub/sub, combined with multicast, might bring.
If we were right, this would be both a competitive game changer for the company and a very large architectural change for our product. We created a prototype and ran the numbers. It became clear that this change would likely improve our scalability by nearly an order of magnitude. This needed to happen but, re-architecting a product that was over 600K lines of C++ code without disrupting the existing sales cycle is, as they say, a non-trivial task, particularly for a relatively small engineering team.
So we broke down that order of magnitude improvement into multiple phases and devised a plan that delivered incremental customer value over several releases. Each release would provide some portion of the scalability benefits. First, we created a shared cache, through which multiple clients ona given LAN could connect to the server. After that, would be a client-specific caching component, to reduce latency further. Next, we released a pub/sub component to update those same client caches. Finally, we delivered the multicast component that updated every client on a given subnet with a single call, whenever the database changed.
With each of these updates, the maximum size of a feasible installation increased. We were able to deliver the Citi deal within six months.
Ultimately, it took three years to deliver on the complete vision that derived from that multicast demo. But every six months we pushed out a new release that scaled significantly better than that the last, continuing to outpace our competition. We won and successfully implemented deals, for the first time, in the multiple-thousand seat range. At one point, we displaced IBM, the incumbent at EDS. After they offered EDS a free, unlimited license for the entire enterprise, EDS told them, “Sorry, even for free your product now costs us too much.”
As a result of taking this incremental approach, emphasizing the delivery of customer value with every release, we were able to build our installed base, increase average deal size, and over the three-year development cycle nearly double overall revenue, while never disrupting existing customers
Read more about Advantary Advisor Alan Kucheck here.