Algorithm development for early stage products

There is a debate between two approaches to algorithm development. I call the first approach “research-oriented”. Here, an algorithm researcher builds an algorithm in a lab environment in MATLAB or Python. Then a software engineer converts the algorithm into production code that can be deployed on the company hardware, such as cloud, mobile, or PC. I label the second approach “full stack”. Here, the algorithm developer is responsible for all the stages of the research, development, and deployment of the algorithm.

I have been working on a demo during the last three weeks. I found myself dealing with tons of technical problems that are not related even remotely to the core algorithm. There was a moment when I started pondering whether it would be more efficient for the company to hire a software engineer that would help the algorithm team to concentrate on its job. And then I understood that the demo would not be possible with such an engineer. The research approach is not only inefficient, it is counterproductive for start-ups. Below I elaborate on why.

The main advantage of the research approach is “specialization”. When everyone does what one knows best, the final result is definitely better. A researcher can spend more time on research questions when she is not “bothered” by tons of various seemingly unrelated technological problems. However, as the most specialized species are not the best survivors in the evolution game, the over-specialized companies are not the best survivors in the start up game. There are three main reasons:

Adaption in ambiguous environment.
When the product is clearly defined, an engineer can concentrate on developing her part. However, in many start ups the product definition changes together with R&D process. As the company matures, the team begins to understand better the business and the technical limitations, thus altering the initial product.

An algorithm engineer is required to be in full control of the whole pipeline so that she can quickly adapt it to the changing requirements. Without the full control, it is virtually impossible to react fast enough to incoming challenges.

Complexity
The technology is complex and is becoming more even complex. In an ideal world, a python research code can be seamlessly re-written into iOS real time library. However in real life, no one has enough time to rewrite code from scratch. The trick is to utilize existing third party solutions as much as possible. The solutions may work great by themselves, but when connected together, they may cause weird bugs that cannot be figured out by someone who does not understand the whole package.

Resources
Resources are scare everywhere, both in a tiny company and a huge corporation. But start up success is all about management of resources. According to the popular Lean Startup theory, hiring more people than needed to complete the MVP is a waste of resources. Specialization cannot help to make products faster. It can only help to make them better. And this is a waste.

Summary
To summarize, I think that an early stage product developed by a start up requires an algorithm engineer capable of researching, developing and integrating the algorithm into a full product.