I will be curious what kind of dimensions and sub-dimensions you guys will eventually develop: business model, customers, product, team, finance, etc. There may be different ways to measure each dimension; for example, with teams, it seemed to me that you are currently measuring the number of people at the firm. Is that the right dimension? Do you assume that the more people you have, the better you are? How about quality of each person? How about the chemistry of those people as a team? So there is a highly qualitative aspect to each of these aspects.
Yes, of the 5 dimensions the team dimension is probably the most difficult to measure. Here is, in more detail, how we are approaching it:
First of all, there are many different ways we can conceive of measuring team and a few ways we can easily do it with our current data entry method based around surveys.
Currently these are the core sub dimensions we see: (Team Size, Organizational Structure, Team Composition and Culture). We have decent first stabs at measuring all but culture, which I haven’t learned how to evaluate yet. Measuring the quality of each person and the chemistry of the people I’d argue is a team dimension at level 3 depth (Team > Team Composition > Talent / Chemistry), which means it will take us awhile to get to researching it, as I believe there are diminishing returns going deep in anyone one dimension without a sufficient foundation in all.
We don’t assume that the more people a team has the better. To evaluate whether the size of the team is appropriate we have to measure the stage of the market response (Discovery, Validation, Efficiency or Scale). Depending on where the market response is classified the size of the team would then be classified as Scaling Dysfunctionally, Scaling Properly, or Prematurely Scaling. (Not enough people, the right amount of people, or too many people, respectively).
Another important point to make is that research into how to improve team is a few steps removed from the key question for startups: “Am I making progress?”. And any experimentation with new methodology requires this information to be able to truly tell whether it is an improvement over the status quo. For instance, if a team switches to using design thinking methodology religiously how do you know that it has actually improved performance, and that it hasn’t actually decreased performance but design thinking just makes the team feel more productive?
The solution to this problem is that we first have to be able to measure what is only one step removed from progress. In our view, progress ultimately depends on the response from customers in a market. The endogenous dimensions that a startup can control to directly influence the response of customers in a market are Customer Development and Product Development. Any improvement in the Team dimension would “trickle up” and reveal its effect as better performance along the Customer and Product Development dimensions. For instance, startups that move from waterfall development to agile development could show increased iteration speed, translating to greater customer satisfaction.
This means that our internal focus is nailing our ability to measure those dimensions first before focusing on the team dimension. But if you have any new assessments you’d recommend we look into for the team dimension, let us know, so we can start thinking about how to incorporate it now.
Science vs Art / Intuition of VCs
Another way to put this is to think like venture capitalists. Various research has shown that the evaluation criteria of VCs are highly qualitative and tacit, i.e. cannot be expressed in explicit indicators (or dimensions as you call). It’s something about the person (entrepreneur), the team, but little about the business plan, for instance. What your dimensions may capture could be the initial selection process for VCs, but by the time VCs interview entrepreneurs, they are trying to evaluate everything between lines (of your dimensions). And even after this rigorous selection process, many firms invested by VCs still fail. How do you think your benchmark and dimensions can go beyond that?
We haven’t done much research on trying to integrate the Startup Genome conceptual framework with the status quo of how VC’s make decisions, and that could be something to look into further. However here is what I think we can say:
It’s hard to measure the intangibles, but these intangibles translate to measurable tangible metrics (the 5 dimensions) very quickly.. Once a startup releases a minimum viable product, we will be able to gather data across all dimensions. Our hypothesis is that things like speed through the developmental stages will be a strong predictor of success. We’re just getting enough longitudinal data to test this now.
Another way of saying that is, we try to measure performance, rather than perceived potential performance.
It’s also not clear VCs are any good at picking up signals in the intangible noise between the dimensions. I’d bet that many of the winners VCs pick show strong performance along our measurement framework and that VCs are not much better than random when they are drawing on only signals we can’t measure, but they can allegedly access with their golden gut.