🏡 Back Home

This is a public preview of The Small-Scale Research Guide. Currently in-progress. What's complete enough to read:

The Small-Scale Research Guide

0: The Purpose Of This Guide

I wrote this guide to help you create economically valuable research using constrained resources and beginner-level expertise in research. This purpose leads to a specific approach to research that I call small-scale research. As I wrote this guide, I had in mind the ambitious indie consulting business run by someone who has cultivated expertise in a dynamic or emerging un-commodified field.

The purpose of the small-scale research (SSR) aproach is enabling better decisions, often enabling better decisions by your clients as they are informed by your research output. It's likely that many of us will assume that "making better decisions" means "gaining predictive power over the outcome of the decision we're facing". In rare cases, that can happen. But that's also a too-limited view of the ways in which we can improve decisionmaking. Other ways would include but not be limited to:

  • Discovering new options we hadn't considered or been aware of.
  • Reducing uncertainty about the possible outcome (establishing probabilities).
  • Wasting less time considering options that are unlikely to meet our objectives or would represent bad or wasteful paths to those objectives.
  • Gaining a richer, more accurate understanding of the context – the system – within which we are making decisions.

These are all high-value ways of improving decisionmaking. One of the core assumptions built into SSR is that you can improve decision-making dramatically, but you can only create that value for one class of decision, made my a small, niche audiece, at a time. Each SSR initiative needs to be tightly scoped.

SSR also has an application that produces low or no decision-improving value, but high marketing value. This is SSR's social signaling value -- its ability to make you look smart. This is a totally legitimate use of SSR that I tend to discourage because it feels like a waste of potential to me, but you should not let my opinion here influence you too much. I mean this. Do be influenced by my guidance on how to conduct SSR, but consider me a cranky old man when it comes to my opinion on using SSR for its signaling value.

To my knowledge, there is no other single information source with this purpose and audience. That's why this book exists, and I hope you find it very benficial.

1: The Potential Of Small-Scale Research

@NOTE: Get the examples I have in front of readers early on in the book :)

This was merely the first example I could find of a particular genre that is well-represented on the Internet:

vivaldi_2J3S0446jt.png

You see what's going on here, right? The tweeter is mocking those who confidently espouse opinions on Twitter without institutional backing for those opinions.

This is a simplification of a more complex problem:

1: Deep genuine experts really don't want their expertise undermined by folks farming likes and shares on social media platforms.

2: It is actually possible to mis-use tools we don't have the proper training to master, and that mis-use can lead to harm.

So @deisidiamonia is -- in their own cartoonish like-farming way -- acting in defense of the honor and social position of real expertise. This is a big part of the cultural backdrop we consider when we think about small-scale research.

If I buy a $5000 vintage Martin guitar to play 3-chord songs on the weekend, nobody's really been harmed by my usage of this tool, it's just that the tool's potential is utterly wasted on my lack of ability to play it well. On the other hand, if my wife complains of abdominal pain and I try to remove her appendix with a $5 X-acto knife, I'm mis-using a tool in a way that almost certainly will harm her.

But what if I create a survey instrument, field it and get a few hundred data points, and start advising clients based on what I've learned from that survey? Am I mis-using that tool in a way that can lead to harm?

When we consider doing research, too many of us fear something like the tweet I included above being aimed at us. Either a well-trained master of the tool spots an error that was invisible to us and calls us out, a social media like-farmer takes a pot shot at us, or we cause actual harm to another despite our intentions to help. These fears are not entirely baseless, but they are based on a misapprehension of the world of research that I hope to correct in this guide.

I hope to illuminate a small corner of the much larger world of research: the small-scale research corner. The value of using small-scale research to help your clients make better decisions is high, and -- if you're careful with your usage of the tools -- the risk of causing harm is low.

Data Is Powerful

Our culture worships data. And rightly so. Data combined with human ingenuity and sweat is a godlike tool that's pulled us out of a nasty, brutish, and short existence into across-the-board increases in comfort, wealth, health, and technological & human potential. But cults form around deities, and so data is more than just a powerful tool.

Data can also be a way to justify taking a quick shortcut from an inner emotional sense to a haughty, external certainty. We can go further and use data as a social cudgel to attack enemies. Or we can assemble enough data to feel that we walk about in priestly vestments, closer to the divine than the unwashed masses.

Data is powerful. But data is not an unalloyed good, nor is it always the best tool to guide decisions. Data can only be as good as those who produce and consume it. But data can be an instrument for improving decision making and wellbeing, and an ability to produce and consume it should be accessible to us, not just large well-funded institutions and companies. For us to do that, we should start with understanding the broader landscape of research.

The 5,000-Foot View

We'll roughly divide the world of research into 3 not-equally-sized sectors:

  1. Academic/Scientific Research
  2. Small-Scale Research
  3. Business Research

@TODO: illustrative sketch

Academic/Scientific research is what we are most familiar with. Anyone who cites numbers about COVID-19 death rates, case counts, transmissibility, and the like is making use of the output of the academic/scientific research world. If there's one thing that outsiders might know about this world's methods, it's the idea of statistical validity. Most of us don't really understand statistical validity, but we know it's important and difficult to achieve, and if we don't like what a given research product seems to say, the easiest way to discredit it is to find some flaw related to statistical validity.

Small-Scale Research is something you'll come to understand via this guide. Small-Scale Research (SSR) uses methods that untrained researchers can use without getting wacky results to enable better decision-making within businesses. SSR keeps the cost reasonable by keeping the scope very narrow and using methods that generate insight and contextual richness rather than definitive declarations about cause-effect.

Business Research is a superset of SSR that seeks to understand cause-effect in the context of a business decision. Business research also seeks to measure the under-measured in order to help manage risk. And finally, there is a branch of business research that uses inexpensive research methods to earn visibility and trust through social signaling but without supporting (or being on the hook for) any specific decision.

The next chapter of this guide will much more fully explain academic/scientific and business research so we can clearly see where SSR fits in between these two much larger worlds.

It's worth thinking about why we should invest in SSR.

SSR Forces A Literature Review

After you roughly define your SSR question, you will do a brief literature review. If this sounds intimidating or technical, it's actually not. A SSR literature review is like Googling around for stuff, except using specialized search engines (more and more useful and free options are entering this market all the time). It would be unwise to start a SSR project without doing a literature review, because you don't want or need to duplicate prior efforts. So we could say that SSR forces a literature review.

This is a very good thing. Through the lit review, we'll get a crash course in the relevant prior art. If our SSR question is roughly aimed at understanding the value of branding, we'll find that there have been serious academic inquiries into this question. [@TODO: link to a few scite and others with pre-populated queries for this] Perhaps this will cause us to refine, narrow, adjust, or abandon our SSR project. This is good!

In fact, if all that a SSR project did was motivate a few hours of literature review, most of us would dramatically reduce our ignorance about the prior contributions of academic/scientific research to our area of expertise. This may or may not change how we work with clients, but it can't possibly hurt, and it's likely that our expertise will be enriched.

SSR Forces Us To Seriously Consider Context

Once you commit to a SSR question, any anxiety that your mind contains will gather itself and start saying, "but what about this? What if this is connected somehow to the question I'm investigating?" This is good, because this forces your thinking outward from the SSR question to the context surrounding it.

If your question is "does better branding increase sales?", that's a good starting point question! Good! Whatever anxiety resides in your mind will quickly marshal its forces to ask: "what other stuff could increase sales? Or decrease sales even if the branding is helping? Or.... or.... or...?" What's happening here is that you are trying to locate your SSR question within the larger context of anything and everything that could be connected to it. This is a VERY GOOD THING!

Eventually your investigation of the surrounding context needs to resolve into a refinement of your initial SSR question so that you can settle down into research, but this preceding "anxious phase" is good because it forces you to seriously consider context, and if there's one thing that can make you a better consultant, it's a better grasp of your client's context.

SSR Can Create Intellectual Property

Intellectual property (IP) is your expertise packaged and made usable without your direct involvement. For us indie experts, IP is generally not something we invest much effort in protecting in a legal sense or worry about being stolen. Most clients would rather pay us to help apply it, and most competitors are too proud or incompetent to bother with stealing or borrowing it. Sure, there are exceptions, but spending money to protect our IP would be like buying meteorite strike insurance for a car.

SSR can create or enrich IP. "I want to create IP" is not the best motivation for investing in SSR. "I want to understand X better so I can help my clients make better decisions" is a much better motivation, but the SSR that fulfills your desire to more deeply understand X can end up being, or contributing to, valuable IP. (If the speculative nature of all of this puts you off, that may be a sign that your business or thinking is not in a place that's compatible with SSR.)

SSR Can Contribute To Your Point Of View

Your point of view (POV) is like an "intellectual fingerprint" -- a way you have of seeing things that is distinctive in the market. Every person has a fingerprint, and every person has a point of view. But clients do not find every POV interesting or relevant. Points of view with content that clients find useful, challenging, intriguing, or suggestive of a better path forward are the ones that are most interesting or relevant to them.

Your way of seeing the world is informed by where you stand. Do you stand firmly rooted in your own belief or experience, or do you stand more rooted in what data tells you? The content of your point of view will be influenced by the context of where you stand.

I periodically run a workshop that helps consultants clarify and sharpen their point of view. With very few exceptions, participants stand rooted in their own experience, but they want their POV to come more from data. Earlier I said our culture worships data. I wasn't exaggerating, and this explains why most of us want our POV to have the power that data can confer.

SSR can enrich your POV with unique data that you have assembled and interpreted, which can combine powerfully with the output of other, complementary, research.

Ultimately SSR Helps Our Clients Make Better Decisions

This is really the bottom line here. It's the ultimate reason to invest in SSR. Well-designed SSR can help our clients make better decisions, which -- bit by bit -- enhances the health of the market we serve, which creates more and better opportunity for us. If you're seeing something like a spirit of service being the most powerful motivator for SSR, then you're seeing this thing clearly.

SSR Calls For Strength And Humility

Data is powerful. It is also a story we tell ourselves about why we decided a certain way.

This is an argument both for getting more fluent at creating and using data, and an argument for humility around the whole idea of data's value. Some suggested reading for you:

"Alchemy" by Rory Sutherland is a fun, worthwhile read here. "How to Measure Anything" by Douglas Hubbard is a much less fun, but equally worthwhile counterbalancing read.

If you're up for it, read these two books back to back. You'll find yourself suspended in a sort of "intellectual hammock", pulled in two opposing directions with respect to the value of data. This is the right place from which to think about this stuff.

Why Don't We Do More Small-Scale Research?

Let me be clear: well-executed small-scale research is very rare. There are good reasons why.

We mis-aprehend research generally, and business research specifically. We hear the word "research" and tend to assume that means expensive, complex, technical, inaccessible stuff. Most of us don't know about this little niche of accessible, useful methods that untrained but motivated people like us can use to create unique value, and so we hear the word "research" and mentally expand that to "not for me".

We lack formal training in SSR methods. In college, I did one small-scale research project involving surveys and SPSS as part of a senior Political Science thesis project. That was the extent of my schooling's contribution to my understanding of SSR. Maaaybe if we have done UX or product validation work we've had some exposure to SSR-friendly research methods, but the majority of us lack even semi-formal training in SSR methods. I hope this guide helps, but this lack of training partially explains why SSR is rare.

We are intimidated by the idea of research. The like-farming tweeter I referenced at the start of this chapter lives in our head, or at least the social threat they represent lives in our mind as a fear of "getting in over our heads". And so many of us are intimidated by the idea of SSR, because we fixate on the research part (and our associated fears) and undervalue how the small-scale part can make SSR usable and valuable for us.

We see few examples of our peers doing SSR. This reinforces the notion that it's difficult, complex, and risky. We have some examples of SSR used for marketing, but that's just one of several ways research can be leveraged, and examples where it's used for decision support are less visible to us and therefore more mysterious.

We practice an unlicensed profession, and so there's little incentive for us to raise our game beyond what improvisation, gut feel, past experience, "best practices", and a dash of confidence can achieve. Said more cynically, our clients are surprisingly tolerant of really mediocre consulting services, which reduces the incentive for us to level up the quality of our advice using data. Said more positively, often the status quo at a client is so bad that improvisation, gut feel, past experience, "best practices", and a dash of confidence can create a miraculous amount of relative improvement!

All together, these factors cause us to under-utilize research. Again, many of these are good reasons to not invest in SSR. Doing competent but -- let's be honest -- utterly ordinary work can be monetized in totally adequate ways. Building a small team, leveraging a bit of luck, and avoiding making any terrible decisions for 10 or 20 years can buy you two really nice houses, a few college educations for kids, a funded retirement account, and quite a few nice vacations and meals in restaurants. All without touching SSR with a ten foot pole. Not bad!

So after accounting for all the reasons to invest in SSR and considering all the reasons we don't, I think your decision will come down to dissatisfaction. The folks who are willing to invest in SSR tend to be dissatisfied with the status quo. They have a hunger to advance the state of the art. A hunger that, frankly, I have been unable to fully explain. I have this hunger. Some folks I know who "should" be earning more money have this hunger and invest in SSR anyway, despite the "illogical" nature of the investment. And I know others who are earning way more money than they need to live well, can't explain how exactly the SSR will contribute more revenue, and also have this hunger.

The best I've got for an explanation: it's a hunger to understand more deeply. We just have to gain this deeper understanding. Maybe this is driven by even deeper, more primal motivations for status, power, etc. I don't really know.

I do know that you can stop reading this guide if you're sure you don't have this hunger. There are easier, less risky ways to optimize your business to make more money and serve your clients better.

But if you do have this hunger, or if doing SSR is part of your job, or if you're merely curious, then read on. I won't waste your time with anything other than the essential concepts, details, and examples you need to understand and execute small-scale research.

2: What Is Research Generally, And Business Research Specifically?

Small-scale research is accelerated, focused learning.

Research in general is an attempt to understand causation.

Said a bit more eloquently by Kanjun Qui:

I misunderstood the nature of research for most of my life, and this prevented me from doing any. I thought significant research came from following the scientific method until novel discoveries popped out. I'd never contributed something new to human knowledge before, so being a researcher—which required replicating this outcome—felt impossibly far out of reach.

But it turns out the novel discovery is just a side effect. You don't make novel discoveries by trying to make novel discoveries.

Instead, research is simply a continuation of something we already naturally do: learning. Learning happens when you understand something that someone else already understands. Research happens when you understand something that nobody else understands yet.

Source: https://kanjun.me/writing/research-as-understanding

The whole, short piece is well worth a read, but I can't help quoting one more bit:

Research, I realized, is what happens as a byproduct when you try to understand something and hit the bounds of what humanity currently knows.2 At that point, there's suddenly no one who can tell you the answer.

If you care enough about the question, you have to figure out how to answer it yourself, and that's when you start running experiments and developing hypotheses. That rote process of science we're taught in school—to start with a question, generate hypotheses, test with experiments, draw conclusions—it's a good tool, but it doesn't capture the most important element: actually wanting to know the answer to the question!

This... this is absolutely the core motivation of small-scale research (SSR): wanting to know the answer to an un-answered question that you care deeply about. This is, more generally, the motivation of most other styles of research, but the caring about part is particularly important and personal when it comes to SSR.

The way most of us think about research causes us to worry about caring or caring too much. We worry using terms like "bias" and "motivated reasoning" and a few others. Basically, we worry that caring too much will interfere with our general objectivity and the effectiveness of our research. But when it comes to SSR, I've found that the worst output is produced by people who care too little. They're often creating the research as a social signaling tool ("I do research, therefore I have authoritative insight into X."), but they actually have little or no practical use for the insight the research generates. As a result, they can overlook nonsensical or perhaps more subtly flawed results because they have no skin in the game when it comes to how the results will be applied in real world situations, and they lack the context that practitioners have, causing them to not even sense when the results are nonsensical or flawed.

Almost every apparent constraint that comes with the territory of SSR is actually an opportunity in disguise. Small sample sizes are an opportunity to go deeper and gather more nuance. The lack of rigorous statistical controls is an invitation to get dead serious about contextualizing your findings, which helps you avoid self-deception or cluelessly reporting bad data to the world.

I wanted to start explaining the larger world of research with this focus on the most squishy, human part of SSR: caring. Caring about the question, and caring about those who can benefit from an answer to the question (or, more likely, reduced uncertainty around the question). This is both the beating heart that powers and guides SSR, and it's also the thing that others may use to beat up on your method and results. There's no way around how both weakness and strength are bound together in this foundational aspect of SSR.

What Is Research Generally?

In the business context, there are 4 styles of research:

  1. Risk Management
  2. Innovation
  3. Social Signaling
  4. Small-Scale Decision Support

1: Risk Management Research

I've had to invent terminology here because I haven't found useful terminology for business research, at least not the way I need to organize things for you in this guide. There are functional categories like market research, to name one, but those overindex on outcomes/functions and are granular in the wrong way. Anyway!

Risk management research uses research to help manage risk. Yes, that's a circular definition. :) Let's go a bit deeper.

The primary goal of risk management research is to reduce uncertainty or establish probabilities. You could think of this as simply measuring something that's under-measured. The chief apostle of this style of research is Douglas Hubbard, and the most accessible starting point to his point of view is any one of his talks available on YouTube or his book How to Measure Anything.

One of Doug's primary points is that complete certainty is not possible and rarely necessary in a business context. As a result, the kind of extreme rigor applied to many academic or scientific research efforts isn't necessary in a business context. In the business context, there is a ton of value in merely reducing uncertainty, and many situations will allow for significant reductions in uncertainty with just a few measurements. And often those few measurements are easy and cheap to implement, once you have a sense of what you actually need to measure. Here is a good, short, useful article from Doug on this: https://hubbardresearch.com/two-ways-you-can-use-small-sample-sizes-to-measure-anything/

A few examples will help illustrate the kind of situations where risk management research is a good fit:

  • Reducing uncertainty RE: the cost of a potential new government procurement system
  • Quantifying the risk of flooding in a mining operation
  • Quantifying the potential impact of particular pesticides regulation.
  • Understanding the most significant sources of risk in IT security

All of these examples are from a talk Doug gave, and you can find similar examples in his talks, available on YouTube.

Risk management research is best suited to environments that function like closed systems, where you are able to control and measure almost every aspect of the system. Big business enterprises are not closed systems, but they try to function like they are. There's a famous saying (attributed to Peter Drucker, I think?): "what can't be measured can't be managed". This expresses the underlying anxiety about control that's often omnipresent in so many modern organizations, and this desire for control pushes the system's organization and function closer to that of a closed system.

Here's a quick summary of the method that Douglas Hubbard recommends for risk management research:

  1. Define the decision you're focused on
  2. Model the current uncertainty
  3. Compute the value of information that could help reduce that uncertainty
  4. Measure, keeping in mind that your goal is reduced uncertainty, not complete certainty
  5. Optimize the decision, potentially rinse & repeat the previous steps to further reduce uncertainty

Risk management research roughly fits into the deductive category, meaning that you are testing a hypothesis by gathering data, but much more often risk management research is simply measuring something that hasn't been adequately measured. There's often no hypothesis at all; there's simply a question related to a decision and a process of identifying what should be measured, what value that measurement might create, and the work of doing and interpreting the measurement.

The purpose of risk management research is to reduce uncertainty or establish probabilities; quantitative methods are usually the best fit for this purpose. Another one of Doug's oft-repeated points is that expert intuition is frequently wrong, and simple, inexpensive measurements can outperform expert intuition when it comes to accuracy.[1] His proposed remedy is to move away from the qualitative world of expert intuition and towards the quantitative world of numbers and estimated probabilities.

Some recommended reading if you're interested in learning more about the risk management style of research:

2: Innovation Research

Innovation research is done to generate new options or cultivate a more nuanced, detailed, "rich" understanding of a system, process, group of people, or person. People invest in innovation research because they believe[2] that empathy precedes innovation, and the purpose and methods of innovation research are oriented around increasing empathy, or at least equipping us to better understand others if we have an empathetic intent.

You don't have to look very far into the world of innovation research before you find examples of how empathy leads to economically-valuable innovation. Again, "innovation research" is my invented term for several specific modalities, so if you do want to search more on your own, start by looking for "JTBD success stories". More on these specific modalities in a bit.

The canonical example seems to be how the Mars, Incorporated company used Jobs To Be Done research to learn that quite often the "job" that many Snickers customers "hired" the candy bar to do was to serve as a meal when time was tight. Customers seeing the candy bar as less of a candy and more of a portable, calorically-dense meal substitute opened up new options for how Mars could market the product. Some might even say this research helped create a new category.

Innovation research is especially suited to environments that function like open systems -- ones where you are unable to control, measure, or even fully understand the relationships between elements of the system, or the system you’re investigating and other related systems. Innovation research uses an inductive approach, where you proceed from observation -> pattern recognition -> development of a theory, model, or conclusion. This approach is also qualitative in nature; it's not seeking numerical precision, but instead it's seeking a rich, nuanced, detailed understanding of a system, process, group of people, or person along with the same kind of nuanced, detailed, deep grasp of the context surrounding that system, process, group, or person.

This blend of inductive, qualitative research into open systems manifests as one of several branded, semi-defined styles of research:

  • Jobs To Be Done research
  • Customer Development research
  • Ethnographic research
  • Problem Space research

(There are probably others that I am not aware of that deserve to be on this list.)

Some recommended reading:

3: Social Signaling Research

Small-scale research is seeking this: