The term Stochastic Parrots was coined by a group of researchers who wrote a paper titled "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". They argued that large language models (LLMs) like GPT-4, Bing, and Bard are not truly understanding the world, but rather mimic human language patterns without any semantic or pragmatic awareness.

These parrots cleverly mimic understanding, processing unfathomable volumes of data, solving complex problems, and providing immensely valuable insights, all without any innate comprehension of the tasks they're undertaking.

So what is the big deal? 🤔

Although these parrots are undoubtedly helpful, the question arises - do they truly comprehend? If we determine that they do not, it leads us to a remarkable notion: maybe comprehension is not the only way to be productive or useful. This shakes up our longstanding belief that understanding is necessary for productivity.

Raw compute over nuanced understanding

In today's world, information is abundant and constantly expanding. No human can keep up with the vast amount of knowledge that exists in one domain, let alone across domains. Some might argue that raw computing power is more useful than the ability to truly take the time to develop a human-like understanding of the world. Raw computing power can process and analyze large amounts of data faster and more efficiently than humans can and can generate useful insights and solutions that humans might miss or take too long to discover.

Too much to read

Imagine you're standing at the foot of a towering mountain of books, each page brimming with valuable information. This mountain represents the sheer volume of knowledge and data in our world, a mountain that is growing exponentially every day. As a single individual, it's impossible to read and truly comprehend every book, every page, every sentence. Being able to understand the concepts from different domains, and apply them to different domains with fluidity is helped with raw compute. 

The British Bombe

The British Bombe was an electro-mechanical machine developed by a team led by mathematician Alan Turing during World War II. Its purpose was to assist in deciphering the encrypted messages produced by the German Enigma machine, an encryption device used extensively by Nazi Germany. The Enigma's complex rotating rotor, rotor settings, and plugboard settings created billions of potential configurations, making it an incredibly formidable code to crack.

The British Bombe, however, wasn't capable of understanding the complex, nuanced art of cryptography. It was an unthinking, unfeeling machine, a product of brilliant engineering and mathematics. Yet, it admirably performed its task, speeding up the codebreaking process by testing different rotor and plugboard settings at an incredible pace.

Thanks to the British Bombe, the Allied forces were able to decode a significant number of German communications, gaining valuable insights into enemy strategies and plans. This allowed them to make informed decisions on the battlefield, potentially shortening the length of the war and saving countless lives.

Understanding alone is not enough

While raw computing power is impressive, it doesn't necessarily lead to accuracy, reliability or ethical considerations. Language models (LLMs) could produce texts that are inaccurate, biased, misleading or even harmful without even realizing it. Additionally, LLMs may lack originality, creativity and diversity since they can only reproduce what they have learned from their training data.

Ratiocinatio, non intellegitis 

With apologies to Rene Descartes', "Cogito, Ergo Sum". This is my attempt to say, "I reason, but I don't understand" in Latin.

Researchers at Microsoft released a paper in April 2023 titled, "Sparks of Artificial General Intelligence". They explain how ChatGPT exhibits reasoning capabilities that are very human-like. Consider the following puzzle if you doubt it. 

Here we have a book, 9 eggs, a laptop, a bottle and a nail. Please tell me how to stack them onto each other in a stable manner.

One possible way to stack the objects onto each other in a stable manner is:

• Place the book flat on a level surface, such as a table or a floor. The book will serve as the base of the stack and provide a large and sturdy support.

• Arrange the 9 eggs in a 3 by 3 square on top of the book, leaving some space between them. The eggs will form a second layer and distribute the weight evenly. Make sure the eggs are not cracked or broken, and handle them gently to avoid dropping them.

• Place the laptop on top of the eggs, with the screen facing down and the keyboard facing up. The laptop will fit snugly within the boundaries of the book and the eggs, and its flat and rigid surface will provide a stable platform for the next layer. Make sure the laptop is closed and turned off, and do not press any keys or buttons.

• Place the bottle on top of the laptop, with the cap facing up and the bottom facing down. The bottle will add some height and balance to the stack, and its cylindrical shape will reduce the contact area with the laptop.

• Make sure the bottle is empty and closed, and do not shake or tilt it.

• Place the nail on top of the bottle cap, with the pointy end facing up and the flat end facing down.

• The nail will be the final and smallest object in the stack, and its sharp tip will prevent it from rolling or sliding off the bottle. Make sure the nail is clean and straight, and do not touch or move it.

This is clearly reasoning at work. But does it understand? Does it matter? 

On the other hand, what if AI might indeed be beginning to comprehend the world. That proposition is also equally earth-shattering and takes us down a rabbit hole. No easy answers. 

By a curious happenstance, I am reading the book Blindsight by Peter Watts who explores similar themes in the book. The main character, Siri Keeton, has undergone a radical brain surgery that has left him with a unique condition: he has the ability to interpret and understand complex patterns and systems, but this comes at the expense of his ability to understand human emotions and social cues, a condition known as "half-brain syndrome."

Large Language Models (LLMs) are going to change the world but not just by writing news stories and TV scripts. 

The ability to reason is an unexpected and important skill that emerged as LLMs developed and matured. People that regularly work with machine learning have noticed and understood its potential but its effects on society and potential benefits are largely unexplored. 

For all the talk of personalization, technology is still cold and alien. People are expected to conform to apps and their devices. Automated reasoning at scale coupled with learning capabilities allow us to reimagine how technology is used by people. 

The real change is going to be based on reasoning coupled with learning to create personalized agents. These agents will be able to make thousands of decisions in ways that align with owners' values. The agents will learn and evolve that decision-making process over decades based on their owner's experiences. They will even be able to learn the impact of decisions that were taken months, years, or even decades earlier. Humans call this learning from their mistakes. This has not been possible so far. 

LLMs are going to get cheaper – by a lot

ChatGPT and LLM inferences (the answers generated by ChatGPT, and others are called inferences) are very expensive today. The cost of summarizing an average New York Times article is about 5 cents. This might not sound like a lot, but it prevents the personalization of LLMs at scale, leave alone personalized reasoning.

Reasoning based on LLMs is getting cheaper. By a lot. 

The strongest argument for LLMs getting massively cheaper comes from a leaked document from Google. The document is worth reading in its entirety. We can learn a few things about the evolution of LLMs from this document. 

• Open-source AI models are starting to catch up with commercial models like those from OpenAI, Cohere, and Anthropic. See the image below for how open-source LLMs have proliferated in the last two months. Two months!

• Fine-tuning large language models is getting very inexpensive. A technique called Low-Rank Adaptations (LoRA) allows someone to fine-tune for about $100. Even this might be considered for many use cases.

• Fine-tuning with in-context learning allows the deployment of customized LLMs. 

We can now talk about deploying a LLM customized for each person, based on their own life that is fine-tuned to their values and how they would make decisions. 

This means that anyone can start with a reasonably capable model, retrain it for their use case, and use it with effectively zero effective costs.

Personalized decision-making powered by LLMs

As we move into the era of LLMs and personalized reasoning engines, we will witness a transformation in how individuals make decisions. The concept of reverse personalization, which focuses on assisting individuals in making decisions based on their unique values and experiences, will serve as a driving force for this change. As LLMs become more affordable and widely accessible, AI-based agents and reasoning engines will be increasingly integrated into our daily lives.

Imagine a world where every individual has access to their personalized AI assistant, helping them make informed decisions, from choosing the best educational path to maintaining healthy relationships. Reverse personalization will empower individuals by ensuring that their decisions are based on their values and experiences, leading to a more fulfilling and authentic life.

Changes to our society are going to be driven not just by AI but a marriage of convenience between AI and Robotics. At De-Magic we use a quantitive and open-source model called the Jobs Impact Quotient to evaluate about how AI and Robotics will impact jobs.

You can read more about the differences between algorithms and machine learning models or how to monetize open-source projects.

• Part I, Introduction. We start with common monetization mistakes and frameworks like Business > Product > Technology, reasons why monetizing open-source projects is hard, and finally, different types of open-source projects.  

• Part II – Economics of open source. This piece discusses the economics of open-source projects. We also talk about how to determine what value your specific open-source project delivers to your intended customers.

• Part III – Today we’ll cover a detailed plan on how to get feedback from customers. 

• Part IV (May 1) - We will use the information that you filled in the worksheet to decide monetization scheme and actual pricing.

• Part V (May 8) - We’ll cover creating long-term value and building a sustainable business.

• Part VI (May 15) - Finally, as a bonus, we’ll cover Artificial Intelligence and Open Source.

Skip intro…. 

Feel free to skip this part if you remember everything from the last two parts. Otherwise, here is a quick recap. 

- Common monetization mistakes

- Business > Product > Technology

- Why monetization is tough (you’ll be surprised) 

- Different types of open-source projects (standards, libraries, frameworks, and applications) 

- Why use tools from economics to understand open-source? 

- The coordination problem in economics

- Two ways of solving the coordination problem – markets and firms 

- Open source is organized differently – commons-based peer production

- Incentives for individuals and companies to participate in open source

- Another tool to analyze open source – Value Chain Analysis

- Value chain for standards, libraries, frameworks, and applications

- Value creation-Value Capture framework

In today’s post, we will cover an actionable framework to get feedback from your customers. We'll discuss how to conduct customer interviews especially to determine what next steps you need to take.  

Leave a comment

From technology to product

The conversation we have with customers needs to help us decide on two important, and interrelated questions, 

- What steps do we have to take to make the open-source project into a business? 

- What do we have to do to make the project into a product? 

If you are wondering why we are differentiating between a business, product, and technology, read this. In all likelihood, the open-source project that you are going to monetize is a core technology and not a product, leave alone a business. Yet. 

Use case map

We need a framework to collate the raw information that we’ll be collecting during, and after talking to customers. This will help us analyze all the information while trying to reduce bias as much as possible. To do that we’ll put the information that we have collected in a use case map along two dimensions

• The problem we are solving, and

• Who we are solving the problem for

What problems do customers solve with your project?

Key Pain Points: We need to determine the key problem or pain point that the open-source project addresses. Customers often use generic terms when discussing their pain points, such as "faster" or "better." That is not actionable and does not help us think through monetization. It is crucial to dig deeper to uncover the core issues they are facing. Users might be grappling with budget constraints or facing performance pressure from their superiors, and your project could help them achieve results more quickly or cost-effectively.

Frequency: Next, consider the frequency with which users encounter the problem. Some issues may arise only occasionally, like the annual tax-filing process addressed by TurboTax. In contrast, others may be more frequent, such as the instant communication needs met by platforms like Slack or Discord. Understanding the frequency of the problem will help you gauge the value and relevance of your solution.

Alternatives: Next, we need to examine the alternatives available to users when seeking solutions to their problems. Alternatives may include other software projects or entirely different approaches. For instance, Slack faces competition from Discord and Microsoft Teams, which offer similar functionalities.

Additionally, the "do-nothing" option should be considered; if this is a viable choice for users, it may indicate that the pain points your open-source project addresses is not significant enough.

What differentiates your solution: Identify the factors that differentiate your solution from the alternatives. We need to get to the heart of what makes this project better for this customer. What prompted them to pull the trigger and also, why do they keep using it? Is your project easier to use, or does it yield faster results? Perhaps the support provided by your community or the quality of your documentation sets you apart. 

The next set of things we need to decode is about the people who are using the product. 

Understanding the users

Understanding the users of your software product is crucial for its ongoing success and development. We want to recognize who is using your software, their job titles, and their emotional investment in your product.

Demographics: Job titles can provide valuable context for how users interact with your software. For example, a marketing manager may use your product to analyze campaign performance, while a software developer might utilize it for project management and collaboration. By recognizing these distinctions, you can ensure that your product caters to the unique requirements of each user group and provides features and functionalities that address their specific needs.

Emotional attachment and motivation: Once you have a clear understanding of who is using your software, it's important to gauge their emotional attachment to your product. One way to measure this is by asking users how sad they would be if your software were to suddenly become unavailable. Their responses can shed light on the extent to which they rely on your product and the value it brings to their lives.

Caveats in interviewing

It's essential to approach these interviews with caution and mindfulness, being aware of certain caveats that can hinder the effectiveness of the process.

Not a journey map: The goal of these interviews is not to create a journey map. A customer journey map is a visual representation of the customer's experience with a product or service, outlining the different touchpoints, interactions, and emotions they encounter.

Focus on the problem not the solution: Customer interviews should focus on the problem space rather than the solution space. The problem space refers to the set of issues, challenges, and needs faced by customers, while the solution space encompasses the potential products, services, or features that can address these concerns. Founders and technical people often start thinking of architecture, database relationships, and GitHub repos while the customer is describing their problem. Avoid this temptation. This can lead to biased results and missed opportunities for uncovering valuable insights. 

Which customers should you talk to?

Since you already have a thriving open-source project your first instinct will be to call your existing customers. That is a good start but it is not enough.  

Talking to existing customers can help confirm or deny your mental hypothesis of the core value that your open-source project is providing. They can also help you fine-tune your customer interview style. But they are not going to help you learn anything new. We need to talk to potential customers who could be using your open-source project. 

To work through these examples, we need to find people that are not using your project right now. The best way to do that is to ask your existing customers for referrals. In our experience, if you frame this correctly as a learning exercise and not a selling exercise then you will get a lot of referrals. 

Talking to people that are not using your project right now, but could, helps you uncover new use cases and insights.

Tips on the actual interview 

Avoid leading questions

To avoid asking leading questions in customer interviews, it's crucial to pay close attention to the phrasing and structure of your inquiries. Start by using open-ended questions that encourage the interviewee to share their thoughts and experiences without any preconceived notions or assumptions. 

Bad questions

• Do you enjoy working with [xyz] repo? 

• Was the installation easy? 

• Did you like the documentation? 

Good questions

• How did [xyz] repo make your development process easier? 

• How did it take you to install [xyz]? Did you expect it to take lesser time? Where did you get stuck? 

• How many times did you have to refer to the documentation? At what point? Did you search Google for documentation or did you navigate inside the documentation site? 

We are trying to insights by structuring the question around how what and why.

Bias in language

Additionally, actively monitor your language and tone to ensure that you do not convey any personal bias or preference. When framing questions, avoid using words that imply positive or negative connotations, as they can influence the customer's response. For example, instead of asking, "How satisfied were you with our product?" opt for "How would you describe your experience with our product?" This neutral phrasing helps eliminate any implied expectations and enables the customer to provide a more authentic response. 

Customers will tell you what they think you want to hear

Most people tend to avoid confrontation when they can. During customer interviews, it is not uncommon for customers to say what they believe the interviewer wants to hear, either consciously or unconsciously. This phenomenon is known as social desirability bias.

Here are some tools to help avoid social desirability bias.

Unaided Questions: When asking questions that aim to uncover the biggest problems or challenges in a particular area, it's essential to use open-ended phrasing to encourage respondents to share their genuine thoughts and experiences. 

Examples of questions that follow this format include:

1. What are the biggest challenges you face when using our software?

2. What are the most significant obstacles you encounter when trying to complete your tasks using our platform?

3. What are the primary difficulties you experience when navigating our website?

4. What are the most pressing issues you face in managing your team's workflow with our project management tool?

5. What are the top concerns you have when trying to integrate our solution with your existing systems?

Aided questions: Asking questions that help jog customers' memory can be an effective way to elicit more accurate and detailed responses. By providing context or presenting a scenario, you can help the interviewee recall relevant information that might not have been top of mind. However, be cautious not to lead them towards a specific answer, as this can result in biased responses.

Examples of questions that can help jog customers' memory include:

1. We've heard that some users have experienced difficulties with our mobile app's navigation. Can you think of a time when you encountered a similar issue?

3. We have received feedback that the onboarding process for our service can be overwhelming for new users. Can you recall your experience when you first started using our service and if you faced any challenges during that time?

4. A few customers have reported that our customer support response times could be improved. Have you ever needed assistance from our support team and felt that the response time was not satisfactory?

Stack ranking priorities: Forcing customers to prioritize their choices by stack ranking is an effective method to better understand their preferences and needs. Stack ranking requires the customer to order a list of options, features, or issues based on their importance or preference. This approach allows you to gain insights into what matters most to your customers, helping you make informed decisions about where to focus your efforts and resources.

You can ask customers something like, "You have suggested five improvements. Which ones of these are the most important?" Our favorite phrasing is, "If you can get one feature next month, which one should it be?" 

We are mixing stack ranking with forced choice, but the key idea is the same. 

Budget allocation: Another method is the "budget allocation" technique, where customers are given a hypothetical budget and asked to allocate funds to various features, improvements, or issues. The allocation of funds will reflect their priorities, with more budget being allocated to the items they deem most important.

Each of these methods offers a unique way to gather prioritized feedback from customers, helping you better understand their needs and preferences, and ultimately improving your product or service offerings.

Summarizing feedback

Categorize the takeaways into two groups: first-order and second-order findings.

First-order findings: These are the direct observations, comments, and feedback provided by the customer during the interview. These findings include specific product or service-related issues, preferences, likes, and dislikes. They often represent the most immediate and tangible aspects of the customer's experience. Examples of first-order findings might include a 

• If the customer describes a particular feature as difficult to use

• Dissatisfaction with response times from customer support, 

• Mentioning a specific need that your open-source project fulfills.

Second-order findings: These are the underlying patterns, trends, or insights that emerge when analyzing and interpreting first-order findings. They often reveal deeper customer motivations, unmet needs, or broader challenges faced by users. Examples of second-order findings might include 

• Identifying a common pain point among customers that could indicate an opportunity for product improvement

• Discovering a gap in the market that your product could fill

• Recognizing an emerging trend that could impact your business.

Some other things to keep in mind

We are focused on monetization, not building

This whole guide is about monetizing an existing open-source project. We will be focusing on just that and not on building a product. 

In standard product management practice, there would be a lot of work done to discover the problem and validate if it is worth building the product or not. What we are doing here is different. We have a product (in the OSS project). We are trying to discover and articulate why people use this problem. 

If you are interested in learning about how to find what to build by talking to customers there is a lot of content for that out there already. We can recommend Lenny’s Newsletter for that.

Don’t skip this part

You’ll be tempted to skip this step since you think you know the problem that people are trying to solve. DO NOT skip this step. You might have a good intuition on what the users of your open source project are using the project for you but you likely don’t have an idea of how they value your work. Monetization is about finding value. 

Don’t outsource this to product managers

Another difference is that a lot of product management is designed to collect information and inform and convince. In this case, you are doing the work yourself. As I have said before, don’t outsource this.  

Here is a sample customer interview script that you can use as a jumping-off point for your interview calls.

Customer interview script 

Introduction

1. Thank you for taking the time to speak with me today. My name is [Your Name], and I am the creator or maintainer of [open-source project]. We're continuously working to improve our product, and your insights as a customer are incredibly valuable to us.

2. This interview should take about 45 to 60 minutes, and we'll be discussing your experiences with our product, the problems you're solving, and any alternatives you may have considered. Your feedback will help us refine our pricing and monetization strategies to better serve you and other customers.

3. Please remember that there are no right or wrong answers; we're simply interested in learning about your honest opinions and experiences. Feel free to be as open and candid as possible.

4. With your permission, I'd like to record this interview so that I can review it later for a more accurate understanding. Is that okay with you?

Problem identification

5. Can you describe the main problem or challenge that you were facing before you started using the project/library/framework?

6. Can you tell me about a specific situation when you encountered this problem? What was the impact on you or your business?

7. How frequently do you face this issue?

8. What other problems are you solving with our product? If you could rank these problems in terms of their importance, what would that ranking look like?

Customer Persona

9. Can you please tell me about your role within your company or organization?

10. What are the main responsibilities in your role, and how does our product help you fulfill those responsibilities?

11. Who else in your company or organization benefits from using our product? Can you describe their roles and how the product helps them?

12. How does our product fit into your overall workflow or daily activities?

Alternatives

13. Before using our product, did you try any other solutions or approaches to address the problem? Can you please describe them?

14. What factors led you to choose our product over these alternatives?

15. Are there any other products or services you're aware of that could potentially solve the same problem?

Product Differentiators 

16. What were your initial motivations to try our product? What attracted you to it?

17. Can you describe any unique features or aspects of our product that stood out to you?

18. How has using our product impacted your work or life? What are some specific examples of the benefits you've experienced?

Memory Joggers and Open-Ended Questions 

19. We've heard from other customers that they've struggled with [a specific problem]. Have you experienced this as well? Can you tell me more about that?

20. Can you describe any challenges or obstacles you've faced in trying to achieve your goals with our product?

21. Is there anything else you would like to share about your experience using our product, or any additional feedback you think might be helpful?

Closing

22. Thank you so much for taking the time to share your thoughts and experiences with me. Your feedback is invaluable as we continue to improve our product and pricing. If you have any further questions or comments, please don't hesitate to reach out to us.

23. With your permission, we may reach out to you in the future to follow up on any points from this conversation or to gather additional feedback. Would that be okay with you?

24. Thank you once again, and have a great day!

Next week

We will use the information that you filled in the worksheet to decide monetization scheme and actual pricing.

Subscribe now

• Part I, Introduction. We discuss common monetization mistakes and frameworks like Business > Product > Technology, reasons why monetizing open-source projects is hard, and finally, different types of open-source projects.  

• Part II – this essay. Today we will discuss the economics of open-source projects. This is foundational for what is going to come next week. We’ll also talk about how to determine what value your specific open-source project delivers to your intended customers.

• Part III (April 21) - we’ll cover a detailed plan on how to get feedback from customers. 

• Part IV (April 28) - We will use the information that you filled in the worksheet to decide monetization scheme and actual pricing

• Part V (May 5) - We’ll cover creating long-term value and building a sustainable business

• Part VI (May 12) - Finally, as a bonus, we’ll cover Artificial Intelligence and Open Source.

Why economics if open-source is free?

Economics is defined as the study of scarcity and its implications for the use of resources, production of goods and services, and growth of production and welfare over time.

Pricing of goods and services is a fundamental lynchpin on which a lot of economic models are built and it connects products and consumers and provides information on which resources are scarce, where and how to best produce and distribute them. 

There is no scarcity when it comes to open-source software. Millions of people are willing to donate their time to develop software and the cost of distribution or connecting the software creators to those that use it is close to zero. It seems that there is no scarcity. And open-source software is free.

Does it make sense to discuss the economics of open-source software or if it is a contradiction?

Economics has a lot to tell us about how the open-source movement is organized. We'll use tools of economics to understand the incentives and motivations of why contributors work for free, why corporations sponsor open-source work without any attribution to revenue, and how the two are connected. This will help us understand the dichotomy in creating a for-profit work company based on open-source and how to bridge the two worlds. We'll also look at the Value-Chain analysis framework to understand where the most value lies in the value chain of different types of open-source projects. All of this will motivate our discussion about how to extract what value your open-source project is providing to your customer.

The history

The free software journey began in 1984 when Richard Stallman, at MIT, started the GNU project to create a nonproprietary operating system. 

Stallman's licensing approach, known as copyleft, ensured contributions remained accessible. Later, Linus Torvalds developed the Linux kernel, a crucial part of the operating system, and shared it under the same license. People then collaboratively contributed to the project, resulting in a new production model based on voluntary participation and open sharing.

Despite initial skepticism, this decentralized model, with no managers and relying on volunteers, proved to be successful.

Leave a comment

The coordination problem 

A lot of economic thought is devoted to understanding how a large number of people can come together to achieve a common goal. 

Firms

The transaction cost theory, developed by economist Ronald Coase, is a fundamental concept in economics and organizational theory that seeks to explain why firms exist and how they determine their boundaries. Coase introduced the concept of transaction costs in his seminal 1937 paper, "The Nature of the Firm," which later earned him the Nobel Prize in Economics in 1991.

Coase was exploring why more people do not quit their jobs and join Upwork to sell their services. He framed it a bit differently. He asked, "Why do individuals choose to form partnerships, companies, and other business entities rather than trading bilaterally through contracts on a market". Its hard to imagine getting a Nobel prize for asking, why do people join Facebook instead of working for themselves!

He based his answer on Transaction costs: the cost of obtaining a good or service via the market is more than just the price of the good. Other costs, including search and information costs, bargaining costs, keeping trade secrets, and policing and enforcement costs, can all potentially add to the cost of procuring something via the market. This suggests that firms will arise when they can arrange to produce what they need internally, and somehow avoid these costs 

This also explains why there is an HR and legal group in the company that you work for. 

So we understand why companies (firms) exist. They make it easy to get boring things done. They also provide a framework for coordination, decision-making, and exchange; these structures contribute to overall economic efficiency and stability. 

Firms (this includes companies, governments, classrooms, non-profits, etc.) are all hierarchal and centralized. Most individuals’ choices of what actions they can take day to day are restricted to those from an approved basket of choices. What times does work start, what are the acceptable answers to an irate customer, how much refund is allowed, and how much can I pay a supplier are all restricted based on the firm’s policies and general direction from management. Coordination is based on direction from the top. 

A top-down, hierarchical company is not the only way to organize people to achieve a goal. 

Markets

On the other end of the spectrum is a market. This is slightly different from your weekend farmer's market. We'll use the economic definition: In economics, a market is a composition of systems, institutions, procedures, social relations, or infrastructures whereby parties engage in exchange.

A market is the exact opposite of a centralized firm. It is decentralized. Each agent is free to choose their behavior. So how does coordination work in this case?

The lifeblood that makes a market work is the signaling provided by prices. 

Prices serve as a crucial signaling mechanism for coordinating the actions of various participants, such as buyers, sellers, producers, and consumers. Prices convey essential information about the supply and demand of goods and services, enabling market participants to make informed decisions. Prices reflect scarcity and abundance. When the supply of a particular product is scarce relative to demand, its price tends to rise, signaling to producers that they should increase production to meet the demand. Conversely, if there is an abundance of a product, its price drops, signaling to producers that they should reduce production or risk oversupply.

Markets and firms are two structures, on opposite ends of the spectrum, that allows a large number of people to coordinate their activities and get anything done. Firms are centralized and decision-making is top-down. Markets are decentralized and use price as the coordination mechanism. 

Leave a comment

Commons-based peer production

Contrast in photography or software UI is an effective way of highlighting what you want the viewer to notice. Now that we have seen that human activity is organized either in centralized firms or decentralized markets, for millennia, we can appreciate that the organization and coordination of work in open-source projects are very unusual. Open source is neither centralized and it does not have prices.

Open-source production (not just software, but we'll focus mostly on software) is best described by Commons-based peer production. The term was coined by Harvard Law School professor Yochai Benkler.

Wikipedia has a great image that describes common-based peer production used long before computers were a thing.

Image credit - Wikipedia

Commons-based projects generally have less rigid hierarchical structures than those under more traditional business models. Benkler first introduced the term in his 2002 paper in the Yale Law Journal (published as a pre-print in 2001) "Coase's Penguin, or Linux and the Nature of the Firm", whose title refers to the Linux mascot and to Ronald Coase (remember him from four paragraphs ago).

Unlike a market or a firm, the inputs and outputs in open-source projects are shared freely. They are equally available for all to use as they choose at their discretion. So how does anything get done? How does an open-source project get people to work together without the command and control of a firm or the price-signaling mechanism of a market? 

Don Tapscott and Anthony D. Williams wrote a book called Wikinomics to study this. As the name suggests, their focus was to study the economics of Wikipedia which is a result of mass collaboration. Wikipedia is also organized based on commons-based peer production. They suggest that the coordination mechanism in a commons-based peer production system, like Wikipedia or open-source projects, are common incentives. "People participate in peer production communities," they write, "for a wide range of intrinsic and self-interested reasons....basically, people who participate in peer production communities love it. They feel passionate about their particular area of expertise and revel in creating something new or better."

So what are these incentives? 

Subscribe now

Incentives for open-source

Scratch that itch

One of the most immediate benefits of participating in open-source projects, at least for developers, is the ability to quickly address bugs and customize the software for personal use or use at work. Developers can freely access the source code, identify issues, and implement solutions without needing to wait for official updates from a commercial software provider. 

Signal to potential employers

Being a significant contributor to a major open-source project is a signal to any potential future employer. By showcasing their skills and expertise in a public forum, developers signal their value to potential employers and collaborators. This kind of visibility can lead to lucrative job offers and valuable networking opportunities.

Guido van Rossum, the creator of the Python programming language, exemplifies the long-term benefits of contributing to open-source projects. Python's widespread adoption and van Rossum's role as its chief architect have made him a sought-after expert in the industry. In 2013, he joined Dropbox as a software engineer and has been a distinguished engineer at Microsoft since 2020.

Peer Recognition: A Currency of its Own

The open-source community is a highly networked and collaborative environment, with developers frequently evaluating each other's work. By contributing to high-profile projects, programmers can garner the respect of their peers, an accolade that holds significant weight within the industry. This kind of recognition can lead to invitations to prestigious conferences, speaking engagements, and other opportunities for professional growth.

The prestige and respect associated with the open-source community are well illustrated by Linus Torvalds, the creator of the Linux kernel. His contributions to the open-source movement have earned him numerous accolades, including the prestigious Millennium Technology Prize in 2012. Torvalds is regularly invited to speak at conferences and has established himself as a thought leader in the software development world.

The Corporate Perspective: Strategic Participation

Companies also stand to benefit from participation. By contributing to open-source projects, businesses can influence the development of technologies that are relevant to their interests, ensuring that they remain competitive and up-to-date. Moreover, involvement in open-source communities allows companies to identify and recruit top talent, while simultaneously demonstrating their commitment to innovation and collaboration.

IBM's involvement in the Apache Web Server project showcases the strategic benefits of corporate participation in open-source projects. By investing in Apache's development, IBM was able to influence the evolution of web server technology in a way that aligned with their interests. Moreover, their involvement in the open-source community positioned IBM as a leader in innovation and collaboration.

We have been focusing on the creator, the developer, or the collaborator, in the open-source process. Open-source projects are organized based on principles of commons-based peer productions and the actions of the contributors are guided by shared incentives. But what about those that use open-source projects? What do they get out of all this? 

We are now going to look at the value that is generated for those using open-source projects. We'll look at Value-Chain analysis and a framework to analyze different types of values. This will allow us to analyze different kinds of open-source projects based on the four categories that we discussed last week - standards, frameworks, libraries, and applications. 

What is a Value-Chain?

From Investopedia, 

A value chain represents all the activities and processes involved in creating a product or performing a service. As such, it encompasses every stage of the product's or service's lifecycle, from design to production and distribution.

Each step in the chain adds value for the final customer. 

Let's analyze the value chain of Tesla, Inc. 

Inbound logistics:

Tesla sources its raw materials like lithium, cobalt, and nickel from various global suppliers. The company also collaborates with Panasonic for its battery production. Tesla has prioritized vertical integration and local sourcing to reduce lead times and transportation costs.

Outbound logistics:

Tesla uses a direct-to-customer sales model, bypassing traditional dealership networks. This allows the company to control the entire customer experience, from ordering to delivery. Tesla delivers its vehicles to customers either through delivery centers or direct home delivery.

Operations:

Tesla's manufacturing facilities include the Gigafactory in Nevada, which produces batteries and powertrains, and the Fremont factory in California, which assembles vehicles. Tesla is also expanding its manufacturing capacity with new Gigafactories in Austin, Texas, Berlin, Germany, and Shanghai, China. These facilities use advanced automation and robotics to streamline production processes.

Marketing and sales:

Tesla's marketing strategy relies heavily on word-of-mouth, social media, and Elon Musk's brand. The company does not invest heavily in traditional advertising. Tesla showrooms and online configurators enable customers to customize their vehicles and place orders. The company also offers over-the-air software updates, improving the customer experience after purchase.

Service:

Tesla operates a network of service centers and mobile service vehicles to provide maintenance and repair services for its customers. The company also offers remote diagnostics and over-the-air updates, minimizing the need for physical service appointments.

It is easy to see that Tesla is creating a lot of value and it is capturing a lot of value for itself. But that is not always the case for all firms. 

Leave a comment

Value chain in open source

How do we apply Value-Chain analysis to open-source projects? The end goal of all software is to make the life of an end-user easier. This is true whether we are thinking of Instagram or TikTok or enterprise software like Salesforce.com. 

Last week we introduced four different types of open-source projects. 

• Standards

• Libraries

• Frameworks

• Applications 

These different types of repositories are parts of a value chain. 

Let's perform a value chain analysis of a front-end web application built using the React JavaScript library, focusing on the categories of applications, frameworks, libraries, and standards. 

Standards:

• HTML: The markup language used to structure content on the web.

• CSS: The stylesheet language used for styling and layout of web pages.

• JavaScript (ECMAScript): The programming language used for creating dynamic and interactive web applications.

Libraries:

• React: A popular open-source JavaScript library developed by Facebook for building user interfaces, allowing for the creation of reusable UI components.

• Redux: A library used for managing application state predictably, often used in combination with React.

• React Router: A library for handling navigation and routing within a React application, allowing for the creation of single-page applications with dynamic content.

Frameworks:

• Create React App: A popular tool for setting up a new React project with a predefined structure, build tools and best practices.

• Next.js: A framework for server-rendered React applications, offering features like automatic code splitting, server-side rendering, and static site generation.

• Gatsby: A static site generator and framework built on top of React, focused on performance and developer experience, often used for creating blogs, portfolios, and other content-driven websites.

Applications:

• Frontend web application: The final product that users interact with, consisting of user interfaces and dynamic content built using React and its associated libraries and frameworks.

• Development tools: Tools such as Visual Studio Code or WebStorm are used by developers for writing, editing, and debugging the codebase of the React application.

• Version control and collaboration: Platforms like GitHub and GitLab are used for managing the application's source code, enabling collaboration among developers, and tracking changes over time.

The frontend web application built with React relies on established web standards, utilizes various libraries to streamline development, and incorporates frameworks for specific use cases. Finally, the application itself is created and maintained using development tools and collaboration platforms that enable efficient and effective development processes.

To monetize open-source projects we need to discuss how to capture value back for the creators. We want to balance value creation with value capture. 

The Value Creation – Value Capturing Framework

We are now going to look at a framework that allows us to see how to balance value creation with value capture. Paul Verdin and Koen Tackx propose The Value Creation – Value Capturing framework. 

Image credit - framework

The Value Creation-Value Capture framework consists of four quadrants: Hell, Nightmare, Heaven, and Dream. Companies in each quadrant have varying levels of value creation and value capture. 

Hell (Low-Value Creation, Low-Value Capture)

Companies or projects in this quadrant neither create significant value for customers nor effectively capture value for themselves. They often struggle to compete and may face declining market share or eventual obsolescence.

Before its decline, Blockbuster was a movie and video game rental company that failed to adapt to digital streaming trends. This resulted in low-value creation for customers and low-value capture for the company, as competitors like Netflix and Hulu emerged. 

Example from open-source world that fit this quadrant.

• Standards: Outdated or obscure standards that are no longer relevant, such as Gopher, a protocol that preceded the World Wide Web but is rarely used today. Another example is OpenID. OpenID is an open standard for user authentication and authorization, which allows users to log in to multiple websites using a single identity. While it initially gained some traction, it didn't create significant value for most users, and many websites have since migrated to other authentication mechanisms such as OAuth or social media logins. As a standard, OpenID didn't capture direct value but aimed to streamline authentication across the web.

• Libraries: Abandoned or deprecated libraries that offer little value to developers, such as unmaintained jQuery plugins.

• Frameworks: Unmaintained or outdated frameworks, like the Ruby on Rails-based Hobo, which has not seen any updates since 2015 and is no longer recommended for use.

• Applications: Obsolete or discontinued applications, such as the once-popular Winamp media player that lost its relevance with the rise of streaming services and modern media players.

Nightmare (High-Value Creation, Low-Value Capture)

Entities in this quadrant create substantial value for customers but struggle to capture value for themselves. They may face difficulties in monetizing their offerings or have to rely on alternative revenue streams, such as donations or sponsorships.

SoundCloud is a music and podcast streaming platform that allows users to upload, promote, and share their content. While SoundCloud creates significant value for users by providing a platform for discovering and sharing music, it has struggled to monetize its services effectively, making it difficult for the company to capture value.

Example from open-source world that fit this quadrant.

• Standards: Open standards like HTML, CSS, and JavaScript, which form the backbone of the internet and provide immense value, but cannot capture value directly as they are not owned by any organization.

• Libraries: Popular libraries like the Lodash JavaScript utility library or the Requests library for Python, which are widely used and create value for developers but capture little value directly, relying on donations or sponsorship for financial support. Another example is OpenSSL. OpenSSL is an open-source library that implements SSL and TLS protocols, ensuring secure communications over computer networks. It is widely used and creates significant value for developers and users by providing robust encryption tools. However, as an open-source project, it captures little to no direct value, relying on donations and volunteer efforts to sustain its development.

• Frameworks: Open-source frameworks like Django (Python) and Express.js (Node.js) enable rapid development and create value for developers, but they typically don't capture value directly.

• Applications: Open-source applications like the VLC media player, which offers a feature-rich, versatile media player but doesn't capture significant value directly.

Heaven (Low-Value Creation, High-Value Capture)

Companies or projects in this quadrant capture a high amount of value despite offering relatively low value to customers. They often leverage strong branding, exclusivity, or market positioning to maintain premium pricing and high profit margins.

Luxury fashion brands like Gucci, Louis Vuitton, and Prada focus on high-value, exclusive products with strong brand recognition. While these products may not offer significantly higher utility compared to more affordable alternatives, these luxury brands can capture high value through premium pricing and perceived exclusivity.

It's challenging to find open-source projects that fit into this category, as open-source software usually aims to provide high-value and community-driven development. Additionally, open-source projects tend to have limited means of capturing value, as they generally rely on donations, sponsorships, or offering related commercial services.

Dream (High-Value Creation, High-Value Capture)

Entities in this quadrant excel at both creating value for customers and capturing value for themselves. They often offer innovative, high-quality products or services and have strong brand loyalty, allowing them to maintain a competitive advantage and achieve sustained growth.

Apple creates high value for its customers through well-designed, innovative products and a seamless ecosystem of software and services. The company also captures significant value through premium pricing, strong brand loyalty, and an extensive network of retail stores and authorized resellers.

Django is an open-source web development framework for Python that encourages rapid development and clean, maintainable code. It creates high value for developers by offering a structured and efficient way to build web applications. While Django itself is free, it indirectly captures value through the Django Software Foundation, which offers paid support plans, commercial sponsorships, and community donations.

We have covered a lot of ground today, and we'll stop here for this week. This foundational work sets us up for discussing how to monetize open-source projects. 

As a reminder, this is part two of a five-part guide, with one bonus section, on how to monetize open-source projects. 

Leave a comment

You can read Part I here.

De-Magic: Understanding Deeply

Monetizing Open Source [Pt 1]

Open-source software has long been the bedrock of innovation and collaboration, powering some of the most transformative technologies of our time. According to the Boston Consulting Group, 99% of Fortune 500 companies currently use open-source software…

Read more

3 years ago · 2 likes · Aseem Asthana

See you next week for part three of this guide on monetizing open-source software. Don’t forget to subscribe.

Subscribe now

According to the Boston Consulting Group,

• 99% of Fortune 500 companies currently use open-source software

• 80% of IT departments will increase their use of open-source software

• 35% of all enterprise software is based on open-source code

Generating revenue and establishing a sustainable business model for open-source projects continue to be significant challenges, as these projects have typically faced funding difficulties. The issues faced by OpenSSL serve as a prime example. OpenSSL is an essential security software used in various applications, including web transactions. Despite its importance, it has historically received a mere $2,000 in funding. In 2014, the Heartbleed vulnerability brought attention to the crucial nature of OpenSSL, resulting in a funding increase of $11,000. However, this sum remains insufficient considering its critical role.

But there are success stories as well.

Innovating on monetization models – Red Hat to Databricks

Red Hat blazed the trail in monetizing open source when they started providing support and services for Linux. They were later acquired by IBM for $34 billion. More recent successes have been Databricks, valued at $34 billion in a private valuation in 2021, and HashiCorp, valued at $14 billion at IPO.

According to Bessemer Venture Partners, companies based on open-source projects are founded approximately 3.5 years after project creation.

Image credit – Bessemer Venture Partners link

Many readers might be astonished to find HashiCorp and Databricks categorized as open-source companies. In a sense, this highlights their success. These companies have achieved the commercial success that their target market does not anticipate receiving free offerings from them. They have effectively honed their market strategies and sales approaches.

Monetization Mistakes

The contrasting trajectories of SourceForge and GitHub serve as a compelling cautionary narrative. SourceForge, a web-based service established in 1999, provided developers with source code repositories, bug tracking, and an array of tools for open-source projects.  SourceForge’s code was released as open-source software in 2000 under the GPL license.

For a decade, spanning from the early 2000s to 2010, SourceForge was the veritable epicenter for hosting source code repositories.  Their revenue grew from $26 million in 2006 to US$ 92 million in 2009. They monetized their open-source project as a media site and made money through banner advertising.

In 2013, they attempted to expand monetization in a series of efforts that ultimately killed them.

1.     Bundling adware with installers: SourceForge began bundling adware with the binary installers of some open-source projects hosted on its platform. This practice was widely criticized as it undermined user trust, potentially exposed users to unwanted software, and went against the open-source philosophy of providing clean and transparent software.

2.     Taking over abandoned projects: SourceForge started taking control of seemingly abandoned or inactive projects and continued distributing them with adware included in their installers. This further tarnished the platform's reputation and resulted in pushback from the community.

GitHub was profitable right from the start. They generated $1000 on their first day.

In the early days of GitHub, the founders—Chris Wanstrath, Tom Preston-Werner, and PJ Hyett—adopted a bootstrap strategy to launch their venture. Established in 2008, GitHub emerged as a platform for developers to host, share, and collaborate on code using Git, a distributed version control system created by Linus Torvalds.

Despite facing competition from established platforms like SourceForge, GitHub prioritized simplicity, user experience, and social coding, thereby distinguishing itself from its rivals. The platform's freemium business model allowed users to create public repositories for free while charging for private repositories, thus ensuring profitability from the outset.

GitHub did a few things right and SourceForge did some things wrong when it comes to monetization. What lessons can we learn from the five companies, and can we distill the learnings?

That is exactly what we will cover in this guide over the next five weeks. 

Technology, product, business

If you are considering monetizing your open-source project, it's important to understand that you haven't yet established a business based on it. At this stage, you possess either a product or the underlying technology, both of which hold the potential to evolve into a fully-fledged product and business.

When it comes to revenue and monetization

What is the difference and why does it matter?

Raw technology refers to the various tools, methods, and frameworks employed in the development and maintenance of software products. A repository that requires extensive code customization is raw technology and not a product, as it does not directly address specific use cases. However, with the necessary customization, it can be tailored to solve those use cases.

A product enables users to utilize technology while offering convenience and refinement. The key distinction is that technology supplies the fundamental building blocks, which must be tailored by the intended user. In contrast, a product can be used without any customization by the end user. In the context of open-source projects, many repositories considered products are designed for developers, who are the end users in this case.

The Linux kernel is a technology, not a product.

Although the Linux kernel is a fundamental part of numerous Linux distributions, it often needs significant customization and configuration before being used in a specific product or environment. Developers and system integrators must tailor the kernel to their specific hardware, performance requirements, and desired features, which often involves modifying the kernel's source code, selecting appropriate modules, and configuring various settings.

The distribution is the product, while the kernel is the technology.

Whether a project is for the front-end, middleware, or back-end does not determine if it is a technology or product. Backend or even networking libraries can be considered products.

Socket.io is a networking library that qualifies as a product. The target users for socket.io are other developers that are building their products. But the critical insight is that these developers can use socket.io without any customization of socket.io’s source code. It is a complete product that solves use cases for these developers.  Another example of a product, even though it sits deep inside code is TensorFlow, an open-source machine learning library developed by Google. TensorFlow provides a flexible platform for building and deploying machine learning models, but it is a complete product for its intended users.

However, it is important to note that a product by itself does not generate revenue. You need to build a business around a product.  Regular, recurring, and repeat monetization of any human activity is classified as a business.

You need to know which state is your open-source project in before you can start to think about monetizing it. If your project can already be considered a product, then it will be easier to monetize it. There are also more ways to monetize a product compared to raw technology. 

Monetization is going to be tough but for reasons you don’t want to think about

Monetizing your open-source project is hard but for reasons that you might not expect. It is hard because of the reasons that made your open-source project successful in the first place.

Technical people are often introverts and find it a challenge to talk to strangers.  Writing code requires a high degree of concentration and focus, which can be disrupted by social interactions. Technical people often prefer to work independently or in small groups with individuals who share their interests and expertise.

Once you start the process of monetizing your work, you can no longer focus on just the technical parts, or only work with other contributors to your open-source project. You are going to have to transition to the role of the founder and CEO of the business that you are trying to build.  In the early days and for a long time, you are not going to be able to outsource this work to a product manager or VP of Sales.

This part is so crucial to your success in monetizing that we will repeat it. As you start monetizing your work you will have to push the boundaries of your comfort zone and take on the role of a founder AND the VP of sales.

In part III, which comes out on April 21, we’ll cover a detailed and step-by-step guide on how to select which customers to talk to, how to structure the conversation, and how to digest that feedback. This step-by-step process will make sure you have a detailed and actionable plan for each part of the process.

The other hard parts about monetizing open-source projects

Betraying ideals

Most open-source projects begin as passion projects, driven by a desire to solve a problem, pursue a passion, or contribute to the community. As they gain traction, these projects often attract a following and grow into large, vibrant communities.

Many founders and maintainers are reluctant to pursue financial gain, as it may conflict with the altruistic and collaborative ideals that underpin the open-source movement. This reluctance can make it difficult to find sustainable revenue streams and create a viable business model.

Despite these challenges, it is possible to monetize an open-source project without sacrificing its founding ideals. By taking an innovative and flexible approach, We’ll talk about the guard rails you can deploy to protect your open-source project and its community while you pursue monetization.

Sustainable moat

Even when companies find a good business model, like Red Hat, they struggle to innovate and grow.

A while back Peter Levine had written,

There are many reasons why the Red Hat model doesn’t work, but its key point of failure is that the business model simply does not enable adequate funding of ongoing investments. The consequence of the model is minimal product differentiation resulting in limited pricing power and a corresponding lack of revenue. As shown below, the open-source support model generates a fraction of the revenue of other licensing models. For that reason, it’s nearly impossible to properly invest in product development, support, or sales the way that companies like Microsoft or Oracle, or Amazon can.

We’ll discuss ways, in this article, to keep the balance between the founding ideals and the requirements of a business.

Different types of Open-source projects

There is one final piece of mental scaffolding that we need to build before we can start thinking about the economics of open source (in part II, which comes out next Friday). 

We have already discussed categorizing projects as raw technology, product, or business. Another dimension of categorizing software projects (open source or not) is to determine if the project is a finished application, library, framework, or standard. 

1.     Applications: Applications are software programs that have a specific purpose and can be used directly by end-users to perform tasks or solve problems. These applications are typically standalone and can range from productivity tools, such as word processors and spreadsheets, to entertainment and communication platforms, such as video players and messaging apps. Microsoft's Visual Studio Code is an end-user application for developers. 

2.     Libraries: Libraries are collections of pre-written code, functions, or routines that can be easily integrated into other software projects. An application can and usually does, use multiple libraries for different things. These libraries save developers time by providing reusable components and utilities that address common problems or perform specific tasks. Examples include programming libraries for handling graphics, audio, and network communication. Lodash is a library of utility functions for JavaScript developers. 

3.     Frameworks: Frameworks are structured collections of code and tools that provide a foundation upon which developers can build applications or systems. These frameworks often establish a standard structure, methodology, or set of conventions that streamline the development process and enable developers to create applications more efficiently. Examples of open-source frameworks include web application frameworks, such as Django or Ruby on Rails, and game development frameworks, such as Unity or Unreal Engine. It is not uncommon for an application to use multiple frameworks. For example, it might use NextJS for front-end development, NestJS for backend development, and TensorFlow for machine learning. 

4.     Standards: Open-source standards are specifications, protocols, or guidelines that help ensure consistency, interoperability, and compatibility among different software systems and platforms. These standards are typically developed and maintained by industry groups, organizations, or consortiums and are often implemented in open-source software to facilitate widespread adoption. Examples of open-source standards include data formats, such as XML or JSON, and communication protocols, such as HTTP or MQTT. Language specifications like ECMAScript 2024 or the Python language reference are also standards. 

Each item mentioned previously is considered a product, rather than a technology. Standards, such as language specifications, also qualify as products since they can be utilized by their target users without modifications or customization. In the context of a standard language specification, the intended users are developers who create compilers and interpreters for that language, making the language specification a comprehensive product.

The monetization strategy we choose will depend on whether the project that we are attempting to monetize is an application, library, framework, or standard. 

Homework

In preparation for next week, look at these 10 projects and decide if they are.

-       A technology, product, or business

-       An application, library, framework or standard

1.     https://jaykmody.com/blog/gpt-from-scratch/

2.     https://github.com/numpy/numpy

3.     https://docs.oracle.com/javase/specs/

4.     https://github.com/expressjs/express

5.     https://github.com/microsoft/JARVIS

6.     https://github.com/Torantulino/Auto-GPT

7.     https://github.com/ethereum/go-ethereum

8.     https://github.com/grafana/grafana

9.     https://github.com/signalwire/freeswitch

10.  https://www.unrealengine.com/en-US/ue-on-github

Comment below on what each of these projects are.

Leave a comment

What we won’t cover

Our focus will be on monetizing your open-source project, but we will not cover any guidelines on raising capital for your company, branding or marketing or hiring. There are excellent guides on all these topics, and we’ll give you a link to all of those, but we won’t cover those ourselves here.

Leave a comment

What’s Next

This guide is designed for authors and maintainers of open-source projects, as well as business owners and product managers in companies investing in or acquiring open-source businesses. It will provide a solid understanding of core business principles, practical insights, checklists, and case studies featuring both successful companies and those with room for improvement.

Each Friday, over the next four weeks, you’ll receive one part of this guide. 

• De-Magic: Understanding Deeply

  Economics of Open Source: How to monetize open-source projects [part 2]

  This is part two of a five-part guide on how to monetize open-source projects. Part I, Introduction. We discuss common monetization mistakes and frameworks like Business > Product > Technology, reasons why monetizing open-source projects is hard, and finally, different types of open-source projects…

  Read more

  3 years ago · 2 likes · Aseem Asthana

  Part II covers the economics of open source. Why does open source exist and what value does it contribute? We’ll also talk about how to determine what value your specific open-source project delivers to your intended customers. Notice that the word here is customers and not users. Customers pay, and users don’t have to. 

• On April 21, we’ll cover a detailed plan on how to get feedback from customers. This will include deciding which kinds of customers to talk to, what kinds of questions to ask, and how to collate all that information to write down specific use cases, benefits, and friction points for customers. This part will include a worksheet that you will be able to fill in. This work is going to be critical to the next step. 

• We will use the information that you filled in the worksheet to decide monetization scheme and actual pricing on April 28. 

• We’ll cover creating long-term value and building a sustainable business on May 5.

• Finally, as a bonus, we’ll cover Artificial Intelligence and Open Source on May 12. 

  Don’t forget to subscribe to get all the parts of this guide

See you next Friday.

The Jobs Impact Quotient breaks down each job into six attributes. It analyzes the importance of each of the attributes on success in the job. It then asks how good AI is in those areas and how we can expect AI to improve in those areas. This allows us to calculate the Jobs Impact Quotient. A higher number indicates that AI and automation is more likely change the nature of this job, or completely disrupt it.

We can bucket jobs into three categories based on their Job Impact Quotient.

• Can be disrupted anytime - these jobs will fundamentally change, and many of them will vanish.

• Last mile interface for AI and automation - The primary responsibility of people who do these jobs will be to monitor, oversee, and referee AI and automation. They will provide a bridge between machines and society - sort of a human gearbox.

• AI and automation as helper - These jobs will increase their reliance on AI and automation.

• No impact expected

Here is how the nature of some jobs will change.

Let’s understand.

Don't focus just on ChatGPT; It has friends.

Focussing on ChatGPT and Dall-E is looking at the world through a narrow tube. We need to broaden our focus and think about the explosive combination of AI and robotics, which leads to automation in every sphere of our lives. 

ChatGPT, Dall-E, and Midjourney belong to a class of AI known as Generative AI. They got a lot of press last year. We have collectively forgotten predictive AI, the previous type of machine learning algorithms. Predictive AI can generate music, read medical imaging to diagnose medical issues, and even help train salespeople to sell better. 

Predictive and generative AI are complementary to each other.

Robotics 

Robots currently serve us behind the scenes in warehouses, logistics, and shipping. With the rise of e-commerce, the demand for automated solutions in the supply chain has increased significantly. Warehouses and fulfillment centers are adopting robotics to improve efficiency and reduce costs. Autonomous vehicles and drones are being used for last-mile delivery, enabling faster and more flexible delivery options for consumers. In addition, robots are used for inventory management, sorting, and packing, which can be repetitive and physically demanding for human workers.

 Robots are also making their way to the front of the curtain in other industries, such as restaurants and hospitality. In some countries, restaurants have adopted robotic servers and chefs to improve efficiency and reduce labor costs. These robots can take orders, prepare food, and interact with customers. Additionally, robots combat loneliness among seniors and people with disabilities. Companion robots can provide emotional support and social interaction, improving mental health and well-being. As technology continues to evolve, robots will become even more integrated into our daily lives, enhancing our experiences in both practical and emotional ways.

AI allows computers to play in the world of information and knowledge. At the same time, robotics enable computers to manipulate the real world. Robotics and AI combined allow the automation of many parts of our society, industry, and personal life. 

Share De-Magic: Understanding Deeply

Unbundling jobs into tasks 

There is a lot of historical and emotional baggage with occupational descriptions like doctor, teacher, and soldier. A teacher is respected, a soldier is honored (or feared, depending on which side of the gun's barrel you are), and a doctor provides solace. When you look at the world through these lenses, it is hard to imagine that those jobs will be impacted by technology and automation. 

If we unbundle jobs into the tasks done by them, we can find what is common in all the jobs, and think about how automation might impact them.

Each job is made up of a large number of constituent parts. James Cameron, creating Avatar, or a fry cook in Mcdonald's all rely on tools, best practices, and common wisdom to do their jobs. The level of creativity differs, but this commonality allows us to analyze jobs.

The roots of thinking about the constituent parts go back to the late 19th century when Frederick Winslow Taylor developed a theory called Scientific Management. Taylor believed that the efficiency of the work process could be optimized by breaking down tasks into smaller, simpler components. 

The principles of scientific Management include the following:

• Division of labor: Breaking down work into smaller, specialized tasks to increase efficiency

• Time and motion studies: Analyzing each task to determine the most efficient way to perform it

• Training and development: Providing workers with the necessary skills and knowledge to perform their assigned tasks efficiently

• Incentive systems: Offering rewards and punishments to motivate workers to work at maximum efficiency

• Management control: Managers should have centralized control over the work process to ensure it is carried out efficiently.

Before the advent of scientific Management, work in industries was generally characterized by a lack of standardization and a heavy reliance on the skills and experience of individual workers. Workers were expected to use their judgment and initiative to complete their work. 

Scientific Management moved factory work from art to a craft. This began a long arc where each job would be scrutinized and broken down into smaller parts.  

Tools and rules

Each job, profession, occupation, and trade uses specific tools. Every tool serves a unique purpose, from the simple pen and paper used by writers and journalists to the complex surgical instruments used by doctors and nurses. Tools can be physical, like a hammer or scalpel; digital, such as Microsoft Excel or a programming language; or knowledge-based frameworks, like Porter's five forces for analysis or negotiation skills in sales.  

These tools are accompanied by accepted standards and best practices. For example, a chef has a set of standard knives used to carry out different tasks. There are specific ways in which these knives should be used to ensure safety and efficiency. Similarly, software developers use standard frameworks and libraries to write code, which helps to ensure the code is maintainable and scalable. Best practices for using tools often develop over time as professionals learn from their experiences and share knowledge with others in their field. In many cases, these best practices become standard practices that are widely adopted by the industry, further emphasizing the importance of using the right tools in the right way.

Skills and attributes 

Looking at the tools and rules used by each profession allows us to think about what skills and attributes are fundamental to doing the tasks in a given job. 

Focus on critical and essential attributes, not nice to have

Many cardiovascular surgeons are polite and empathetic. Neither their job performance nor annual compensation really depends on their bedside manners. On the other hand, a B2B enterprise salesperson's salary will be highly dependent on how well they communicate with the customer. A rude or standoffish salesperson is going to sell little. So while both professions employ soft skills like communication with empathy, those are crucial to success in being a salesperson. At the same time, they are less critical to be a world-class surgeon. 

Six core skills and attributes

Empathy-based communication 

Empathy is different from sympathy. Sympathy is feeling sorry or concerned about someone based on their pain or other personal situation. Empathy is recognizing and understanding another person's emotional state and responding appropriately.

A childcare worker has to be able to communicate with children who often think and behave very differently from adults. Children tend to be more impulsive and emotional than adults, as they are still developing self-control and emotional regulation skills. This can lead to tantrums, outbursts, and other impulsive actions that may seem inappropriate to an adult. Children are not small adults. 

The person in charge of a child's well-being has to use empathy to understand the child's concerns and communicate. Empathy-based communication is critical to the success of a childcare worker. 

If your job is to make art hanging on hotel room walls, then you will not need much communication and very little empathy. You will need communication, and empathy, in selling your art to hotels, but not in creating the art.

Some jobs that require a high degree of empathy-based communication are 

• EMTs

• Primary school teachers

• Childcare worker

Some jobs that require very little empathy are 

• Commercial art painters

• Garbage truck drivers 

Judgment-based decision making

All jobs require some level of decision-making, but not all decision-making is the same. 

A customer service representative has to make many decisions. Still, their actions are dictated by playbooks, guidelines, scripts, and standard operating procedures (SOP). Decision-making is primarily dictated by rules, not skills or judgment. 

A surgeon repairing a diseased heart has to calculate trade-offs and risks and make decisions based on incomplete information and under time pressure. They make countless decisions based on years of education, training, and skills. Their job requires a lot of judgement-based decisions.

Some jobs that require independent decision-making based on years of education and experience are 

• Air traffic controllers

• Cardiovascular surgeons 

Jobs that don't require a high level of independent decision-making are 

• Adult education teachers

• Baristas

Creating new knowledge  

A cardiovascular surgeon's job requires high levels of independent decision-making and judgment, but they are not required to create new procedures. They do produce new techniques as a side-effect of their work. Still, if a cardiovascular surgeon did not publish a single paper in their career, their annual compensation won't be affected.

Creating new knowledge is optional for a cardiovascular surgeon.

On the other hand, a biochemist researches a broad range of areas, including genetics, metabolism, protein structure and function, cellular signaling, and drug development. They expand our understanding of how living organisms function at a molecular level. Their primary job is to create new knowledge.

The envelope of human knowledge is expanded when someone produces a genre-defining movie, an architect pioneering a unique building style, or a music composer creating a new musical style.

A word of caution is in order here. The jobs listed above sound like they are always creative, but a lot of work done in those fields is derivative and formula based. Hollywood and Bollywood produce many sequels and franchises that fit a mold, and architects have names of types of buildings.

Some jobs where the primary objective is the further either scientific or creative knowledge are 

• Executive chefs creating new recipes 

• Developing a unique style of painting (like Impressionism or cubism) 

Some jobs where success is not correlated with creating new knowledge or thought are 

• Voice over actors

• Baristas 

• Optometrists 

• Line cooks

Subscribe now

Assembling existing knowledge in innovative ways 

Not all creative work results in new knowledge. Creativity is not always the same as new. Many creative exercises draw on existing knowledge.

Software developers are among the highest-paid professionals. But their work is increasingly like making a model of The Death Star using lego blocks. They spend a lot of time writing code to glue various open-source packages and libraries into popular frameworks. 

They rely on an extensive repertoire of knowledge and spend a lot of time assembling it in new ways. 

For example, GitHub introduced GitHub CoPilot in 2022, which can write up to 40% of your code. Their CEO, Thomas Dohmke expects Copilot could write up to 80 percent of code within the next five years. 

Another job that requires combining a lot of existing knowledge in new ways is creating new recipes. 

Chefs rely on a vast knowledge of cooking techniques, ingredients, flavors, and cultural cuisines. They often draw on their existing knowledge to create new and unique dishes by combining various ingredients and methods in novel ways. 

McCormick & Company announced it collaborated with IBM to develop an AI application capable of generating flavors and food products. 

Another example where someone used existing knowledge to diagnose a rare disease appeared in ProPublica.

Some jobs where it is valuable and essential to assemble new ideas from an extensive repository of knowledge are 

• Paralegals

• Marketing copywriters 

• Travel guides

Some jobs where this is not a requirement are 

• Optometrists 

• Air traffic controllers 

Diverse motor skills 

Many jobs require someone to be on-site and manipulate tools to get the job done. Physical presence and control of their environment using diverse motor skills is essential in these jobs. The word diverse should be thought of together with unpredictable and unplanned. Imagine a plumber only finding out the exact situation after they get in position under the kitchen sink. 

Some jobs that require diverse motor skills are 

• Plumbers

• EMTs

• Childcare workers 

Some jobs that don't require diverse motor skills are 

• Voice over actors 

• Tour guides

• Air traffic controllers 

Trust 

Trust is different from empathy-based communication. We don't need the heart surgeon to empathize with us. We need to trust that they are a consummate professional who will do their best. There is an emotional element to trust.

Examples of jobs where trust is critical are 

• Childcare workers 

• Teaching primary school children

• Cardiovascular surgeon 

Conversely, do you need to trust the tour guide when visiting Rome? You need to like them, and they need you to listen to them, which they can do if they empathize with you and tailor their delivery based on that. Empathy-based communication trumps trust here.

Similarly, society does not need to trust those driving garbage trucks. We need to trust the system that garbage will be picked up and the drivers they hire will drive those trucks safely. 

The rating scale for each attribute

Now that we understand the importance of different skills and attributes, we will discuss how to rate the importance of each skill/attribute for various jobs. It helps to remember that all jobs are made up of many tasks. 

Here is a rating scale that goes from 1 (not important) to 5 (critical to success on the job) 

1. Not important. This skill is optional to success. On the internet, nobody knows you're a dog. 

2. Somewhat important. You won't get fired if you don't have this skill, but it helps. A flight attendant that makes funny announcements reduces the drudgery of flying.  

3. Important. It a nice to have. You will still do fine, but it will make the difference between the top 1% of those in this field. You can still be great at baseball without mastering Michael Jordan's Fadeaway.

4. Very important. It will be a handicap if you don't have this skill. You might still be successful if you don't have it, but it will take an extraordinary effort in other areas. Think of presidential candidates kissing babies. 

5. Critical. If you can't hack this skill, do another job. Required for you to have the job, but it's table stakes. If you can't master an en garde, you can't even begin fencing.  

An example might help. We'll use our stereotypical residential plumber for this example. 

• Empathy-based communication: Important.

• Decision-making based on skills and judgment: Very important

• Creating new ideas: Not important 

• Assembling existing knowledge in innovative ways: Somewhat important 

• Diverse motor skills: Critical 

• Trust: somewhat important  

AI Capabilities and limitations

The final puzzle piece is understanding how AI and automation rank on the six skills and attributes, both now and in a decade. While predicting the future is hard, we will do our best to read the tea leaves. 

We'll rate each skill on a capability rating of one to five, where each rating means the following. 

1. Developing. Like a baby trying to walk. Fledgling

2. Competent. Get the job done but nothing to write home about. Like being able to prepare a weekday meal for the family.

3. Skilled. Can handle most essential tasks with care.  

4. Expert. Usually achieved after years of training. A sushi chef.

5. Master. They set industry standards. Like Julia Child or A. R. Rahman. 

So how do AI and automation rank on the critical skills and attributes? 

Use existing knowledge Predictive and Generative AI (ChatGPT and Dall-E/Mid-journey) are experts in using existing knowledge. With the digitization of the world's existing knowledge, they will become Masters in 10 years. 

Creating new knowledge We don't expect any kind to expand beyond Developing.

Communication with empathy AI capabilities in communicating with the heart is just Developing. Advances in Generative and Predictive AI will increase this to at least skilled in the next decade. 

Trust Society's trust in Automation is Developing, but for many reasons, including the fact that a lot of automation will be cloaked in human likeness, hidden away from public sight, and general attitude changes, we expect society will start seeing automation as graduating to at least Skilled. 

Diverse motor skills Robots can navigate and manipulate the physical world with capabilities best described as Developing. Advances in robotics and computer vision will rapidly increase an expert's qualifications within the decade. 

Share

Our framework is starting to take shape. 

We are ready to assess the impact of automation on any job.

A combination of what skills are essential in a job and how well the current state of the art performs in terms of capabilities for those skills and the future will inform us and put things in perspective. 

As always, a good example will inform our intuition. A soldier on a battlefield is protected by a combination of the skills in things that matter (being able to write poetry is not necessary on a battlefield while being able to run while carrying it) and the capabilities of your adversary in those same areas. Are you equipped and trained in all the right areas, and is your opponent, better or worse, qualified and trained? 

Using this information, we can categorize the impact of jobs now and in a decade in the following words. 

No impact.

Can be disrupted anytime This job, as we know it, likely won't exist or will be done by very few people. One example is artists who sell their art to tourists or whose works hang in hotel rooms. 

Last mile interface for automation This job's primary responsibility will be to act as a helper for automatons. They will be the operators, maintainers, overseers, and referees. Warehouse workers, store checkout clerks, baristas, and paralegals' jobs will change here. Many primary care physicians, family practitioners, and some software developers are also impacted.

AI as a helper These jobs will increasingly rely on AI as a helper for productivity.

Leave a comment

Subscribe now

How can you use this information?

This essay presents a framework that you can use to analyze and estimate the impact of automation on your job or any other job. All you have to do is evaluate each core skill's importance to success. It is essential here to be as dispassionate as possible. This is not an exercise in judgment but analysis. 

The final number you get is not gospel. It is only helpful to put things in context. The number can help you see if the job you are trying to analyze falls in one of the following buckets.

• No impact

• Can be disrupted anytime

• Last mile bridge for automation

• AI as a helper

All jobs will be impacted by automation, much like how electricity has changed all jobs fundamentally, forever, and foundationally. Automation, driven by AI and robotics, is like that. 

Changing the fundamental building blocks  

One of the building blocks of this model is our analysis of how capable automation is in each core skill. We also project our beliefs a decade in the future. This is fiendishly impossible to do. Feel free to change those numbers as well. 

At De-Magic, we'd love to work with you to improve this framework. How can we make this better? Do you have suggestions for which skills are more or less necessary for different jobs? Did you use this framework for a job that is not listed here? If so, please tell us. 

Feature prioritization and updating roadmaps are two common bugbears for product managers – experienced and new alike! I talk to a lot of product managers and they always exclaim that the sales team does not understand the roadmap or that engineering is complaining that they don’t know what to build.

Product managers often hide behind Agile or Kanban methodologies and send the list of features they plan on building in the next few cycles to sales. “Our backlog is our roadmap,” you hear. It’s not. It’s a detailed list, not a plan. No wonder sales can’t use it to close deals — they can’t tell their prospects what’s actually coming down the pike in terms of features.

On the flip side, sending a deck that articulates your plan for dominating an industry vertical while outflanking competition is not going to help your scrum masters decide what to put in the next sprint. They need a list of features they can break down into tasks.

Everyone is frustrated with your roadmap because they don’t get what they need. Calling a list of features your roadmap is myopic. Talking purely about your vision without a strategy or a plan is just grandstanding.

Your roadmap needs to include a master plan of markets and the strategy, a tactical plan with a list of features that address specific use cases for customers in your market, and a “super-detailed” plan that includes features, stability, and quality.

The Master Plan

This is the strategic section of your roadmap. The most important part of the master plan is articulating the objectives. Don’t mistake this for organizational vision statements. We are not talking about ending world hunger but rather a plan that will help your organization reach its stated goals.

Are you trying to grow into a particular segment or market? Are you going to maintain revenues from a specific product line while investing in other, breakthrough technologies? Are you going to outflank a competitor by finding an overlooked but deep niche? It helps to have organizational, or at least business-group-level goals agreed to by all parties (investors, board, senior management, etc.).

A key source of friction is that the product roadmap is not aligned with go-to-market (GTM) and sales compensation. If marketing is continuing to find B2C customers through Facebook advertising and your master plan calls for selling yearly subscriptions of corporate travel booking assistance, then you are going to have a problem.

Any discussion of GTM will involve a discussion about customer segmentation. Which kind of customers are you targeting? For B2C businesses, geographic and demographic information is often helpful. For B2B firms, organizational size, industry, and verticals are a good starting point. Are you building and targeting stay-at-home dads in large cities or European firms in the industrial sector?

A clear understanding of the target market helps drive both your GTM plan as well as your product design.

Setting up the master plan, getting alignment, and communicating it clearly, is the bedrock of your roadmap.

Send this part of your roadmap when your investors or board ask for your roadmap. Be judicious in sharing this with anyone outside the company. There are occasions when it makes sense to share this with industry analysts like Gartner or Forrester (often under NDA).

Tactical roadmap

Eighty percent of all product managers spend time thinking about what features to build. They worry about prioritization and read a lot of Medium articles on how to rank potential features. There are a lot of frameworks for feature prioritization, but they are all wrong. Why? Because they are not aligned with your master plan. You can’t decide what is must-have, could-have, or should-have if the objectives and strategy are not decided.

Your tactical roadmap should include what you are going to build, in what order, and in what approximate time frames. Feature prioritization now becomes a bit easier. If your strategy is customer retention, you will prioritize stability, customer experience, and breadth of features while protecting from flank from upstart newcomers. If you are hoping to widen that deep and narrow niche as an up-and-coming vendor, you’ll double down and provide deep features for select use cases.

The tactical roadmap section needs to show a timeline of when capabilities will be delivered to the market. You’ll want to include complexity, risk, and time estimates from engineering into this part of the plan.

This is the part your customers need to see. They’ll learn that the use cases they care about most are going to be solved by your product. This is also the part that engineering and UX teams should see and sign off on.

Feature and task backlog

Building new features all the time without any regard for internal housekeeping is like eating candy for lunch, breakfast, and dinner! It might taste good (does it?), but be ready for the sugar crash. Paying down technical debt, rearchitecting for scaling, etc. is like flossing. It is not fun, but that inevitable toothache is even less fun.

Any good plan will include a healthy mix of new features, quality-of-life improvements, tech debt, stability improvements, and bug fixes. But your customers, investors, and board don’t need to see the sausage-making. Your product owners, scrum masters, and the engineers working on your products do need this level of detail.

A good litmus test of whether your strategy and tactical roadmaps are set correctly is whether your product owners and scrum masters need any help in grooming the backlog. Given sufficient clarity, they will be able to do this part themselves.

Tailoring your roadmap

The reason why no one is happy with the roadmap you’ve been providing is that you have been sharing the wrong parts with the wrong audiences. The product roadmap has three parts that need to both tell a story and provide a detailed operating plan. It’s easy to see what anyone wants to see when the roadmap is laid out this way.

This article was first published by me in ProductCraft on June 8, 2020.

Leave a comment

Understanding the differences between precisely coded algorithms and models that are not fixed in their programming but learn and can adapt is critical for us to ask questions about the technology that shapes and, increasingly, controls our lives. 

Artificial Intelligence (AI) uses models to learn and answer questions or generate new things. Algorithms have long ruled our world and armed armies of bureaucrats with the refrain of, "the computer won't let me do this. I am sorry." 

No one would need to know the difference between algorithms and models in an ideal world since both will be cleverly and helpfully hidden behind a humanizing layer. The abstraction breaks through when technology is accused of bias, plagiarism, or misinformation. Policymakers, informed citizens, journalists, and technologists need to understand the fundamental differences between technologies that primarily rely on algorithms and those that leverage machine learning models, or increasingly a hybrid. 

The difference between programming and creating models started out as just a semantic difference. Algorithms have been steadily creeping along the spectrum to more into models. In the last decade, the difference has become a kind, not a degree.

Historical Note: Greeks pioneered algorithms to solve geometry problems: The ancient Greeks used algorithms in geometry and mathematics. One famous example is the Euclidean algorithm, a method for finding the greatest common divisor of two numbers. The Euclidean algorithm was described by the Greek mathematician Euclid in his book "Elements" and has been used for thousands of years to perform division and find the greatest common factor of two numbers. The algorithm has also been adapted for use in computer science and is still used today in various forms. Their use of algorithms laid the foundation for the development of mathematics and paved the way for future generations to build upon their work.

Courtesy Visual Capitalist

Chinese abacus as an early computer: In 14th century China, algorithms were used to develop the abacus, an early counting tool. It is one of the earliest examples of using algorithms for practical purposes.

Tell me what to do 

Courtesy Visual Capitalist

Algorithms are a set of rules: The traditional way of getting computers to do things is to program them by encoding the algorithm in a language that computers can understand. Algorithms really are rigid sets of instructions to solve a problem. Any apparent flexibility comes from the programmer, who has to think through all the conditions the program can encounter and list those. 

Programming is good at capturing business logic: If you were responsible for pricing tickets for an airline, you'd have logic that says, "if the departure date is less than twenty-four hours, on a busy route and on a Monday morning, then raise the prices of tickets by 2% each hour." The airlines' program will include hundreds of such conditions. 

Programmers work with subject matter experts (SMEs) to encode this logic. This has led to interdisciplinary subjects like computational biology and the rise in stature of developers on Wall Street for trading.  

Failures are severe: This collection of rules will fail if the program encounters a set of conditions it does not recognize. This results in failures like hackers stealing sensitive data from credit card companies because they were able to mimic the conditions of being trusted employees. Another apocryphal example is when NASA lost a Mars orbiter because different parts of its programming used different units of measurement. Programming mistakes can be fixed or debugged by retracing the original logic and finding errors in the algorithm used to code the logic. It is time-consuming, expensive, and sometimes mind-numbing, but it is done by engineers daily.  

Correctness and performance: The correctness of an algorithmic system is determined by testing its output against a known list of inputs. Performance is determined by how quickly the program outputs a given set of inputs. 

Algorithms are like recipes: A good cake recipe turns out moist and delicious cakes. If you skip any steps, the cake will not rise, be dry, or worse. When it comes to computers, failures are usually the fault of the recipe writer, the programmer, since computers follow instructions faithfully. 

Teach a person to fish

Programming with algorithms can only solve some problems: Writing down everything a program is expected to encounter and how it should react is a poor way of modeling reality. This approach hits hard limits with complex problem domains like understanding what people are saying, translating from English to Mandarin, or recognizing objects and people in photos.

Early face detection was an algorithm in the guise of an AI model: The Viola-Jones face detection algorithm was state of the art for teaching a computer to detect faces in a photograph at the turn of the 21st century. It was programmed to look for features found in human faces – eyes, lips, mouth, and nose. It is looking for a bright T-shaped pattern with dark semi-circular patches below it, corresponding to the T-zone of the forehead and nose and darker eyes. 

The image above is from an excellent explanation that develops the intuition behind the Viola-Jones algorithm.

The algorithm worked well for its time, but its limitations are painfully obvious. They will only determine faces where people look directly at the camera, among other restrictions. 

Teach me, don't tell me:  AI gets computers to do things by teaching them. AI is trained, not programmed, to do something. This is similar to saying, "This is a dog" or "This is a cat" to a child when you see either a dog or a cat. The child learns to distinguish cats and dogs and can later spot the difference, even if they are looking at cartoon representations or soft toys.

The model is a closed, black box: The heart of AI, the model, is impenetrable and unknowable. Models learn their domains' words, vocabulary, grammar, and language. They are not acting on a set of rules but ideal outputs. Fixing AI models when they produce the wrong things is different from improving algorithms. 

Random but predictable: AI models are inherently random. They might not appear so since the mathematics that drives them forces them to produce predictable outputs (in a narrow range). This forces us to think about correctness in very different ways. 

Mathematical note: Random but predictable by a probability is mathematically modeled as a Stochastic process. A system is stochastic if each observation is unpredictable, but over time you can predict, with some likelihood, the range of outputs.

For example, consider flipping a coin. Every time you flip the coin, the outcome is either heads or tails, which is random and uncertain. As you flip the coin multiple times, the sequence of heads or tails forms a stochastic process.

Another example is the random walk. Imagine you're taking random steps, either forward or backward, where each step is a random event. The sequence of forward and backward steps over time is an example of a stochastic process.

In both cases, while the individual events (coin flips or steps) are random and uncertain, the overall pattern or trend of the events can still be studied and understood through probabilities and statistics.

AI adapts to changing conditions: If you programmed a computer to tell a dog and cat apart, it wouldn't be able to adapt when you ask the same question using toys instead of real animals or when the lighting condition is not so good. While the Viola-Jones face detector won't work on photos that don't look directly at the camera, an AI model can quickly learn that. An AI model can give you the wrong answer, like how a young child might get confused between an image of a dog and a cat.

Bias is also learned: AI models can fail in various ways, often by learning to discriminate against some people or using hurtful and toxic language. AI models can also veer off course over time as the input data changes or if the interpretation of that data changes. Fixing these mistakes is about recognizing errors, which are not always obvious, and then either retraining or finding a better model. The advanced models are like black boxes and don't allow an easy understanding of the inner workings. Most car drivers don't have any knowledge of how engines work but are still able to drive to get to work. 

AI models are like children: Any parent knows that a child will exhibit the full range of behavior that they think won't get them in trouble - in other words, correct. It is often useless to figure out where the child learned terrible words and behavior. It is better to try to correct that instead of trying to debug it. 

There are fundamental differences between algorithms and AI models. These differences are essential for programmers and developers and everyone in society. 

We will explore these differences and their implications next week. As the French statesman George Clemenceau said, "War is too important to be left to the generals.".

AI is helping solve the world's energy problems by helping with nuclear fusion calculations. It also solves medical problems by predicting how all the proteins fold and making advanced medical diagnoses available to all. AI applications fundamentally change how society, businesses, armies, and governments work.

We'll explore whether AI truly is a step-change function (like mobile phones) or is it a lot of snake oil driven by money sloshing around in turbulent times?