A Skeptic's Guide to How Cognitive Biases and Reasoning Errors Fuel AI Hype, Social Learning, The Hidden Cost of ChatGPT is the Erosion of the Digital Commons, and more.
Enterprises are failing to see enough value add to justify purchasing new generative AI systems like Microsoft CoPilot. Why? Adapting to AI is a complex change that requires different methods for evaluation and change management.
Underneath the excitement surrounding generative AI is a cautionary reaction: enterprises are failing to see enough value add to justify purchasing new systems like Microsoft CoPilot. For anyone who has found generative AI to be useful, this might come as a surprise. How can I find value as an individual, but my company doesn’t?
Gartner's recent research highlighted that while 63% of companies are piloting GitHub CoPilot, many struggle to quantify the tool's impact on developer productivity in tangible benefits. What could be the reason for this?
Our theory is that established valuation techniques—return on investment, cost-benefit analysis, total cost of ownership, value stream mapping—do not capture the value (and cost) of complex change. Implementing traditional software is a complicated endeavor: there are many interconnected dependencies that need to be designed and implemented effectively. Despite the challenges, though, complicated processes can be broken down, analyzed, and forecasted with a reasonable degree of accuracy. An organization can establish value by forecasting that if employees use a new system for certain tasks at forecastable costs, the enterprise will realize a predictable benefit.
Introducing generative AI, however, does not yield to traditional analytical tools. Unlike traditional software which performs predictably according to specifications, generative AI is designed to be unpredictable, and its effects emerge through interactions with humans in the system. Emergent benefits cannot be used in calculations like ROI, CBA, and TCO precisely because they are emergent and can't be predicted in advance. This may be why, in a recent analysis by a16z, 60% of companies implementing generative AI believe ROI is positive but aren't measuring it precisely or aren't even considering ROI at all.
How can you validate the adoption of generative AI with concrete economic metrics rather than relying on mere enthusiasm and fear of missing out?
We use complexity science to help inform how organizations can adapt to the complex change of generative AI. Complexity features help us understand the complex change that comes from introducing generative AI:
Emergence: The most important feature of complex systems, emergence results from agency and self-organization in the system. Generative AI is a general purpose technology, highly flexible, and is often used to spark human creativity and learning. How it is used emerges from the individual and collective interactions between people and machines. The advantages of generative AI rely on the inherent openness and unpredictability of its adoption paths.
Adaptation: People adapt to complexity by learning and evolving—and their reactions may be challenging to anticipate. In a generative AI-enabled organization, people will cooperate, communicate new ideas, and adapt their behavior based on the AI’s performance. Adoption strategies must account for all facets of adaptation, acknowledging that whether through deliberate planning or not, adaptation to generative AI is unfolding in real time.
Dynamic: Complex systems are always in flux and rapid changes can come from the edge of the system. Generative AI is different from traditional software in that it is capable of learning—from new data and from people’s behavior. This dynamism means that the system is not stable enough for traditional valuation methods. The challenge is to create complexity-sensitive valuation methods which recognize the unique context of each adoption situation.
Open: Complex systems have many external influences and interactions. Generative AI is mostly built on foundation models which can change (often rapidly, usually unpredictably) underneath the applications that enterprises are evaluating. If and how people use generative AI is also open, and it may not be possible to bound the expected uses for evaluation.
Self-organizing: Novel, higher-level properties (like value to an enterprise) arise from the interactions of lower components like people and AI systems. While this represents a distinct advantage of complex systems, it also means that introducing generative AI will not yield to top-down control or pre-defined instructions for use or non-use. Dictates and top-down mandates aren't effective. Ignoring covert adoption introduces complex risks and unknown unknowns.
Feedback: Complex systems are sensitive to feedback and an outcome can influence an input, positively or negatively, directly or indirectly. This means that how generative AI performs can create positive or negative feedback in a system, and change can be accelerated or suppressed. This feedback can also affect future costs, as inference costs can be based on usage. User behavior driven by the nature of feedback cycles is an under-appreciated aspect of value assessment.
Distributed control: Complex systems do not have a central governing body—no one has total information nor total control. Due to the distributed nature of generative AI tools like CoPilot, how, when, and why it is used cannot be controlled. The best an organization can do is design incentives, steer, cajole, and reward people for good skill development.
Path dependency: Perhaps most important to the current wave of generative AI evaluation is path dependency. Future decisions in complex systems depend on decisions that preceded them. Path dependency creates lock-in, which makes people nervous because poorly constructed solutions can close down future options. If an organization adopts generative AI, what might it not be able to do in the future? And, if an organization does not adopt generative AI, what might it miss out on?
Our long-time readers, audiences, workshop attendees, and strategy clients may have been wondering why we have mashed AI and complexity together over the past few years. An honest reflection would reveal that we didn’t have a clear explanation at the beginning—rather, we were following our intuitions. But, now, we can see the connection clearly: adapting to AI is a complex change and requires different methods for evaluation and change management.
Stick with us as we will be writing more about AI, complexity, and complex change management this year. And reach out: we’d love to hear your reactions and experiences.
This Week from Artificiality
AI and Complex Change. Across the Artificiality community, organizations are grappling with a new organizational dynamic characterized by complex change. Change management itself is changing due to the need to adapt to generative AI. Rather than being a linear, sequential process, with each step building upon the previous one to drive successful change, change is getting much more loopy to adjust to the emergence from our new human-machine systems. Complex change recognizes that change is non-linear, emergent, and deeply interconnected with the system in which it occurs. This is even more important as we adapt to the complexity of generative AI.
AI and Decision Factories. AI's potential is vast but many applications remain incremental, simply grafting AI onto outdated frameworks. This is why we embrace complexity. We think about designing incentives, understanding self-organization and distributed control, and planning for emergence and adaptation. Large Language Models have broken up the human cognitive chain: from information gathering to prediction, deliberation, judgment, and action. This decoupling matters. Cheaper information, cheaper predictions make judgment and action more valuable. The advent of agentic AI could integrate all these steps, potentially bypassing human involvement in certain decision chains. That is indeed the goal of agentic AI: automating human decisions so that humans get to take more actions. That is, do more with less. Adapt and evolve.
Why RAG Beats Fine-tuning AI. Enterprises face a critical choice in their generative AI adoption strategy: fine-tuning or Retrieval-Augmented Generation (RAG)? While fine-tuning has been the go-to approach for early adopters, a new study suggests that RAG may be the more powerful and sustainable path forward. The study highlights RAG's ability to dynamically retrieve and incorporate verified information, making it a more reliable and accurate approach for deploying generative AI at scale in enterprises, where AI is increasingly relied upon for high-stakes decisions and customer interactions.
Bits & Bytes from Elsewhere
Microsoftannounced new tools in Azure AI to "help you build more secure and trustworthy generative AI applications." At first glance, these seem quite helpful: tools to help block prompt injection attacks (like malicious content in an email that can create havoc when an LLM summarizes the email) and hallucination prevention (comparing responses with "grounded documents"). While useful, these tools make us wonder: is the next wave of enterprise generative AI development going to be focused on patching over the systems' inherent weaknesses? How much more problematic will these weaknesses be as companies provide generative AI systems with agentic capabilities?
Databricksannounced DBRX, an open source, mixture-of-experts model that is derived from the company's acquisition of Mosaic last year and trained using data processed, in part, by tools from another Databricks acquisition, Lilac AI. We find this announcement interesting because a) we're closely following the open vs closed source decisions of large enterprises (aka Databricks' customers), b) we're interested in the efficiencies of mixture-of-experts models, expecially inside the enterprise, and c) we're interested in generative AI business model evolution. On the last two points, Ali Ghodsi, CEO of Databricks said that the cost to train DBRX was around $10 million (significantly lower than other major models) and that "While foundation models like GPT-4 are great general-purpose tools, Databricks’ business is building custom models for each client that deeply understand their proprietary data" (a business model that works well on top of open source).
New researchaffirms our previous stance that generative AI detection tools do not work and are a very bad choice for schools and universities. Not only is the detection accuracy low (40%), the accuracy is dramatically reduced when a human changes the content that is generated by AI (17%). We work with multiple higher education institutions and feel their pain in adapting to generative AI. But, trying to prevent usage through detection tools will only falsely accuse students, damaging both the student and institution credibility.
Tom Davenport (who we interviewed on Artificiality) published an article in HBR about research sponsored by AWS. The premise of the article is that while enterprises are excited about generative AI, few have tackled new data strategies required to implement generative AI. The survey of chief data officers found that 93% of CDOs agreed that data strategy is crucial for getting value from generative AI but only 57% had made the necessary changes. One of the most striking parts of the report is a quote from Jeff McMillan, Analytics and Data Officer at Morgan Stanley Wealth Management:
“We have been curating our document-based knowledge for a while. Every single piece of research content has to be reviewed by a registered compliance person, so we know the training content is of very high quality. Even in non-research content, we have a team that scores individual submissions on issues like tagging requirements, broken links, presence of a summary up front, and we give each document a grade. We also had to spend a lot of time thinking about different content sets and optimizing the results. For me, the most important thing about these models is that they have to be transparent. The user should know this is what I put in, this is what came out, these are the documents the output came from, and this is confidence level for that question.”
Now imagine the effort your enterprise might need to take to process any unstructured content you'd like to train an AI on. At least financial institutinos like Morgan Stanley have existing compliance processes and teams—everyone else will have to start from scratch.
Where in the World are Helen & Dave?
Several upcoming events to highlight—you can see everything on our events page.
Artificiality Pro. State of AI and Complex Change Report: Q2 2024 Update. Join us on April 10th for our new, quarterly research update on AI and complex change. Reach out to learn more about joining Artificiality Pro as an individual or enterprise.
Charter Pro. Making Decisions with Generative AI. Join us on April 10th for a presentation with our friends from Charter about making decisions with generative AI. Sign up here.
ASU+GSV Summit. Join us on April 15th at the ASU+GSV Summit in San Diego. No presentations scheduled yet but we may do a pop-up or impromptu gathering depending for the Artificiality community.
Starbucks Innovation Expo. Join us on May 14-15 as we return to the Starbucks Innovation Expo in Seattle for the fourth time to talk about Generative AI & Data Culture.
Dave Edwards is a Co-Founder of Artificiality. He previously co-founded Intelligentsia.ai (acquired by Atlantic Media) and worked at Apple, CRV, Macromedia, Morgan Stanley, and Quartz.
