Complexity Management

Systems are an arrangement of parts or elements that together exhibit behavior or meaning that the individual constituents do not.^[1]

"In ordinary language, we often call something complex when we can't fully understand it."^[2]

That is to say, when it is:

• Uncertain
• Unpredictable
• Complicated
• Just plain difficult

A complex system then could be described as a set of interrelated and difficult to understand:

• Components
• Services
• Interfaces
• Interactions (including those with humans and the environment)

Considering that, how do we make systems that are inherently complex more understandable?

For an adequately large company, its associated systems and subsystems can be defined as a complex system. When those systems need to be maintained, repaired, upgraded, and/or updated, or enabling changes made to alter the way we interact with them, it may require:

• A total system view
  (usually a defined system context and model)
• Objectives and needs analyses
  (a teleological analysis of a system)
• Requirements analyses
  (including stakeholder, business, and system requirements definition)
• Configuration management process
  (usually involving DevOps, SecOps, git, design control, environment management [like dev/test and blue/green deployments] and test & verification processes)
• Interface Management process
  (usually involving implementing and documenting complex technologies with contextual ontological mapping, such as an Enterprise Serial Bus)
• RMA analyses
  (reliability, maintainability, and availability, i.e. how often does it break, who can fix it, and how easy and cost effective is it to fix)
• Observability processes and technology
  (often, implementing methodologies such as MELT - Metric, Events, Logs, and Traces)
• Producibility analyses
  (can it easily be produced and implemented elsewhere? Does a local supply chain support it? Does open source support it or a third-party company? Big example in information systems is Infrastructure as Code [IaC] - such as Terraform)
• Trade analyses
  (considering the predecessor, R&D, new innovations in the field, strategic advantage, and future valuations, which implementations and technologies are the best fit)
• Even more, such as FMEA (failure modes and effects analyses), risk analyses, modeling and simulation, cybersecurity analysis, data governance and enterprise architecture considerations, regulatory, etc…

Complex systems can:

• Be tricky to understand.
• Lead to emergent behaviors and unknown system dynamics.
• Lead to 'Butterfly Effects' with widespread changes when systems are replaced.

When looking at 'systems of systems' (SoS), emergent behaviors and effects can be tricky to identify and quantify ahead of time. It's useful to compare a proposed digital transformation against a 'total systems view' by practicing enterprise systems engineering and architecture.

Tackling Domain-Associated Bias

Also adding to the complexity of implementing proposed digital transformations is the bias introduced into the perspective of those from different backgrounds and fields that will inevitably be providing input into the project. Integrated, cross-functional development teams can provide a lot of useful perspectives, but often times translating between the various fields of knowledge or establishing common terms can be very challenging. Systems engineers are great candidates for ensuring that many different perspectives can be integrated, as they tend to have a very wide breadth and significant depth of knowledge across various domains.

Although traditional systems engineers (of the aerospace, vehicular, weapons, etc. systems variety) may not have a background in business, it is important for those systems engineers practicing enterprise systems engineering to have foundational to intermediate knowledge in business domains. this might mean education in accounting, finance, information technology and systems, management, human resources, marketing, and more.

Luckily, many industrial and systems engineering programs and professionals get curriculum that involves many of these topics, especially in the context of systems optimization. For example, both industrial engineers and cost accountants are concerned with cost optimization problems and implement models using methods like Theory of Constraints (ToC) and Activity-Based Costing (ABC). Additionally, most systems and industrial engineers are familiar with concepts such as value engineering, lean and agile process methodologies, common systems development lifecycles, and quality management systems.

To illustrate the importance of integrating domain perspectives, imagine an integrated development team for a building system, that involve architects, HVAC engineers, electrical engineers, civil engineers, construction planners, project managers, procurement professionals, marketing professionals, and more. Architects may have a primary perspective of design, utility, space, and abstraction. HVAC engineers may be concerned with thermal aspects of the building and interior spaces. Electrical engineers will think about the building in terms of the power draw of the technologies inside, and where wiring may need to be run. Civil engineers will be concerned with the soil under the building, the type of foundation, and the above-ground materials needed to construct the building envelope, considering the weight that must be supported. Planners and project managers will be concerned with timelines and quality related to service contracts, and procurement professionals will be worried about material lead times, local supply chains, and the overall producibility of the building depending on its location and the economic environment it's built in.

A professional from any individual domain is going to have a different primary concern for the system, that may not consider or even overlap with the perspectives of others. This is where the systems engineering profession shines and can help integrate knowledge from various implementation specialists into the plans, schematic models, and processes of the system from conception through design, implementation, operation, maintenance, and disposal without bias and while maintaining a total-system view. They'll also be able to maintain multiple viewpoints that capture the contextual ontologies maintained within each domain, with an additional layer of context of the objectives, needs, and goals of the whole system.

The above figure^[3] from Kossiakof et al.'s Systems Engineering Principles and Practice, 3rd Edition highlights this issue very well.

Gall's Law and Gall's Inverse

In 1975, John Gall, faculty of the University of Michigan, published his systems research into the book Systemantics: How Systems Work and Especially How They Fail. The most recent edition of this book, published in 2002, is titled The Systems Bible. Many subsequent scholars have referenced, quoted, and found influence from the rules and principles provided in this literature. This includes many who made great contributions to systems and software engineering research and field: Ken Orr^[4] (one of the creators of the Warnier/Orr diagram), Grady Booch^[5] (one of the creators of UML, the unified modeling language), and many more.

The rule presented in Systemantics: How Systems Work and Especially How They Fail that is famous in systems thinking is referred to colloquially as ‘Gall’s Law’, although it isn’t called this in the book. Galls Law refers to this excerpt from page 71:

“A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”^[6]

The first sentence refers to Gall’s Law, and the second and third refer to what is often called ‘Gall’s Inverse’. When we look at complexity in systems and read case studies related to failed systems and implementations, we will often see complexity and unknown variables as the cause for the failure. Gall’s law works because of what we stated about complexity before – it inherently comes with uncertainty, is unpredictable, is complicated, and is plainly just difficult to deal with. The best way to manage complexity is to avoid starting out with complex system designs. As you’ll see as we move forward, the best way to avoid complex system designs for enterprise information systems is to reduce the number of system elements, interfaces, and integrations we need to manage.

There are various other frameworks that make similar points which are worth learning about, but in specific relation to decision making and decision support systems rather than purely to systems design. One introduced to me by my colleague Andrew Stearns is the Cynefin framework. Created by Dave Snowden, this framework breaks down decision-making context into five domains – clear, complicated, complex, chaotic, and confusion.

It is my hypothesis that Gall’s Law is effective because it allows one to start from a place of understanding, and as it is extended it allows for incremental analysis of emergent behavior, adding an element of predictability and a chance to roll back changes to a working and well understood product with each forward moving iteration. Take for example, a medical study related to weight loss. If you had each participant change their diet, exercise, and take supplements – if an effect was found, how would you be able to attribute it easily to any of those three variables? It would be difficult. A good study would try to find a way to isolate and eventually provide for various permutations of the study variables to determine their individual effect and contribution, using the well-known effects of each as a basis for developing hypotheses for the next step. Gall’s law works in this same way – through simplicity, we can know and understand effects of individual variables, and then analyze emergent changes with the introduction of each new variable (system element, interface, etc.).

It's important to review some modern abstractions used in the literature for information systems. They can vary in size, type, purpose, and complexity, and often many act together to create distributed and complex systems of systems, and it can be hard to build good mental models of how these systems work together in the enterprise ecosystem. Building digital literacy is one of the first steps we should take towards demystifying and lowering the relative complexity and effectiveness of our digital transformation initiatives, which will require professionals of non-technical backgrounds to participate.

Some of the main types of systems to consider are (derived from Jeanne Ross's 2018 article Five Building Blocks of Digital Transformation^[1] and 2021 book Designed for Digital: How to Architect Your Business for Sustained Success^[2]):

• Operational Backbone Systems
• Digital Platforms
• Third-party Platforms
• Middleware and Integration Services
• Cybersecurity Platforms
• External Developer Platforms

Among these, these three (defined by J. Ross) are particularly important to note in modern information ecosystems:

Operational Backbone Systems

"A set of integrated and shared systems, processes, and data that ensure efficiency, reliability, and transparency of operations and transactions."^[1]

e.g. ERP & SCM, CRM, HRIS, PMIS, etc. These tend to hold an organization's Master/Transactional Data, usually in a third-normal form format within a relational database. They tend to be complex but highly reliable, stable, and available. But the data contained therein may be hard to extract value from.

Digital Platforms

"A repository of business, technology, and data components facilitating rapid innovation of new offerings and enhancements."^[1]

e.g. Business Intelligence and Analytics Platforms, Decision Support Systems, Productionized AI and Machine Learning Models, Knowledge Sharing systems, etc. These tend to hold data at varying levels of aggregation and granularity, blended from one or more source systems. They might be seeking to provide OKR and KPI information via dashboards, provide information or services to users by leveraging a model on top of the data (such as an LLM chat bot or AI agent), or more. These will include systems where we see innovation but also a lot of complexity in mapping of data that was captured in different contexts from systems with highly differing designs. They will force us to consider how to map those various ontologies to derive value and actionable insights. They tend to have more issues with reliability and availability due to all their upstream dependencies and their complex designs.

Third-party Platforms

Digital business and technology services and systems provided by other companies/firms.

e.g. Transaction processing systems, Data governance and catalog systems, Business Intelligence and Analytics systems, etc. (such as major cloud providers, snowflake, databricks, square, and other similar vendors). A lot of these firms are so-called 'modular producers', or ecosystem drivers providing entire platforms.

Complexity in Information Systems

Complexity, as it relates to information systems, is highly correlated to several factors. Some important ones are:

• Number of interfaces
• Complexity of interactions over those interfaces
• Inherent complexity of the system components/elements

An interface as used here describes where and how systems integrate (kind of like a service contract, that certain interactions will be provided at internal system element boundaries and the external system boundary). A classic representation of an interface is a ball and socket, or a trailer hitch and ball. This is a simple but good example of an interface. In the trailer example, the 'fit' at the system boundary between the vehicle hitch and trailer ball mount allows for the transfer of driving force between the system elements.

An interaction here can be thought of as an action or verb describing what happens at an interface. An interaction could be transfer of force, electricity, heat, fluids, air, data/information, etc.

e.g. an interface could be an API developed for SAP, the intent of which is to provide data over a network using HTTP. The interactions occurring over that interface would be HTTP request and method (GET, POST, PUT, DELETE). Maybe, the request is a GET request used to source the data for a PowerBI report.

When it comes to external interfaces, complexity usually centers around integrating systems that operate at different levels of abstraction and granularity, and which use different taxonomies and ontology. How an application chooses to serialize and marshal its data is very important, since clients may need special implementations to be able to deserialize the data and might benefit from caching to reduce processing time for the marshalling process. For marshalling to occur effectively, some semantic mapping may be required to bridge levels of abstraction in the data between systems, and to logically join the contextual ontologies the systems are representing for various business components of the larger enterprise.

Systems don't just have external interfaces; they usually have many internal ones. Especially systems designed using an object-oriented approach. Internal interface complexity is highly correlated to coupling (level of runtime object dependency) and cohesion (runtime object reliance on internal functions) of runtime objects. This is also apparent in systems that utilize a "microservice" architecture - although the infrastructure attempts to decouple services, the lightweight interfaces that connect those services may be very complex and hard to maintain and manage if the underlying logic lends itself to tightly coupled, low-cohesion logic implemented across those microservices. Essentially, good design patterns, high modularity, and decoupling of services from their dependencies by using good interface design, implementation, and management can greatly reduce information system complexity.

An example of good decoupling technique is by using interface-based programming to provide interfaces as stand-in types for parameterized constructor injection, such that runtime object dependencies injected can be of any type, so long as they provide the expected interactions defined in the interface. This increases maintainability, creates ease when updates are needed, and can help to completely decouple runtime objects from their dependencies.

Generally, the use of more software systems results in more components in the whole Enterprise Information system of systems, meaning more:

• Interfaces, that likely have their own special documentation and request formats
• Interactions to manage and secure between system components
• More inherent complexity in system implementations across the Enterprise

The above figure^[3] from Ross, Weill, & Robertson's Enterprise Architecture As Strategy: Creating a Foundation for Business Execution shows tradeoffs with flexibility taken to move towards a more mature architecture.

That is why it makes sense, when possible, to design services and systems to be modular, and reduce the total number of system components utilized where possible. There may be strategic exceptions to this which might create a positive tradeoff between complexity and modularity (such as using a microservice architecture) - but because many enterprises have a low enterprise architecture maturity, simplification and reduction of the overall number of components is often a much more effective strategy. The process of reducing the number of localized or siloed systems and applications used in an organization is referred to as the rationalization of systems - and it is directly and intimately tied to the architectural maturity of an organization. As organizations become more mature and rationalize more of their systems, they can start to realize additional benefit by working to develop expertise in modularizing and decoupling those systems, and then learning to build expertise in leveraging and exploiting APIs (using the 'new thin enterprise architecture').

The above figure^[4] from Ross's 2006 working paper Enterprise Architecture: Driving Business Benefits from IT shows the four defiend enterprise architecture maturity stages. Note the overlap of modular architectures with a high architecture maturity, and siloed applications with a low maturity as well as the steps bridging them, and consider Gall's Law. Seeing the IT budget values - consider how strategically effective firms acheive operational efficiency before investing in more complex architectures.

Trying to leverage concepts such as bespoke services, modular microservice architectures, and the creation of bespoke services underlying flexible user interfaces can be much more successful when you are working from a simple base – this will help you effectively manage emergent behaviors across the enterprise. Remember - Gall's law states that a project should start simple and work towards complexity only very gradually, using proven frameworks and processes. The inverse of Gall's law also applies here: A complex system designed from scratch is not likely to work and cannot be made to work. This directly correlates with research performed by the MIT CISR.

Because of the way various kinds of enterprise systems depend on one another, it makes sense to focus effort on rationalizing and strengthening your organization’s operational backbone systems. The master data in those systems is ultimately fed to many other systems and acts as the informational basis for functionality and decision making from your digital platforms. From a strong operational backbone, it becomes much more manageable to develop and extend your digital platforms, and the provenance of the information is more clear.

That said, the first and maybe one of the most important guardrails we want to use to manage complexity during concurrent digital transformation projects and to help us plan future ones is to ensure that they don't result in a larger number of localized, siloed applications. Instead, digital transformation projects should focus on rationalizing systems and reducing the overall complexity of systems to allow them to scale more effectively, and to help to manage and focus on possible emergent behaviors across the enterprise information systems. Not every organization will need to exclusively focus on rationalizing systems - not every organization will be at the same place on their transformation journey.

Andy's Advice:

Seek to find solutions that rationalize, or at least standardize, systems in line with a strategic technology roadmap. Use Enterprise Architecture and Business Architecture guidance as guardrails. Try to avoid customization of systems. Organizations with a high enterprise architecture maturity should focus on building integration expertise, modularizing services, and leveraging/exploiting APIs.

It's important not to shoot from the hip when investing a non-trivial portion of your CapEx and OpEx into digital transformation initiatives. A traditional way to decide on which projects to take on is related to Demand Management, where business units and practices make requests to IT, and an IT portfolio board or management group review and select which initiatives to take on based on demand. But this is an inherently flawed approach when strategic business objectives are not considered (enterprise architecture rationalization, strategic positioning (such as five forces), revenue increase, expense reduction, etc.).

There is a bias from IT that business units and practices will understand the value (cost spent for the functionality you get) of the transformation initiatives. But often, business units and practices have no insight into the actual overhead, costs, complexity, or work required to stand up certain solutions, and may not understand the risk. It also often ignores an important moment to implement value engineering and lean practices. Demand Shaping^[1], a concept presented in Jeanne Ross's 2014 article Demand Shaping: The IT Unit's New Passion, will help business units to fully quantify solution value by understanding exact costs and what functionality they are getting from those costs. This will allow them to predict the functions' effects on the activities that drive their costs and help to generate revenue. It also opens an opportunity for a cross-functional business-IT team to work together to implement solutions that result in whole business process re-engineering, maximizing the value the business can get from technology enablement by receiving consultation and advisory from an IT function that is involved in the strategy of the business.

The above figure, adapted from Ross's 2014 working paper Demand Shaping: The IT Unit's New Passion^[1] shows the six components Demand Shaping as presented in the article. Descriptions are provided below.

IT Cost Transparency

• Exposing costs of IT services
• IT leaders help business unit leaders understand financial impact
• Requires explicit links between IT costs and their cost drivers

IT can enhance this process by partnering with cost accountants and industrial engineers to create activity-based cost models for IT activities, which will help businesses understand better what drives IT costs other than the exposed direct costs before they get a chargeback. Having this transparency will allow any business practice looking into a solution to more easily make a case for it and assess its feasibility against the costs and revenues associated with their existing process. Early feasibility assessments save a lot of overhead time and risk on conceptualizing and implementing solutions that weren't feasible from the get-go due to their associated costs, only discovered when the chargeback shows up. If you always make feasibility assessments a first step, you can eliminate a lot of wasted time and effort.

Strategic Program Management

• Identify finite set of strategic goals
• Assign program leader to manage all projects for goal

Include Enterprise Systems Engineering and Enterprise Architecture practices alongside program management to help handle system requirements, complexity, and verification of high-risk systems.

Strategic Technology Roadmapping

• Helps leaders to address the biggest gaps
• Provides a sequence to implement infrastructure services & business capabilities
• Planning means services can generate value immediately as implemented

IT can enhance this process by working with cost accountants, industrial engineers, and/or finance to find an optimized sequence of investments, compared against a company's forecasted working capital by looking at the future value of predicted revenues generated, and costs reduced for a particular initial investment/cost, among all the proposed digital transformation projects. For transformations to immediately start generating value, bottlenecks in targeted processes of the value chain should be targeted first, using a Theory of Constraints trade analysis. Then, among those highest revenue-generating options, Activity-Based Costing models may be prepared to find solutions with proposed processes that result in the highest expense reduction. Calculating the future values, initial investments, risk, and projected net profit among solutions using this hybrid approach will result in a robust trade analysis that can be altered according to changing risk appetite and working capital. Once prepared, this model can serve as a strong source of direction in choosing which transformation initiatives to undertake. Caution should be taken to leverage enterprise architects’ experience and note which solutions in a trade analysis can be utilized across many transformation initiatives with minimal or no customization, which may not be apparent for trade analyses focusing on individual projects. This model can be paired with less quantitative frameworks, such as five forces or the BCG growth-share matrix to accurately assess market position and integrate market knowledge into the transformation strategy. Further, a dedicated business architecture practice can help greatly with process modelling to create process models and quantify current and proposed process value.

Agile Development Methodologies

• Engage business users in design work with fast feedback cycles
• System users and sponsors respect roadmap and impacts of changes

Balance Agile development with adequate requirements gathering and modeling. Instead of loose stories and epics, develop full use cases, storyboards, objective scenarios, and detail out needs, goals, and feasibility to find opportunities for automation. Filter user requests and demands against a value engineering check and create lean processes to ensure work isn't being done that works against the rationalization of systems, causes complex or excess systems customization, and that implements guardrails of enterprise and business architecture guidance. Ensure that agile development still considers important project development considerations, such as RMA analyses, producibility, regulatory and governance requirements, cybersecurity, etc. at the product and portfolio levels.

Post-Implementation Reviews

• Expose what does and does not lead to business value

"Repeated MIT CISR studies suggest that nothing accelerates organizational learning about the value of IT as much as post-implementation reviews"

Integrate post-implementation reviews with an institutionalized knowledge management process. You can do this by codifying lessons learned and by encouraging sharing of project success, setbacks, and failures. Be open to creating case studies from these lessons that provide valuable stories, context, and learnings that can be shared across the organization.

Business Relationship Managers

• Start conversations about what works and what doesn't
• Articulate needed capabilities and help design roadmaps through design thinking processes

BRM interactions with leadership will lose efficacy if leaders have a low digital literacy and understanding of the value of IT enablement in their organization. BRMs should also have an adequate understanding of business and enterprise architecture guidance for the organization and should be able to set expectations about feasibility based on corporate guidance. BRMs are uniquely poised to ensure that leaders in various business practices understand why it's important to build enterprise-wide capabilities instead of focusing on custom tools and localized, siloed applications.

The above figure^[2] from Ross's 2006 working paper Enterprise Architecture: Driving Business Benefits from IT shows the four defiend enterprise architecture maturity stages and associated evolving management practices. Note the alignment of enterprise architecture maturity stages with use of demand shaping principles by management.

Andy's Advice:

In addition to Demand Shaping, implement a Design Control process to manage any complex transformations in the program portfolio, which requires Enterprise Architecture and Enterprise Systems Engineering review integrated into the strategic decision-making process and the project's configuration management process once started.

Talent Management

Management of transformation initiatives means foraying into often unexplored areas and technology. This creates a need to develop an appropriate skillset to be able to perform design and implementation of proposed transformations. Talent should be trained or acquired strategically based on the KSAs (Knowledge, Skills, Abilities) needed to implement the planned solutions laid out in your multi-year technology transformation roadmap to minimize fit gaps in planned work. There are various roles it's important to understand when it comes to digital transformation strategy and processes. Here are some roles and associated titles - which is incomplete, but useful for building a mental model:

Strategically vision, optimize, architect, and verify the value of IT:

• Enterprise Architects

Ensure strategic business objectives are met:

• Business Architects
• Business Relationship Managers
• Digital Transformation Specialists
• Solution Consultants
• ERP Functional Consultants

Design and verify complex and custom systems:

• Solution Architects
• Enterprise Systems Engineers

Implement systems:

• Software Developers
• Business Analysts
• ERP Implementation/Technical Consultants

A focus should be put on mapping KSAs and availability of existing talent to determine where 'fit gaps' exist. Considerations such as promotion schedules and turnover should be taken into consideration, to assess the state of the talent pool when proposed future transformation projects are started. Additionally, when solution implementation cost drivers include labor hours and a lot of highly complex or specialty work is to be completed, proposed salary, contract, and/or consulting costs should be considered and related to technology selection during trade analyses and as a part of feasibility analyses. Many failed projects are a result of unmatched expectations on the expertise and knowledge needed to implement a solution, and how common or expensive it might be to enlist those with the KSAs and capabilities to implement solutions. This can be one of largest driving factors in the overall feasibility of the solution and shouldn't be taken lightly. Additionally, validation and verification processes can help to ensure that talent has the capabilities, such as prior project experience, portfolio projects for employees, and more. To help over-reliance on expert knowledge, knowledge substitution, mandatory team learnings/trainings, and development of best practices and guidelines by experts to codify their knowledge and processes can be used as a strategy to help augment KSA fit gaps in a larger project team.

Fit gaps in KSAs may also be due to cultural and human resource issues. A lack of psychological safety and good working atmospheres combined with complex work can result in unexpected turnover on high-cost and high-risk initiatives. It may also make it harder to accurately plan the costs of initiatives, as expert salaries in the tech space make up a significant portion of project costs.

Change Management

Organizational change management will likely be required to minimize workforce and culture impact of a constantly evolving technology ecosystem. Although many technologies underlying a lot of the new products and services on the market are the same, new names and abstractions change, and specific brands bring their own abstractions to the market, making technology look confusing to an outsider (for example, the word 'cloud' just refers to the rental of data center equipment as a service instead of buying and maintaining your own private data center assets - but to an outsider may sound like a newer technology). There are many frameworks for change management in an organization. A few include Kotter 8-Steps Framework, the Lewin unfreeze-change-refreeze framework, The McKinsey 7-S Framework, the Cooperrider Appreciative Inquiry Framework, and more. A lot of time can be put into analyzing the different change management frameworks. Often industrial-organizational psychologists and HR professionals, either as employees or contracted consultants, are tasked with determining the best ways to approach change based on a company's culture and environment and are dubbed 'change management consultants'. Change management is an important topic to become familiar with as a leader, as it is cruicial to our ability to ensure the success of transformation projects, including digital transformation projects.

While studying organizational transformation in graduate school, I was introduced to a great book by my instructor Kevin Smith, called Switch by Chip and Dan Heath. It provides three general principles to create change, by providing a metaphor for change involving riding an elephant. In this metaphor, the rider on the elephant is the logical part of one's mind and the elephant is the emotional part of one's mind. General principles to create change, according to Switch:

• Direct the rider
  • Find what works well locally, and do more of it
  • Script critical moves
    Reduce ambiguity by providing clarity in the change & avoid decision paralysis
  • Point to the destination
    Clearly and concisely state goals and where you want the change to go
• Motivate the elephant
  • Find the feeling
    Identify how people feel about the change
  • Shrink the change
    Break up big changes into small, achievable steps
  • Grow your people
    Improve people's skills to build confidence in undergoing change
• Shape the path
  • Tweak the environment
    Create an environment where the change can be successful - reflect on the Fundamental Attribution Error
  • Build habits
    Turn the change into habits, similar to James Clear's Atomic Habits
  • Rally the herd
    Since folks tend to follow the lead of others, try to rally the majority towards the change

I think in terms of digital transformation, all of the above apply, and also correlate strongly with topics we've already covered. Here, I provide some Digital Transformation-related examples for each item:

• Direct the rider
  • Find what works well locally, and do more of it
    Design patterns and frameworks, reusable and modular code, human factors analyses, design thinking & ethnographic research
  • Script critical moves
    Strategic Program management, Gall's Law (complex systems = decision paralysis and ambiguity), requirements gathering, modeling (schematic like UML and mathematic like cost and parametric) - involve stakeholders
  • Point to the destination
    Objectives, goals, and needs analysis - involve stakeholders
• Motivate the elephant
  • Find the feeling
    Human factor analyses, design thinking - connect with end-users and build excitement - what do the end users connect with the most?
  • Shrink the change
    Gall's Law (change seems less daunting in simple systems), agile development methodologies, lean methodologies, sprints, gradual feature-based transitions - involve end-users
  • Grow your people
    Expert knowledge substitution and augmentation, training, KSA analyses and planning
• Shape the path
  • Tweak the environment
    Corporate culture, psychological safety
  • Build habits
    Implement best practices, such as EA guidance and demand shaping
  • Rally the herd
    Have leaders and respected experts evangelize best practices

Andy's Advice:

Engage in some form of enterprise-wide KSA management and planning to align your people with your technology roadmap and IT project portfolio for enterprise-scale business capabilities. Use change management frameworks to ease the effects of change on all stakeholders.

Active management of knowledge sharing processes is not only essential to realizing value from digital transformations but can be a strategy for helping to decomplexify them. Actively managing knowledge in an organization has many benefits. It can help to highlight areas where innovative techniques are locally producing great results, and areas and systems that have a need for change. Knowledge Management institutionalizes the sharing and codification of knowledge, and thus comes with methodologies and processes for knowledge capture. It allows for greater visibility of otherwise siloed projects, processes, methodologies, and techniques across the organization where they may have otherwise been invisible - allowing for their review and use to develop standards and best practices. Standards and best practices developed by the organization can then be used across the organization to provide direction and align people, systems, and processes to build enterprise-wide capabilities.

There are two general ways in which knowledge is managed, as defined by Hansen, Nohria, and Tierney in the 1999 HBR article What’s Your Strategy for Managing Knowledge. The first is called codification and refers to people-to-documents translation of knowledge, where people record their knowledge using written natural langauge or models such that it may be stored in some KMS (Knowledge Management System) and later interpreted by others. The second is called personalization and refers to people-to-people processes where knowledge is shared by word of mouth or demonstration.

A big consideration in knowledge management is how to facilitate and improve knowledge sharing, and doing so in such a way that it can lead to improved codification or personalization of knowledge. There are several strategies for this.

Fundamental Theory of the Firm

Modern knowledge management strategies can be tied largely to "The fundamental theory of the firm", a concept referring to a classic (1977) article from Administrative Science Quarterly on how firms can be organized called Markets, Bureaucracies, and Clans by William Ouchi, referring to mechanisms of mediation or control over value exchange in a firm when competing interests and boundaries exist. This fundamental theory of the firm looks at levels of goal incongruence (differing goals across the firm leading to differing approach) and performance ambiguity (how well individual performance standards are defined given a technologically advanced or closely integrated industry where teamwork is common, and technologies change rapidly) to determine which organizational structures are the most efficient. Markets (organizations driven largely by reciprocity and prices) are said to be most efficient when goal incongruence is low and performance ambiguity is low. Bureaucracies (highly institutionalized and standardized, hierarchical and rule-driven organizations marked by legitimate authority) are said to be most efficient when goal incongruence and performance ambiguity are both moderately high. Clans (flat organizations defined by common values, beliefs, and traditions where goals are well-aligned) are said to be most efficient when goal incongruence is low and performance ambiguity is high.

In this framework, Market organization fails because of uncertainty, bounded rationality, and opportunism related to value exchange. Rules imposed by Bureaucratic organization helps to 'solve' many of these concerns by imposing rules, which puts a check on opportunism and helps to reduce ambiguity by providing expectations in the organization that can be relied on. When Market organization is too volatile and Bureaucratic organization not effective due to rapidly changing standards and requirements (such as in big technology), Clan organization can be most effective where goals, culture, views, and values have a high level of overlap and cohesion in the organization, and where teamwork is already common, creating a good environment for more 'flat' and less 'hierarchical' approaches.

Three Knowledge Management Strategies

In most modern firms, you may see different and differing levels of these organizational structure approaches implemented to organize different areas, which may be more or less efficient given goal congruence and how well performance metrics are able to be defined. The Three Knowledge Management Strategies paper pulls from this fundamental theory to determine knowledge management approaches that follow a similar structure - but refer to clans as 'communities' and bureaucracies as 'hierarchies'. Instead of value being exchanged in the framework, knowledge is. Below are descriptions of each strategy. While reading, it is important to think on the 5 characteristics of data quality (Accuracy, Completeness, Reliability, Relevance, and Timeliness) and how the knowledge management strategy may affect the measure of each.

One of my favorite papers is from MIS Quarterly Executive and was introduced to me by my mentor Dr. Bill McHenry during my graduate studies. The 2005 paper is called Three Knowledge Management Strategies: Knowledge Hierarchies, Knowledge Markets, and Knowledge Communities by Alan Dennis from Indiana University Bloomington and Iris Vessey from the University of Queensland.

In the 'Three Knowledge Management Strategies' framework, a knowledge market is a knowledge management strategy where knowledge is created as needed, tends not to be validated, is low quality with a high variance, and tends to have a long lifespan which creates a question of relevance and correctness. Like a market in the fundamental theory of the firm, knowledge markets tend to be unregulated, individual-driven, and opportunistic.

A knowledge hierarchy is a knowledge management strategy where knowledge is seen as a formal resource, is created with intent to maximize its value, undergoes extensive validation and aggregation by experts, is comprehensive, has a high quality, and tends to have a short lifespan (due to constant evaluation of data quality). Like a bureaucracy in the fundamental theory of the firm, knowledge hierarchies are highly institutionalized and rule-based, driven by authority and with the organization directly exerting control over the process.

A knowledge community is a knowledge management strategy where knowledge is seen and exchanged like a communal resource but isn't governed by corporate process or authority, is created either with intent or as needed, may be provided and undergo validation or not, may not be comprehensive, has an average quality, and has a medium lifespan due to its likely timeliness and relevance but lack of authority-driven urgency to be quickly updated. Like a clan in the fundamental theory of the firm, knowledge communities are highly driven by shared cultural values and tradition. In other literature and research, these communities are often referred to as Communities of Practice and are known to be informal groupings of those who wish to create, spread, and share knowledge on niche domain topics of shared interest. In these communities, there tends to be people from across organizations, practices, verticals, departments, and teams - coming together just to share knowledge.

According to the article, knowledge hierarchies are best suited for environments where knowledge changes quickly. This means that seeing knowledge as a formalized resource and creating institutionalized processes around it is a great way to prepare yourself for managing complexity in your oncoming digital transformations, as the industry and markets constantly evolve.

Knowledge Management and Complexity

Given some background on knowledge management and how it is employed strategically within an organization, it's valuable to understand how it helps to decrease complexity. When formal, institutionalized knowledge management strategies are employed by an organization and knowledge seen as a resource and actively leveraged via substitution, that knowledge can also be used to create standards that align with enterprise architecture maturity goals and your strategic technology roadmap. Technology advocates, evangelists, specialists, and experts can refer to overarching enterprise-wide strategies and help to drive otherwise informal knowledge sharing (via knowledge communities) and also knowledge markets towards the technologies, platforms, processes, and techniques that will help to build enterprise-wide capabilities. These experts can also help to increase the value of knowledge markets and communities by providing validation and verification of concepts and information provided, and even take on the onus of funneling information that may be used to augment the company's own best practices into an institutionalized process whereby it is turned into formalized and verified knowledge.

Because the heirarchical knowledge management strategy (like bureaucracies from the Fundamental Theory of the Firm) may present a hard adjustment to knowledge capture due to rapidly changing information and performance metrics, knowledge communities and markets may become key sources of information for the company's best practices. As mentioned above in the HR Change Management sections, change is harder when ambiguity is present. Employing knowledge management in your organization in this way will allow you to accelerate digital transformation projects and help those tasked with implementing them and overseeing them to act without worrying about ambiguity on what a best practice or approach for implementation may be. As multiple transformations are occurring, they can all refer to the company's formalized and verified knowledge to help reduce feedback loops that occur in configuration management and design control processes, reduce customizations, learn of techniques and reusable work from elsewhere in the organization that were effective, and implement solutions that will align with the organization's strategic technology roadmap while working to build enterprise-wide capabilities and rationalize systems.

Knowledge Management and GenAI-Driven Transformation

There is an additional and very large benefit to codification of verified knowledge in hierarchical knowledge management strategies: a solid basis for GenAI quality. One of the main use cases that has made GenAI so popular in the last 2 years is its ability to perform knowledge substitution and augmentation for specific questions and asks without the need to leverage an expert's time directly. The main drawbacks and dangers of GenAI are hallucinations (essentially, presenting incorrect information) and lack of important context (like an organization's strategic technology roadmap or enterprise architecture strategy). Institutionalized knowledge management strategies directly tackle this by creating a formalized process by which knowledge is verified and validated by experts and has a purposefully short lifespan to account for data quality.

Andy's Advice:

Seek ways that natural language processing (NLP) and newer generative AI (GenAI) techniques might be used both to facilitate and to perform Knowledge Management (especially knowledge substitution strategies), practice guardrail engineering, perform creativity tuning, and leverage techniques such as Retrieval Augmented Generation to mitigate negative effects of inevitable hallucinations. Make sure that expert knowledge substitution performed by use of LLMs is contextualized with corporate ontologies, best practices, enterprise architecture, and corporate strategy so all provided advice is aligned with the direction and goals of the organization. Most of all, don’t put the cart before the horse and try to develop a GenAI platform without developing an institutionalized Knowledge Management codification strategy and building out a knowledge management system. Otherwise, it will greatly reduce the quality, value, and effectiveness of all your future GenAI strategy.

Understanding the characteristics of ‘systems of systems’ (SoS) can help us to see the challenges inherent to working with constantly evolving and dynamic enterprise systems.

Here are some points to consider when it comes to managing concurrent transformations, that affect many systems:

• Complexity increases significantly as systems and interfaces change concurrently, and as new systems and interfaces are introduced concurrently due to potentially unpredictable emergent behavior, anomalies, or anti-patterns forming (remember Gall’s Law). They will be harder to predict, detect, and understand.
• Leaders planning for changes to one system may not know which interfaces or interactions with other systems may still be compatible or relevant if ongoing initiatives, processes, and enterprise architecture aren’t well documented and codified (make available and transparent a strategic technology roadmap, enterprise architecture guidance, IT cost transparency, and maintain strong knowledge management strategies and systems).
• Planning around compatibility and effectiveness becomes complex.
• Stakeholders may exist at various experience levels, and have varying levels of digital literacy and/or understanding of component systems and processes.
• ‘Tug and push’ dynamics may be at play between initiatives due to competing interests of stakeholders. (Remember the importance of rallying the herd and in leveraging BRMs to align leaders with strategy).

When we start looking at systems from the enterprise-wide perspective and realize the complex nesting nature of systems, the value of rationalizing systems becomes very clear. As you perform your many digital transformation initiatives, if you are constantly adding new systems or localized siloed applications, you are creating more downstream complexity related to integration, and weakening the quality of opportunities related to enterprise-wide capabilities.

Imagine that each of these systems have their own individual ontologies related to how data is stored, and their own methods for requesting and providing services. Connecting two systems may result in complex work needed to create unifying ontologies for mapping data semantically across these systems. Systems could end up creating dependencies on one another that can result in additional inefficiencies, such as temporal, topological, message-based, or order-based dependencies. Managing and planning for more of these interaction characteristics across what are possibly incompatible interfaces makes it much more complicated to analyze factors we described at the start of this article, such as trade analyses, RMA analyses, producibility analyses, FMEA, and observability processes. It may also result in lower quality data from an enterprise-wide perspective by affecting accuracy, completeness, reliability, relevance, and timeliness. Forcing systems designed at very different levels of abstractions to communicate and work with one another may lower the reuse of the data contained therein for other systems and may result in complex middleware or data pipelines.

Setting expectations upfront and showing your leadership a goal of creating unifying and rationalized systems will help to reduce tug-and-pull dynamics from competing views and interests – as you mobilize your organization towards a common goal via change management practices. At the enterprise level, complexity is unavoidable, but by applying several rules of thumb, we can effectively manage it.

Ackoff’s Art and Science of Mess Management and Competing Management Interests

In 1981, Russell Ackoff (Professor Emeritus of Management Science at the University of Pennsylvania’s Wharton School of Business, architect, city planner, trailblazer in systems theory, best-selling author, distinguished professor, and head of his own management education and consulting firm) published an article titled The Art and Science of Mess Management. This article presents a framework for approaching problems, that align very closely with how individuals may react to the various pressures applied to them in social systems. Ackoff defines a problem:

“…a situation that satisfies three conditions: First, a decision-making individual or group has several courses of action available; second, the choice made can have a significant effect; and third, the decision maker has some doubt as to which alternative should be selected.”

Ackoff presented three approaches for problems: resolving, solving, and dissolving. Each of these approaches presents a different view of how to manage problems and deal with the pressure they cause. Resolving is to select a course of action that is good enough, or that satisfies and suffices. It is often based on personal experience and judgement and is referred to by Ackoff as the ‘Clinical Approach’. Solving is to select a course of action that will yield the best possible outcome via optimization. It is based on research, and quantitative scientific methods, using mathematical and schematic models and simulations – and is therefore referred to by Ackoff as the ‘Research Approach’. Dissolving is to change the nature and/or environment of the system and associated entities related to the problem, in such a way that removes the problem. It is based on design patterns, systems thinking and engineering, organization, and process re-engineering, and is referred to by Ackoff as the ‘Design Approach’.

Notably, problem resolving tends to take place by managers, who defend their position by citing a lack of resources, including information and time, and that real problems are too complicated and messy to use alternate approaches. Problem solving tends to take place by managers seeking to grow and thrive instead of surviving, researchers, and consultants, who seek to find optimal solutions to problems. Problem dissolving is rarer and tends to take place by a subset of managers, researchers, and consultants whose focus is on development rather than growth. Ackoff distinguishes development from growth in stating that development is “To increase one’s ability and desire to improve everyone’s quality of life”. In Ackoff’s 2004 paper Transforming the Systems Movement, presented at the Third International Conference on Systems Thinking in Management, Ackoff provides more context on the difference between growth and development:

“Growth is an increase in size or number. Development is an increase in competence, the ability to satisfy one’s needs and desires and those of others. Growth is a matter of earning; development is a matter of learning. Standard of living is an index of national growth; quality of life is an index of its development. Development is not a matter of how much one has but how much one can do with whatever one has.”

What does this have to do with competing interests when it comes to concurrent digital transformation projects? Let’s consider an adequately large enterprise with an IT steering committee, performing demand management as opposed to demand shaping to select projects to take on in their portfolio. Managers across the enterprise will tend to lean on problem resolving approaches and may tend to propose solutions they think will work for their team, likely with some amount of bias towards their approach (remember domain-driven bias, introduced earlier?). If an IT steering committee takes on these transformation initiatives, they likely will result in the creation of more localized, siloed applications and solutions whose teleology (purpose, goals, function for existence) aren’t synergistic, and will likely not integrate easily or well with outside solutions or systems and will likely not be reusable due to the introduction of domain bias in its ontologies. Enterprise Architecture maturity will lower, and the IT architecture of the overall enterprise will increase in complexity due to the increase of system elements and interfaces. An optimal solution isn’t presented, and while the siloed applications may resolve a problem temporarily, some problems are likely to arise again later – possibly, problems related to unexpected and negative emergent behaviors related to their participation in the enterprise that weren’t thought out during a feasibility analysis of the solution. This fulfills the manager’s interests by temporarily resolving the pressures faced by their team but is likely to result in more upstream and downstream issues for IT, Enterprise Architecture, and leadership when the company grows or processes change, and the siloed application becomes an obstacle as opposed to a boon.

Now, lets consider a large enterprise where there is an IT steering committee, they are performing demand shaping as opposed to demand management, and the organization has hired third-party consultants to help implement the solutions using industry-wide best practices, but enterprise architecture has no control over which systems are implemented, and implemented systems are largely modernization efforts focusing on growth over development. The IT steering committee will likely employ business relationship managers, maybe project kickoff committees, and domain managers and leadership may have higher digital literacy. The consultants and IT steering committee will likely lean on problem solving approaches and will likely attempt to locally optimize systems but may still be heavily influenced by domain bias for the verticals, teams, or management they are working with. Enterprise Architecture guiding principles may be employed, but enterprise architects have no control over the design of the solutions provided by the consultants, so solutions may not be working together in the enterprise system of systems to build enterprise-wide capabilities and solve problems that may be caused by emergence – they solve systems locally but may create more issues when considering the enterprise as a whole. Growth may be achieved, but the problems they are tackling still exist, just abated by the solutions.

Finally, let’s consider a large enterprise where there is an IT steering committee practicing demand shaping, and they also employ and include enterprise architects and enterprise systems engineers in the selection process and strategy alongside management. Solutions are selected and created based on a strategic technology roadmap developed to rationalize systems in the enterprise architecture, and enterprise architects and enterprise systems engineers participate in the solution design and implementation, performing design control as a guardrail to the enterprise’s interest, to ensure that solutions built will help to move forward capabilities at the enterprise level. The enterprise architects ensure the IT strategy, and enterprise systems engineers help to ensure a unifying ontology and enterprise-accepted semantics are used in system development, eliminating domain bias in the system and improving the ease of systems integration across the enterprise. The integrated team of strategic program management, enterprise architecture, enterprise systems engineers & industrial engineers, and functional stakeholders (domain management) all work together to ensure the system will drive the enterprise forward, and if architecture is mature enough, maybe even reused for use cases many teams have for similar solutions across the enterprise. This would be problem dissolving, since the selection of each digital transformation initiative would be undertaken to affect the enterprise architecture design in such a way that removes problems, likely across the enterprise, including problems caused by emergent behaviors in complex systems. They would be selected because they would develop the enterprise and increase the quality of life for everyone involved as opposed to growing individual functions within silos. Domain management may have conflicting interests with this process, since it may not focus on their specific team and domain's semantics but rather a unified view of the enterprise - and may require additional learning and training on their teams, adding some additional pressure.

By teaching domain management that focusing on enterprise-wide capabilities will remove their teams’ problems if implemented correctly and create benefit elsewhere throughout the organization, managers may become excited about being able to be the epicenter of transformation for the organization, helping to reduce this conflict and the tug-and-pull dynamics. Increasing their digital literacy will help them to understand that while the solution may not be bespoken to their team or domain, that it can fulfill the same use cases and result in more positive for the whole as opposed to localized optimization and growth.

Work to invest in strengthening, rationalizing, and simplifying core systems across the enterprise to make the master data of your operational backbone systems cleaner, more understandable, simpler to aggregate and analyze on your digital platforms, and to create stronger and simpler decision support systems and business applications. This includes not only simplifying and reducing system elements and interfaces, but considering the semantics and ontologies used within each system element in the enterprise system of systems. Consider use of a unified ontology for the enterprise to ease semantic mapping issues inherent to systems integration.

Ensure requirements, business capabilities, enterprise architecture guidance, and projected outcomes of future-state systems and processes are well documented and available to the whole enterprise so they can see the goal and destination and a clear vision of the future.

Implement a design control process (Ackoff problem dissolving approach and configuration management best practices) to manage at least any complex, high-cost, or generally high-risk transformations in the program portfolio, which requires Enterprise Architecture and Enterprise Systems Engineering review and a clear configuration management process.

Create robust quality management programs and utilize integrated and cross-functional development teams for complex systems with a high risk and large investment or cost.

Create an enterprise-scale technology roadmap enabled by business relationship managers that accounts for planned transformations based on Demand Shaping principles to ensure stakeholders of individual transformations can align with an enterprise-scale direction and vision. Consider integrating Theory of Constraints and Activity-Based Costing models at various levels alongside trade analyses in preparation of your roadmap for throughput and cost optimization locally and at larger scales, and involve systems engineers, cost accountants, and finance specialists.

Ensure proper knowledge, skills, and abilities exist in your workforce to architect, design, apply, and implement the required systems and processes. Work to ensure that leadership, regardless of domain, has a high digital literacy and regard for enterprise architecture and strategy.

Implement a post-implementation review process for all digital transformation initiatives in order to ensure knowledge on lessons learned and potential issues are available to stakeholders and leaders of new initiatives.

Actively manage knowledge sharing practices to help facilitate innovation and new ideas, as well as ensure teams across the organization maximize the benefit and use of new capabilities from all transformations by fostering communities of practice and purpose and codifying expert knowledge from the transformation initiative.