I define Organizational Lore as the sum of all the nuggets of undocumented knowledge within a company that are passed along by word of mouth. Most organizations have some and many have critical processes that rely on it. Often ignored, Organizational Lore can be extremely costly to your operations and left unchecked can pose a real risk to your business.
Story Time!
A company that I’m familiar with sells a decision support tool that runs off a custom database. Each customer has their own database and the data is received from a variety of different sources. New data must be standardized and loaded into the customer’s database regularly. The company has a complex data management operation where Data Analysts are responsible for:
- Verifying the incoming data quality
- Transforming the data (including writing the data transformations)
- Verifying the quality of the transformed data
- Loading it into the customers database, which can be heavily customized depending on the type of data being stored
- Quality checking the database load using their customer-facing application
The Data Analyst job requires a broad range of technical and analytical skills that must be learned on-the-job. There are manuals, however no one could pick up the manual and do this job. Why? Because the manual describes the most “vanilla” case, and in reality every client has their own “flavor”. Someone, likely the person who worked on that customer previously, has to be available to explain the quirks, answer questions, and help with the inevitable troubleshooting. Does this sound familiar? How successful do you think this company is at consistently producing high quality databases to support their products?
The cost of Organizational Lore
The effects of Organizational Lore in these situations can manifest in several ways:
Unpredictable quality and an inability to recover quickly
Mistakes are going to happen when your processes depend on Organizational Lore. Recovering from a failure is harder because no one is familiar with your exact “flavor” of the process. Product Support tries to help, but is less effective and has to escalate more often. Quality suffers and your customers lose faith in your ability to deliver reliably.
Steadily growing complexity
When a problem came up at the example company they would change their standard process to prevent the problem from happening again. They would add steps, create checklists, or add stopping points in the process for review from a senior resource. The result, however, was a frustratingly slow and complex process that took more work to complete and didn’t improve quality.
Turnover trouble
The burnout rate is high for jobs like the Data Analyst in the example above. Not feeling secure in your ability to do your job is stressful, particularly for new employees. Worse, when you lose senior people you lose the part of the lore that was stored in their head, which leads to….
The loss of “why”
“Why do we have to do this step?”
“I don’t know, we’ve always done it that way.”
Over time processes can evolve beyond the bounds of institutional memory. People know they have to perform a step, but they don’t know why. This makes it very difficult to troubleshoot problems or make operational improvements. No one wants to break the system!
Diminished security
It’s hard to secure something you don’t understand and holes can be buried in mounds of complexity. It’s as simple as that.
A silver lining?
Is there a positive side to Organizational Lore? Possibly. Processes that are hard to learn are even harder to duplicate. If your business relies on this kind of a process it could be a competitive strength. However, process that rely on Organizational Lore tend to get more complex over time and a new competitor might be able to create a new process without all the baggage and your advantage would quickly become a disadvantage.
What to do about it
If you’re in a situation like the example company it’s not as hopeless as it might seem. There are things that you can do to get back on the right track:
Simplify!
Processes that evolve over time can get huge and messy. Special cases and work-arounds accelerate this. It’s a good practice to reexamine why things are done the way they are and to consider alternatives. And, don’t forget to listen to the people that actually do the work. They live the process every day and will have important thoughts on how to fix it.
Insist on documentation and watch for smokescreens
Although hopefully rare, I’ve seen instances where Organizational Lore has been created intentionally to provide “job security.” Be open-minded and pay attention if someone tells you that they can’t create documentation because the process is too complex. It may very well be true, so you need to assess the process and related technology to understand what’s going on. But if it’s not true, you may have a knowledge hoarder that needs to take part in some mandatory cross-training.
Move the lore from your people to your technology
Often times Organizational Lore compensates for deficit technology. If the system can’t handle a special case, then the people in operations have to work around it by passing around lore, creating their own tools and utilities to fix problems, etc. This is okay in the short-term to avoid an operational slow down, but fixing the technology should be a high priority. Remember, much like Technical Debt, you can also create Operational Debt. The time your people are spending on fixing problems and creating workarounds is taking away from the time they should be spending on operational efficiency. Let the technology do the hard work and allow for more standard processes for your folks in operations.
Organizational Lore can be a tough problem to solve, but with the right analysis you can find the technological improvements and process efficiencies you need to make it right.
