Want to Shift from being a Data Monitor to being a Data User? Here's How.

A couple in a TV commercial calls a termite inspector to determine if their house has termites.  The inspector confirms that the house does indeed have a termite problem when he observes the couple’s daughter fall through a termite-damaged stair.  Much to the dismay of the couple, the inspector says he is only a pest monitor and they need to call someone to exterminate the termites.

In this ad from LifeLock [1], the company exclaims they not only monitor problems, but also fix problems when they find them. While I scoff at this commercial, it struck me recently that L&D reporting shares a lot of similarities with that termite pest monitor. We spend too much time monitoring our results rather than using them.   

In an admittedly unscientific poll I conducted this month on LinkedIn (n=30), 87% of respondents used less than half of the data in their reports for decision making. I suspect the people who generate these reports believe their output is useful to someone, otherwise they wouldn't generate it. But the data (and my client experiences) suggest that we spend a lot of time monitoring data rather than using it.  

Why We Become Data Monitors

I doubt anyone aspires to be data monitor. You probably feel a tad guilty about ignoring that 20-page slide deck that shows up regularly in your email.  In the midst of all those urgent and critical to-do’s, the data in those reports likely feels irrelevant and not particularly helpful. To protect the little discretionary time available, you tell yourself: “If there is something really important in that report, I’m certain someone will let me know.”   

We often unwittingly become data monitors because of systemic organizational problems:

• Leaders unrealistically expect us to manage or oversee a large number of measures
• Most data in the reports isn’t pertinent to our role or accountabilities (as my little poll suggests)
• The important insights are not immediately visible
• The data is static and often backward looking    

How to Shift from Being a Data Monitor to a Data User

In David Ulrich’s excellent 2015 book, “The Leadership Capital Index” [2] he writes that organizations need to do two important things with its information (See the infographic below):

• Manage the information, specifically its flow, transparency and quality
• Use the information not only to make decisions but also to solve puzzles and uncover mysteries. 

I’m not going to discuss the management of information in this blog other than to state that it is the foundation required for effective data use. Instead, I will focus on the use of the information.  

Use information to Solve Puzzles

In an article by Millward Brown [3], Phillip Herr wrote, “When we solve a puzzle, we end up with a specific answer — one that is quantifiable, comparable to answers to other puzzles treated in a similar manner, and invariably a go/no-go decision.”

In L&D, we solve puzzles frequently. We may ask, “Did the senior directors get more value from the leadership course than the executive directors?” “Did our data suggest that employees with high manager support were more likely to improve performance vs employees with low manager support?”  Through basic statistical analysis, we can get straightforward answers to these puzzles. We may need another level of analysis to uncover why senior directors get more value from the course, but we now have direction and a means to understand both the “what” and the “why”.  Our answers to these puzzles should help inform our decisions and possible actions.

L&D organizations with whom I work do a solid job of asking questions and solving puzzles.  However, if you suspect your organization is more prone to monitor rather than use data, consider these approaches:

• Explore where you exceed as well as fall short of your goals.  Ask questions that uncover the factors that drive successful outcomes.
• Identify how results vary by key demographics such as geography, course, delivery vendor or location.  When you separate data by demographics you can uncover specific areas that warrant further investigation or even help you solve the puzzle.
• Look for correlations of key variables. Do employees derive more value from their community of practice when the content is more readily accessible? Is value more highly correlated with the number of opportunities to interact with other community members? Correlation data can help you explore relationships that are not visible through trend or comparison analyses.

Use Information to Make Decisions

In almost everything we read about data (big or small), we are reminded that we should use information for decision-making. The challenge is that most reports give us information but insufficient insight.

If “information over insight” describes your organization, you should consider how to restructure your reports to ‘speed time to insight” and ultimately decision making. Consider the following approaches:

• Ensure reports are role relevant.  If you manage L&D leadership programs, the reports should highlight the data relevant to those programs you specifically manage. You should not have to hunt and dig to find what is specific to your programs. If you receive data via a dashboard, when you click to the site or app, it should know who you are and display your content on the first screen.
• Display all results against a goal. Your performance against goal will inform your decisions and actions. The comparison (with appropriate color-coding or symbols) makes it easy to highlight large variances that require attention.
• Annotate your reports. Context is critical; add comments to explain the cause of a deviation from the trend or goal.  
• Use appropriate data visualization methods. Learn how to display data; avoid chart junk or difficult-to-read displays to ensure the message is clear.

Use Information to Uncover Mysteries

I commonly see leaders use information to solve existing problems and make decisions. What Ulrich makes clear is that problem solving and decision-making is not enough. Competitive advantage springs from seeing patterns and themes that highlight emerging problems or opportunities.  This is the territory of "Big Data" and where you need to uncover mysteries.

Unlike puzzle solving, “when dealing with a mystery, no one can address the problem directly; someone has to discover the pattern first. The problem with mysteries is that sometimes leaders don’t even know the right questions to ask to explore them.[4]” 

How do we identify the mysteries we should solve? First, connect the dots across multiple data sets.  For example, what is the relationship between engagement and the use of social/informal learning resources? Second, identify how to mine unstructured data such as emails or online discussions. Third, stay attuned to market and industry shifts and use your own data to assess if your organization is keeping pace.  

Here are specific actions you can take to identify important mysteries to solve:

• Identify sources of information and listening posts such as discussions between performance consultants and their business clients as well as presentations or blogs by industry leaders or researchers.
• Quarterly, identify themes from your listening posts. Discuss articles about shifts in the industry or a success story from a company you admire. Develop hypotheses and then identify how to confirm or refute them in your own organization.
• Incorporate insights into the annual planning cycle. Where should you allocate funds to conduct a proof of concept of a new process or method?  Where do you need further insights about how your employees learn or consume content?

Who owns the shift from monitoring to using data?

If the L&D measurement and reporting team were solely responsible for moving the organization from being data monitors to data users, you could simply pass along this post and ask them to make it happen. But, like that termite problem, it’s not just about the termite exterminator. The homeowner has some responsibility as well. 

Let’s explore how those responsibilities sort out (also see graphic).

• Measurement and Reporting (M&R): To shift the organization from monitoring to using reports, the M&R function needs to rethink its approach to data and reporting. 
• Eliminate one-size-fits all reporting 
• Enable users to interact directly with the data to ask and answer questions, drill down and explore the data beyond the surface
• Guide L&D to focus on the critical few measures to manage the business. Help L&D align its measures to its business priorities
• Coach L&D to use a balanced suite of measures. Include a mix of both leading and lagging indicators. Report on not just efficiency measures but effectiveness and outcomes as well.  With a balanced suite of measures, report users will be provided with a holistic view of their performance. 
• Report users:
• Develop goals for every measure you actively manage. (I can't repeat this enough.) Goals define what ‘success looks like’ and help focus your attention in those areas that need course correction. 
• Schedule time to reflect on the data, ask questions and explore insights. Confirm or disprove rumors or ‘conventional wisdom’ through your analysis.
• Establish a process to synthesize and interpret unstructured information. Qualitative and unstructured data has an important role to play.  Ensure you bake it into your data use regime.

Summary

Being a data user is lot more rewarding than being a data monitor.  Make data use a priority by reviewing your reports regularly with your immediate team. Discuss insights and assign follow-up. With the right data, the right type of report and thoughtful review, you can hone your analytical thinking skills and improve how you lead and manage your function.

---

[1] I do not use the LifeLock product or invest in the company
[2] "The Leadership Capital Index", Dave Ulrich, Berrett-Koehler Publishers, 2015, http://amzn.to/2dKATQH

[3] Solving Puzzles Delivers Answers; Solving Mysteries Delivers Insights. 2011, http://bit.ly/2dQ9kmb

[4] "The Leadership Capital Index", Dave Ulrich (see source above)

Four Approaches to Break the "No One Is Asking" Cycle

In the last 18 months, I have noticed a concerning shift in the sentiment of learning professionals about ramping up the quality of their measurement and in particular their reporting. This sentiment had been quite prevalent 10-12 years ago, but abated for a while. Now it's back with a vengeance. 

The essence of the sentiment is “Business leaders aren’t asking for this information, so there is no need for us to provide it.”  Most L&D practitioners don’t state it quite like that of course. They say, “Our business leaders only want business impact data." Sometimes the stated reason is that business leaders consider L&D self-report data to be useless. Regardless of how they say it, the message is clear: “Business leaders aren’t asking for it (or worse, don’t want it), so we are not going to bother.”

I'm not imaging this shift. Brandon Hall Group recently published a study entitled, "Learning Measurement 2016: Little Linkage to Performance." In their study, they found that the pressure to measure learning is coming mostly from within the learning function itself. In addition, of the 367 companies studied, 13% said there was no pressure at all to measure learning. 

What’s concerning about this situation?

L&D has been saying for years that it wants to be a viewed as a strategic partner and get a seat at the proverbial table.  Yet, the sentiment that “no one is asking for measurement data” is equivalent to saying, “Hey, I’m just an order taker. No orders, no product or service.”   If we are going to break free of an order-taking mentality, then we need to think strategically about measurement and reporting, not just the solutions L&D creates.

The sad reality is that often business leaders do not ask for L&D data because they don’t know what data is available or they don’t understand its value.  Many business leaders assume L&D is only capable of reporting activity or reaction (L1) data. Many still pejoratively refer to L&D evaluations as Happy Sheets or Smile Sheets.  If they believe that is all L&D can do, then of course they won’t ask or exert any pressure to do more.

Equally importantly, innovation would come to a standstill if organizations only produced products and services that resulted from a client request. Henry Ford famously said, “If I had asked people what they wanted, they would have said faster horses.”  Steve Jobs talked about the dangers of group-think as a product development strategy when he said, “It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them.”  Simply because your internal client cannot envision what you can provide does not mean you shouldn’t provide it and then iterate to get it right.  

The big problem with a “no one is asking” mentality is that it is a self-fulfilling prophesy and a downward spiral. At some point, someone needs to break the cycle. It might as well be L&D. (See the Infographic) 

What can you do differently right now?

Fortunately, as an individual learning practitioner you can break this cycle. Here are four approaches (singly or together) you should consider:

1.  Reframe the problem: If truly ‘no one is asking”, stop and ask yourself, “Why aren't leaders asking?” Is it because this information has no value or because they don't understand the value? Is it because they don’t know what L&D can provide?” Based on your answers to these questions, develop a game plan to demonstrate the value of the data you can provide (and then provide it to them). 
2. Find a friendly: If you want to break the ‘negative reinforcing loop’, find a business leader who is willing to work with you, a ‘friendly.’  You can spot a friendly fairly easily. She is interested in how L&D can add value to her business. He views you as a trusted advisor, brainstorming how to build new skills and capabilities within his team. She is innovative and willing to try new approaches even if they might not succeed the first time out. If you have one or two such business leaders with whom you work, seek them out and discuss what data you could provide to answer their questions. 
3. Report on data that matters to the business leader: From a measurement and reporting standpoint, L&D still puts too much of its energy into gathering and reporting data that matters to L&D but is not compelling to the business.  The number of employees who attended courses or the results of your Level 1 evaluation are simply not important to the business. Look beyond your current measures and educate yourself on best practices in L&D measurement.  Integrate questions about learning application and support into your evaluation instruments. This is data business leaders will care about if you show them how it affects them and their organization. 
4. Tell a compelling story: Do you remember the Magic Eye picture within a picture phenomenon? If you held the picture up to your nose, you might see the constellation Orion buried inside the picture. (I never saw anything.)  If you believe your data is meaningful and can help the business, don't use the Magic Eye approach. Don't expect your business partner to find the meaning in the data. Rather, tell the story behind the charts and graphs through dialogue. Help your business partner connect the dots; help her understand the consequences of not acting on the data and the benefits if she does. 

A real life example

The ability of employees to apply training in the workplace depends on several conditions, much of it outside control of the L&D department. Factors such as the motivation of the employee, the opportunity to apply the training to real work and the reinforcement of the direct manager all affect the extent to which training is applied.

A few years ago, I worked with a company that sold complex financial software. They re-engineered their implementation process to simplify the client experience, reduce implementation time and accelerate revenue recognition. The business leaders identified project management (PM) skills as critical to the success of this new approach. 

The Process Transformation Team identified an initial group of employees to attend the PM training and pilot the new approach with several clients. When they reviewed the pilot results they were disappointed. Implementation time had not declined appreciably and the client felt that the process was more complex than expected. The Team Leaders investigated and found that the employees' managers were not reinforcing the training or directing them to support resources when they struggled to apply the PM methodology in a real life setting. They also found that the job aids L&D had created were too cumbersome and not designed to be used in a dynamic client setting.

Imagine you are the L&D partner of the Implementation Transformation Business Leader. What data could you have provided to demonstrate that his people were not building and honing skills in this critical discipline?  

This leader needed a regular stream of L&D data on actual application on the job and barriers to application.  He needed data on employees' perceptions of how this training would impact business outcomes of simplification, implementation time and reduced time to revenue recognition. He needed to understand the barriers to application and where the business was accountable to address the issue or L&D. Moreover, he did not need 

incontrovertible proof that the training was improving business outcomes.

Data from L&D was essential for this leader to take action and address issues that affected his ability to successfully transform a key client process.  Unfortunately, the business leader didn't realize that L&D could help him get ahead of this issue and didn't think to ask. After L&D and the business leader started talking, sharing data and insights, the Leader not only acted, but worked with L&D to develop regular business-oriented reports.

Final thoughts

As an L&D practitioner, you can break the negative reinforcing cycle. Why not regularly provide this type of data and use it to create a dialogue about what other insights you can provide?  Why not take the first step to dispel the belief that L&D has no useful data or insights to offer?  It's in your hands.

Have you had a "No One is Asking" moment? I would love to hear from you. Follow me on Twitter @peggyparskey or connect with me on LinkedIn at https://www.linkedin.com/in/peggy-parskey-11634