Expert Insights

Expert Insights

A Limited Discussion of Limitations and Analytics

A Limited Discussion of Limitations and Analytics
28 May 2019

When I begin a project, especially with a new customer, I always like to ask a simple question: “What does success look like to you?”  For most Biostatistics engagements, the typical answer is simple – valid and unbiased results (ideally with a p-value less than 0.05).  What is often unspoken about this are the limitations and disclaimers that go along with it.  We know that we can’t make inferences about populations we haven’t recruited, endpoints we haven’t measured, or time periods we haven’t captured.  And, even within the limit of what we can say, we use concepts like statistical power to characterize limitations of interpretation.  Because these things are so common and well accepted in clinical trials, we usually don’t take the time to discuss and explore the impact of them.  But, what happens when we step outside the realm of traditional trial statistics and move into consulting and analytics?

If you don’t integrate the discussion of limitations as a key part of the process you run the risk of making poor decisions and could cause people not to trust otherwise useful results.  What happens if the statistician doesn’t warn you that you are under-powered, incorporating the wrong data, or using the wrong inferential tests and things go south?  Where you should be looking at all the factors (analytics included) that contributed to a decision gone wrong, its often easier to simply discount the value or reliability of analytical methods, particularly if you didn’t know there were caveats to start with. This issue is especially true in an environment reticent to embrace analytics as an equal to “gut” and looking for anything to prevent change.

Any analytics engagement should have that kind of discussion early and often; even to the point of influencing the specific business question to be asked or hypothesis to be tested.  Many, if not all, of the limitations that we will face in an analytics problem are related to data – and, often, to data that hasn’t been collected, hasn’t been cleaned, isn’t sufficiently related to the topic, isn’t accessible, or isn’t allowed due to privacy laws.  Although you may start from a perceived ocean of data, you really may have only a small puddle to splash around in. 

The more quantitative data issues can be troublesome, but they may pale in comparison to the problem that perception can bring. A company with 20 years of data history likely doesn’t have 20 years of clean, usable data, but they may not be willing to admit to it.  And, even when a sufficient amount of reliable data is available, that doesn’t mean that the exercise will come to the expected conclusion (or even reach any kind of actionable outcome).  In organizations new to analytics (and especially those that may have some resistance to it) preconceptions may be the greatest hurdle.  Unrealistic expectations may trivialize valid but unimpressive results and engagements that are inconclusive (or go against expectations) can be ammunition for those who may fear the progress.

Taking steps to incorporate an analytically supported decision-making process can be daunting and each step will likely cover far less ground than you want it to.  Given all this, it is important that stakeholders recognize the steps for what they are, see their value, and ultimately see their potential.  The best advice I can give is that the most valuable tool you have at your disposal (whether you are the analyst or the business sponsor) is your voice.  Ask questions, share information, engage, engage, engage, and after all that, do it all again.  And, if after all that, you still have questions, drop me a line, I love a new problem!

Contact us today to find out how we can help with your biostatistics challenges.

Written by Walter Boyle, Executive Director, Biostatistics and Programming at Synteract

x

Submit RFI / RFP

By submitting your information to us through this webform, your business details will be added to our database. In accordance with our Privacy Notice, we may then contact you with marketing information about Synteract that might be of interest to you. We will never sell your details to third parties. You can unsubscribe from marketing communications at any time.

Required

x

Contact Synteract

Tell us how to stay in touch with you:

By submitting your information to us through this webform, your business details will be added to our database. In accordance with our Privacy Notice, we may then contact you with marketing information about Synteract that might be of interest to you. We will never sell your details to third parties. You can unsubscribe from marketing communications at any time.

Required

x

 

x

Download Resource

By submitting your information to us through this webform, your business details will be added to our database. In accordance with our Privacy Notice, we may then contact you with marketing information about Synteract that might be of interest to you. We will never sell your details to third parties. You can unsubscribe from marketing communications at any time.

*Required

x

To complete your download, please choose an option in the Internet Explorer pop-up window at the bottom of this browser.

x

Opt In

By submitting your information to us through this webform, your business details will be added to our database. In accordance with our Privacy Notice, we may then contact you with marketing information about Synteract that might be of interest to you. We will never sell your details to third parties. You can unsubscribe from marketing communications at any time.

*Required