When it comes to research, accuracy is vital to produce empirical results. In order to perfect this process, researchers must always stay skeptical, and, when the time comes, critical.

Lalitha Naveen, associate professor, and Lily Li, research assistant professor, colleagues in the Fox Department of Finance, discovered a misconception while trying to replicate the results of a published paper. The study was trying to prove that CEOs are rewarded through increases in pay when firms experience good luck, but not penalized through decreases in pay when firm performance goes down due to bad luck. 

“We couldn’t reproduce the original results,” says Naveen. “So, it got us to thinking why the results are just not there. We found out that their methodology was wrong.” Even when using the original paper’s flawed methodology and then trying their own, the researchers still could not reach the same results. “The way they approached the problem was all wrong.”

The original paper attempted to understand whether or not there is a substantial correlation between CEO pay and luck in a firm’s performance. Naveen and Li, along with co-author Naveen Daniel of Drexel University, discovered the issue behind replicating this correlation between luck and skill. “By definition, if you believe luck to be something random, then it shouldn’t be correlated with anything,” relays Li.

But, if these results can’t be replicated, what is the answer to the question of correlation between pay and luck for CEOs? The researchers, using their own robust methodology, found a more precise result.

“CEO pay seems to be moving equally up or down depending on the firm’s performance due to luck. It’s not that CEOs are extracting money from shareholders,” says Naveen. “Now, the CEO could still be doing (that). But we are finding they are not able to shield their pay from bad luck,” she concludes.

Their paper, “Symmetry in Pay for Luck,” was published in the Review of Financial Studies Journal, one of the top three journals in Finance.

Both Naveen and Li stress the importance of methodology and constructive criticism in research, especially after their experience in this paper.

“Our type of research is empirical, which means we’re trying to answer questions by looking at the data,” says Li. “So, that’s why using the right methodology to process the data is very important. Because, if the method is wrong, you will reach the wrong conclusions.”

Beyond using the right method, being skeptical is even more crucial.

“I think it’s really critical to be critical. That’s one of the things they teach you in research, is to question everything. That’s how you get the answers and that’s how you evolve. The challenge is to do it in a constructive way,” explains Naveen. “There are too many people willing to say, ‘Hey, your results are wrong,’ without backing it up with data or arguments. And I think that’s what the challenge is for a good researcher.”

Learn more about Fox School Research.

For more stories and news, follow the Fox School on LinkedIn, Twitter, Facebook, and Instagram.