008 No Stats. No Effect.
You don't need to understand the mathematics behind statistics to be able to use them. Learn how to spot product claims that aren't worth your time and money. Load up this example to follow along with in the episode: http://michigansoybean.org/wp-content/uploads/2015/08/2016-Radiate-Trial.pdf
Transcript of Episode 008 of the Plants Dig Soil podcast – “No Stats. No Effect.”
[Intro Music]
Hello! This is Scott Gillespie and welcome to the second season of Plants Dig Soil. In this podcast, you will learn ways to transition from conventional to regenerative practices in agricultural, horticultural, and home gardening systems.
[Transition Music]
The topic of today’s episode comes from questions I commonly get from clients. They see a product and they want to know – is it worth it for my farm? I also see these products coming up in my social media feeds, stories or ads in the farm press, and booths at trade shows at conferences.
The first thing I look for when evaluating a product is statistics from replicated trials. Don’t tune out here, please! I won’t be getting into an in-depth discussion on statistics! Going to statistics comes from my Master’s student time running my own trials. I learned the importance of statistics and I want to convey that to you in an easy to understand way.
So if you’re a farmer, think of sitting in a coffee shop talking to other farmers. Or maybe it’s a night out for wings and a few beers. As a gardener maybe you do this same thing with other gardeners? As I’m recording this podcast it might have changed to all virtual meetings but the point is to think of a time of casual conversation.
You bring up a new product that you’ve seen advertised. One person says – yeah, I tried that it and worked really well. A couple of other people say they tried it but it was hard to tell because the yield was about the same. Finally, one person says, you know, I actually got a little less when I tried that.
What you have here is a trial. The question is – how do you figure out if the product is worth it? In this coffee shop trial, you’d probably go by gut feel. You’d probably give more weight to the people you trust. But in a scientific trial, you need an objective measure to answer your question.
What’s hidden from most trials that get published is the variation. In the coffee shop example, you have four numbers and you can see the variation. One person had a great response, two people had no response, and one person had a slightly negative response. If the average was a slight response and the benefit was greater than the cost you could conclude that it's worth it to use it. But how do you know you’re the one who’s’s going to get the benefit? How can you be confident?
This is where statistics come in. As promised, I’m not going to get into details of how they work. You don’t need to understand how they work to use them. Just like you don’t need to understand how a car works to drive it. So what do statistics on trial data tell you? In short, they help you to sort out whether something is a random effect or if there is a good chance that there is a real effect.
In other words, they help you to see if there really was a difference between treatments or if it was just random chance that gave you different results.
Let’s look at a few examples.
Say there is a product that claims it’ll boost your soybean yield and it only will cost you $4.50/ac. You just add it to one of your normal sprays so it’s not even an extra pass. Let’s say its organic approved so it could apply to any farming system. With soybeans going for $9.00/bu all you need to do is get an extra half of a bushel per acre and it’s paid for itself. Anything extra is more money for you. What have you got to lose?
Back to the coffee shop. Say the first person had a 5 bushel increase, the next two were at zero change, and one person had lost a bushel. The average here is a 1 bushel increase.
Say you go to another town and you find that two people had a 2 bushel increase, one person had a 1 bushel increase, and another had a 1bu decrease. The average is still a 1 bushel increase.
Now for the last set of numbers – say you go to one more place and you hear one person lost 6 bushels, one person gained 9 bushels, one gained 5 bushels, and another lost 4 bushels. The average is still a 1 bushel increase.
When trials are published and there are no statistics included, you don’t know the variation. You only see the average. Statistics give you a measure of the variation and can help you decide if it’s worth it. When you see a bar graph or a table that shows a bunch of numbers that look different but the statistics say there isn’t a significant difference this is why – there’s too much variation in there to know if there was a real effect or not.
[Transition Music]
How am I doing so far? I know it’s not an easy topic to follow but I want you to learn how to critically look at product claims. Feel free to email me your questions – scott@plantsdigsoil.com
While we’re taking a little break from statistics, I want to let you know about being a guest on the GardenFork podcast[1]. I’m sorry about the audio quality – but after recording this I took the suggestion of the host, Eric, and got a better microphone. This podcast fits well with the previous Plants Dig Soil episode called “Plants as Soil Amendments”. In the GardenFork podcast, we cover local soil amendments, troubleshooting raised bed issues, and soil testing. There will be a link to the episode in the show notes and in the transcript.
Okay, back to the show!
[Transition Music]
For a good example, look at the trial that I drew this example from[2]. There will be a link to this in the transcript on my site and in the show notes. It was a very simple setup. Get eighteen farmers to spray the product and leave checks and then measure the yield. It doesn’t say how many times it was replicated but if they can calculate the statistics then it was replicated. In the coffee shop example, there were four replications, and we asked four different people. In this case, there would have been four separate areas where the product was applied and not applied to the field.
Since it only reports the location averages I made up some numbers to show how an average number can come from so many possible combinations. What I like about this trial report is that it’s short and to the point. It all fits on one page. The only thing I have against it is that it’s only one year but I understand that multi-year trials take a lot more effort.
When you look at the trial you’ll notice that one location of the eighteen had a significant effect. Two more had a numerical increase that would be enough to pay for the product and make some money. Three of the locations just barely or not quite paid for the product. The remaining twelve locations cost more than the return.
If you live in the area where the trial showed a significant effect and you can figure out why there was an effect, it may be worth using. However, the evidence that I see is that there is no reason to use this product.
I want to switch to marketing brochures now. This trial was done by a grower organization to test a question that their members wanted to know the answer to. A company could take just the trial where they won and put that in their brochure. How would you know that there were seventeen other trials with no effect? Or if they took out the statistics and put in the three where there were positive numbers above the cost of the product. You wouldn’t know that only one had a statistical effect and that the other two had too much variation to say that the effect was real.
I have seen this used by companies. Websites have trials where they win every single time. It never fails!
I’ve also seen companies refer to having university trials to back up their research. There were no statistics and there was no researcher from the university mentioned. It doesn’t matter that the trial was done on a university if you can’t show that the university conducted it and provided third party data and statistics.
It’s possible the company rented the university land and did their own trial. Or they hired the university to run the trial. If the university is hired to run the trial and send all the data to the company it really has no relevance whether the trial was done on university land.
Another misleading tactic I’ve seen is to give results from replicated trials and provide no statistics. Just because it’s replicated doesn’t make it real. Think of the example I gave where there could be a 6 bushel loss, a 9 bushel gain, a 5 bushel gain, and a 4 bushel loss. It’s a replicated trial – we asked four people. But can you really conclude you’ll get a 1 bushel gain? There’s too much variation there to determine that.
Now for the final topic for today – I want to talk about statistical significance versus biological significance. This was a concept that one of my thesis advisors talked to me about. Just because we find something with statistical significance it may not mean anything biological. Or at least in farming or gardening, it may not contribute to the final yield or the bottom line.
Where I see this at play the most are the pretty pictures. You know what I’m talking about – you’ll be seeing them soon on Twitter. Bigger root masses, greener plants, more leaves per plant! Will a company rep ever put up a picture where their product didn’t show a difference? Is it possible that the farmer celebrity you are following is an influencer – if not paid to promote the product, they may be compensated with free product with the understanding they will promote it to their followers.
Many times these products that give a visual differences don’t translate to yield. Use what you’ve learned today. Look for trials that are done by third parties. Make sure they are replicated, have statistics, and that you see all the trials – not just the ones where there was an effect. If you see some pictures on Twitter that look too good to be true tag me in it! My social media links can be found on my website – www.plantsdigsoil.com.
I have many sources I rely on but it would be nearly impossible to list them all here. Universities, research organizations, and state or provincial extension services are usually a good place to start. If you’re not sure send me an email – scott@plantsdigsoil.com. I’ll help you figure out the trustworthiness of a source.
[Transition Music]
Remember to get local advice before acting upon this information. If you don’t know who to talk to, get a hold of me and I’ll help you find someone. If you’re in my local area and are in need of help, contact me. It's always free to chat. If we get to the point that the scope broadens to consulting work we can work out a plan that fits your budget.
Would you like to keep up with me through a free monthly newsletter? Go to www.plantsdigsoil.com/contact and select the newsletter option. If you haven’t subscribed to the podcast yet please make sure you do that in your favourite app. New podcasts come out once a month so, just like the newsletter, you won’t be overwhelmed with information.
If you’re still listening you’re probably like me and like to know what the catch is. Why am I putting out this information for free? The reason is that I love to learn and I love to share the information. My knowledge has been built up from experiences in my own garden, advising farmers and agronomists in my consulting business, and from reading the latest books and articles on agronomy and regenerative agriculture.
I have a B.Sc (Agr) with an agronomy focus and M.Sc with a focus on Plant Science. Beyond my formal education, I have attained and maintained my Certified Crop Advisor designation and am a member in good standing with the Alberta Institute of Agrologists.
Nearly everything I talk about is from free resources posted to university and research organization websites. Books that used to be hard to track down are available to buy or borrow for nearly anyone with an ereader. The information is out there – sifting through it all is what takes the time.
I make a living entirely from consulting. I don’t sell any products, software, or systems. I strive to be as independent and as unbiased as possible so I can provide the best advice to my clients and help as many people as possible move from conventional to regenerative agriculture.
[Outro Music]
[1] GardenFork. 2020. 536 – Easy Ways To Improve Your Soil. https://podcasts.apple.com/ca/podcast/gardenfork-radio-diy-maker-cooking-how-to/id330291397?i=1000470203491
[2] Michigan Soybean Promotion Committee. 2016. Radiate Trial.
http://michigansoybean.org/wp-content/uploads/2015/08/2016-Radiate-Trial.pdf