Introduction
The scientific literature and popular press are full of articles that purport to have found the perfect alternative to antibiotics (AB) for promotion of growth and control of disease. All of these reports claim to provide incontrovertible proof that the ingredient/product in question is the best possible alternative. Unfortunately, many of these studies – even in the scientific press – are not organized, conducted, analyzed or summarized correctly and therefore, cannot make the claims regarding their efficacy
Studies to determine the potential value of an antimicrobial alternative must be properly organized to properly interpret the underlying biology that may (or may not) be occurring in the animal. There are a number of factors that must be included in any study comparing antimicrobial alternatives. While these may seem “obvious”, many studies that do not provide one of more of these factors. Therefore, it is not truly possible to know whether the responses observed are real. Generally, research reports that are included in scientific literature (i.e., peer-reviewed journal articles) have a greater probability of being properly conducted, analyzed and reported. However, this is not necessarily so. Publication in a scientific journal does not guarantee that the data will work in every situation – or if the results are even real. Some factors that are critical include those listed below.
Negative and positive controls
Negative and positive control treatments are essential to understanding any response of a proposed antimicrobial alternative. A “negative” control without AB or the test product in the feed will tell the researcher whether there was a response to any of the treatments. In some experiments, animals will not show a significant response to the experimental treatments – even to AB. This usually happens when animals are healthy (have an adequate passive immunity) and are exposed to few microbial challenges. When animals have adequate immunity and face little immune challenge, you can’t really expect that an antimicrobial would improve health or growth. Unfortunately, some researchers have conducted experiments without a negative control. They report no significant difference between the treatment containing the test product and the treatment containing the antimicrobial and conclude that the test product must be able to replace the antimicrobial. Well, this may not be the case at all. It may actually be that neither product exerted an effect because the animals were not challenged in any way. Without a negative control, it is impossible to tell whether the product worked at all.
A positive control with an antimicrobial included in the feed at amounts typical in the industry and manufactured, prepared and fed according to industry standards. A treatment containing an antimicrobial is usually included in any study, but it is important to be sure that the research was conducted in a way to allow a response. Including an antimicrobial in the diet at improper levels or fed improperly may allow a researcher to declare statistical significance – but what about biological significance?
Sufficient numbers of animals
This is a very common problem in many research trials comparing dietary treatments in calves. In many cases, calves are assigned to trials at a set day after birth and fed for a period of days or weeks. On small farms, a trial conducted in this manner may take years to finish if large numbers of animals are needed. Instead, researchers will use only the number of calves that “fit” into a calving season, academic year or within a graduate student’s research program and don’t give consideration to the statistical ramifications to this decision. The actual number of animals required to detect a statistically significant response depends on the conditions, treatments, and amount of variability in the measurements being taken. There are methods that researchers can use to determine the number of animals required for a study. These are called “power calculations” and can be found in most statistical textbooks.
Generally, however, it is our experience that measurements of growth, intake, efficiency and on-farm health measurements (fecal scores, rectal temperatures, etc.) under our conditions need a minimum of 20 calves per treatment to see significant differences in most studies. It is unlikely that biological differences can be observed in most studies with less that 20 calves per treatment and readers should be very careful in interpreting data without some indication of the number of calves per treatment.
An indication of the passive immune status of the animals
A biological response to an antimicrobial or alternative depends to a great extent on the passive immune status of the animal. Calves that receive a large amount of passive immunity will react to dietary treatments much differently than calves that receive no colostrum. This measurement can be by the measurement of serum or plasma total protein, IgG or other indicator in the blood (e.g., zinc sulfate). However, an indication of the “starting point” of the animals used in the study is essential to understanding the results observed.An indication of the immunological challenge on the animals
This area is somewhat more subjective, but can be summarized by this analogy. Calves that are born, sold immediately and trucked to a sale barn and then to a calf ranch without the benefit of colostrum, water or milk for a couple of days will probably respond to a dietary treatment differently than calves born on a dairy, fed lots of high quality colostrum and kept in one location throughout the experiment. Conditions of calves prior to the study can sometimes influence a study more than the experimental treatments. Knowing the animals upon which the dietary treatments are imposed is as important as knowing the treatments themselves.
Some researchers attempt to control the amount of immunological challenge on animals by imposing an external challenge. A common approach is to feed animals the experimental treatments for a period of time and then challenge them orally with a pathogen, such as E. coli, salmonella, Cryptosporidium, etc. These are usually more informative studies than those that rely on “natural exposure”. However, it is important that the organism and dosage be correct and well defined. Many research reports provide too little data to properly understand the nature of the organism used in the study.
Measurements that can detect differences
This is so simple, it sounds silly. But there are many studies that measure the wrong things. Typical measurements with calf studies are intake of milk, starter and hay (when fed), body weight, feed efficiency (body weight gain / intake) and fecal scores. Larson et al. (1977) suggested more uniform methods for measuring fecal scores and other measures. However, in many cases, these measurements, although easy to determine, are inadequate to show biological differences among treatments. Intake of feeds – especially milk or milk replacer – are usually inappropriate measurements, since in most cases, the amount of milk replacer is fixed by the researcher. When calves are fed a limited amount (usually 1 lb. or 454 g) of milk replacer powder per day, differences in intake are almost always very small. Are the differences between intake of 454 and 453 grams/day really different? While this may be statistically significant, is it biologically meaningful?
Body weight and average daily gain are affected by many different variables within an experiment. Some of these include the diets used and methods of feeding, weather conditions, immune status of the animals, availability of water, time of the day at weighing and many other factors. Body weight and gain are generally weighed to the nearest 0.5 or 1 pound (in some cases, to the nearest five pounds!). Early in the study, body weight of calves may change little, so there is a significant proportion of error in the measurement. So, interpreting this information can be difficult. Later, when calves may gain >1 lb/day, the proportion of error in the measurement becomes less of a problem. However, as the rumens of calves develop, the confounding factor of gut fill becomes more of an issue. Consumption of large amounts of water or feed, or the voiding of large amounts of feces immediately around weighing time can influence BW measurements. Generally, it is wise to weigh calves at a standard time every day so that effects of meals and gut fill are minimized.
Measuring and reporting changes in calf height or girth on a weekly basis makes little sense. The changes that take place in an animal’s height on a weekly basis are not within our ability to measure in most on-farm settings. Measuring once a month or every three months makes much more sense.
The bottom line is that we need to be sure that the measurements we’re making are consistent with the response we expect. In the case of antimicrobial replacement, it is critical that some indicators of animal health be evaluated. Measures of an animal’s health status are much more informative than many “production” measures. The use of therapeutic veterinary treatments can be a measurement that provides important information on the value of a feed additive that purports to replace AB. However, a well defined and consistent protocol for using these medications must be described to understand how they were used. Of course, the number of calves that die during a study is key. Proper necropsy information to identify the cause of death can help the reader understand why the animal died and how this may have been affected by the experimental treatments. More sophisticated measures, such as concentrations of stress hormones, can give an indication of the physiology of the animal.
A complete description of methods used
Too many studies report very sketchy descriptions of how the trial was conducted. As readers, we too often skip over the methods used and assume that we understand “how trials are done”. This is a big mistake. As consumers of products and information, we must demand a complete description of the materials used in a study. As a rule of thumb, we should expect sufficient detail so that a reasonably proficient person could repeat the study given the detail in the study. It is far too easy to “hide” important details that dramatically influence the interpretation of the data. Even some scientific articles provide insufficient detail to properly judge the validity and applicability of the data.
Beware the means
A common mistake for researchers to make is to spend too much time looking for differences between the means and too little time looking at the means themselves. Researchers have been indoctrinated to look for statistical differences – if there are no statistical differences, there is nothing to talk about. But when there ARE differences, then they’re the only thing that is discussed. Let’s use a “real life” example. A group of calves are purchased and brought to our research facility. We feed them one of two treatments (AB- vs. AB+) in a traditional milk replacer feeding program – 454 g/d of 20% CP, 20% fat product. We measure intake, growth, etc. for 56 days. We find a significant difference in growth rates at the end of the 56-day experimental period. We note that the BW gains were increased by 25% at the end of the experiment! Isn’t that wonderful? However, we don’t consider the fact that during a two month period, calves only gained an average of 100 vs. 125 grams of BW gain per day. We are caught up in the wonder of statistical significance and don’t consider that only very poorly raised (or sick) calves will gain so little body weight during the course of the experiment.
We need to present not only differences among the means we measure, but we also must consider what the means are telling us. If we raise calves ourselves, then we need to compare performance of calves in the studies with performance of our own animals. Are the results typical of what we would expect on our own operations? Do they reflect reality? These are all important questions that can have an effect on the interpretation of our data.
Only positive data?
I alluded to this common problem earlier. Too often, we only see the “effects” of the product in question. We don’t often see when the product doesn’t work. Indeed, this problem occurs with great regularity in the scientific literature. Unless there is something “different”, many authors don’t consider an experiment noteworthy. There’s a lot of information that can be gained from those experiments – when conducted correctly – that don’t show significant differences among treatments. There’s little doubt that any product or concept will not work in every circumstance and under all conditions. In the case of potential AB alternatives, many trials will “fail” because there is too little immunological challenge. Why, even feeding AB don’t always show a positive result. When you are exposed to only trials that show the wonderful, positive results of a potential AB alternative, remember to ask, “where’s the rest of the data?”.
Proper statistical analysis of the data
This is the most abused area of scientific inquiry. Statistics is a tool that assists researchers to organize, conduct and analyze the results of studies and make conclusions based on probability. Unfortunately, many, if not most studies use limited – sometimes elementary – statistical designs and analysis that do not allow a proper understanding of the data. This is especially true when measurements are made on animals multiple times. For example, we may measure the body weight of animals during a study on a weekly basis. We may measure intake of feed and milk daily. This method of conducting a study requires a more sophisticated method of analysis than looking at weeks individually. Use of means separation tests such as Duncan’s or Tukey’s tests often do not properly address the questions being asked. Therefore, it is very important that a proper design be used. When in doubt, a statistician should be consulted before conducting a study!
Beware the generalization
Far too many studies conclude with a statement of something to the effect “these data indicate that [additive X] can replace antimicrobials in calf milk replacer”. Such generalizations are dangerous and usually inappropriate. It is important to remember that research is conducted under controlled conditions – in many cases, artificial conditions. These conditions may or may not reflect the real situation in the industry, or on a particular farm. The determination of whether an additive can replace AB depends on many factors (including many of those mentioned above) that are often held constant in experimental trials. It is almost always necessary to predicate such declarations with the phrase “Under conditions of this study, the use of [additive X] in calf milk replacers resulted in similar responses to antimicrobials”.