Articles A Guide to the Research Process Every day, EMS managers and operations personnel are required to make decisions -- decisions about equipment, levels of performance and continuing education needs. The information for these decisions ultimately comes from the organized evaluation of data, also known as the research process. Unfortunately, the idea of being involved in any type of research brings many EMS veterans to their knees. People tend to have a preconceived notion that research is difficult, time-consuming and, ultimately, not applicable to the everyday activities of EMS. However, if future decisions regarding EMS are to be based on fact – not presumption – the reality is that EMS must incorporate research into the evaluation of protocols, procedures, medications, and equipment. Simply put, research is vital to the practice of EMS and should be fostered and supported in all agencies and at all levels. Research can be thought of as eight organized steps designed to lead to a well-defined outcome. The first step is the formation of a hypothesis, and the last is the presentation of a final report. In between are six steps, each of which builds on the previous and toward the next. To be successful in the research process, it is important that each step is completed before moving on. Idea Development The first step in the process is identifying a problem or question. The source for a hypothesis could, for example, be recurrent problems with implementation of a new or existing protocol, documentation of the educational needs of the EMS agency or investigation of a manufacturer’s claims about a piece of new equipment. Whatever the source, begin by writing down the idea as clearly and concisely as possible. Now that a research topic has been identified, it’s time to prepare a hypothesis for evaluation. Keep in mind that because the hypothesis forms the basis for the research, it should be a statement that can be proved or disproved using the resources within the EMS system or agency. For example, a hypothesis supporting a certain aspect of care in the treatment of penetrating chest wounds would be difficult to prove in an area with few such incidents. While it is important to be as specific as possible when developing a hypothesis, there will be times during the succeeding steps when the hypothesis should, if necessary, be re-evaluated and revised. Literature Review The next step is to determine what other research has been done related to the topic. Discussions of current theories or new developments can be found in such EMS journals as JEMS, Prehospital and Disaster Medicine, Annals of Emergency Medicine and the Journal of the American Medical Association. Often, major publications can be accessed through an online computer search service. A librarian at a local university, medical school or public library can help construct a good literature search. Additional sources for related material can be found by studying reference lists of existing articles, particularly review articles on the subject, or by searching a closely related topic. Although books can become dated, they can be useful for finding background information or developing a historical perspective. During the literature search, it can sometimes be disheartening to find articles closely resembling the one being contemplated. However novice researchers need to realize that just because a question has been evaluated does not mean it is Each EMS system has operational guidelines and protocols that make it unique. Re-evaluating an existing research topic within the scope of one system is acceptable and can be beneficial, especially by providing valuable new information. Also, repeated research in one specific area can contribute to understanding and growth in all of EMS. Once all the information gained from the literature is reviewed, it is time to review, re-evaluate and, if necessary, rewrite the hypothesis. The following questions should be asked during this period: Is the hypothesis still feasible? Is its focus too broad? Does it need to be narrowed? One common problem in research is taking on too much; the more focused the hypothesis, the greater the probability of success. Project Design Proper design of a project not only increases the probability of gaining useful results, but it will make the entire process much less time-consuming. If the literature review reveals several comparable studies, consideration should be given to adapting a similar project design, including outcome measures and patient-assessment techniques. Besides saving time, this enables the researcher to compare results from his study with the literature. All research projects should be discussed with an experienced researcher, even one outside the field of medicine. Most projects involve multiple variables, and statisticians or experienced researchers can help identify possible stumbling blocks or simple evaluation techniques. The local medical control authority may be a starting point for assistance in project design, and community colleges or universities in the area will have statisticians on staff. During the planning stage, it is also important to consider involving an Institution Review Board (IRB). IRBs review projects involving human subjects to check for patient safety and confidentiality. To find an IRB, check with a local medical center. Planning should also include the type of study format to be used in the project. There are two general study formats commonly found in research. A retrospective study reviews historical data and correlates observations, interventions and outcomes from these data. The information is gathered from existing run sheets or patient records. These studies are inexpensive and relatively quick to conduct, and the results can provide valuable information about system performance, patient demographics and efficacy of interventions. The weakness of a retrospective review, however, is that because data are gathered under uncontrolled circumstances, the conclusions are weak. Many researchers use this type of format to develop a historical basis for future research. A prospective study involves the ongoing entry of patients into the research project. The patient-entry criteria are clearly defined, and the patients are followed to an established end point (e.g., admission or discharge from the hospital). While this type of project is more difficult to conduct, its conclusions are more reliable because specific, uniform data are gathered at specified intervals during the course of the patient’s treatment or hospitalization. Furthermore, this type of study allows for alteration in treatment and subsequent observation of outcome for the specified patient population. Regardless of which format is used, it is important to define the characteristics of the patient population used in the project. Be sure the criteria are clearly defined (e.g., age range and presenting symptoms). Examples and ideas for possible patient criteria can be found in the literature; using a patient population similar to those in related studies will assist in comparing results Another consideration is the size of the sample. The statistician or research adviser can help identify what sample size will provide adequate results. If the sample size is too small, the results may not prove or disprove the hypothesis, as it will be difficult to determine if the finding is a random occurrence or is truly significant. But remember, the more data you need, the longer the study will take to complete, and the more likely it is that the data collectors’ interest will wane. Additionally, a bigger study sample and longer study will likely increase the cost. Data Collection Data collection is the heart of the research process. Without it, there is no study and, if done poorly, the project will be unable to prove or disprove the original hypothesis. Data collection involves the determination of data points, selection of the research team and development of data collection tools. Data Points At this point, the researcher should re-review the hypothesis and decide what data points may provide valuable information in proving this hypothesis (e.g., age, gender and mechanism of injury). The data points should be reviewed with the research adviser or statistician. Keep in mind that once data collection has begun, additional points cannot be added. The list of data points should also be reviewed to determine what the information source will be. For example, if a data point is final diagnosis, will that information come from the emergency department admission record, the EMS run report or both? If some of the data points do not have a readily available source, determine what will be required to gather that information. For example, will authorization to review autopsy records or obtain long-term follow-up data from the hospital be needed? If important data cannot be retrieved, the original hypothesis may have to be re-evaluated. The Research Team The next step is to determine who will gather the information. In a retrospective study, one person can usually complete the data collection; a single collector eliminates the variability that can occur with multiple data collectors. In a prospective study, however, the individuals involved in the patient’s care provide some of the data. But whether one person or many people collect the data, it is important to meet with the potential data collector(s) and review the project’s needs. Feedback on this step can lead to better designs for obtaining the necessary data. If field personnel will be used for the data collection, keep in mind that their first task is to provide patient care. Therefore, information for the study must be easily documented. This is fairly easily accomplished if the information is part of the caregivers’ normal routine; compliance with the research will increase if the field data collection is simple and does not interfere with patient care. Also, look into giving collectors some incentive, such as authorship on the publication, money or school credit. Data Collection Tools Once all the data points are established, it is time to design the necessary data collection tools. Remember, the easier the tool is to complete, the greater the compliance. Suggestions from research team members should be incorporated into the form. Additionally, information to be gathered in the field should be limited to one page if at all possible. However, actual form design will depend on the types of data to be collected on team member input. It is important to meet with everyone who will be collecting data to ensure that they receive the same training in using the form. Project Protocol Once the hypothesis has been formulated, the literature evaluated, the size and characteristics of the sample population determined, and the data collection tools developed, the researcher must define exactly how the data collection is to be conducted (i.e., a protocol or guideline must be developed for team members to follow). Suggestions for the protocol format include using a list, flow chart or diagram to demonstrate the steps in the research process. This should be limited to one page if possible, and team members should be encouraged to post it as a reminder. Project Time Line Every project needs a time frame, as even the most compulsive person overlooks seemingly minor details in the development and operation of a research project. Some target points in a research project include data for submitting the project idea to the agency, projected start date, dates for field data collection, date for an interim progress report, dates for data analysis and date for final written report. Because each project is unique – and as projects develop – other targets may be identified or added. Analysis Once all data have been fathered, the next step is statistical analysis. Before beginning, novice researchers should consult with experienced researchers and statisticians as to the exact procedures they should use to examine their data. Again, sources from a local university, medical school or community college may be called on or assistance. General information that is always useful includes tables of descriptive data on the population (e.g., patient age range, mean or gender). Many of the characteristics measured from the sample population can be initially described in a descriptive table format. In some studies, this may be the only type of statistic required. However, in most cases, additional data analysis will be needed to determine if the initial hypothesis has been proved or disproved. To facilitate discussion with the statistician, a list of questions should be prepared that are to be answered from other data. The statistician may also provide insight into additional points to consider. Once the sample is described, it can be determined whether the research data support the hypothesis. It is important to look at the results carefully and in the context of that particular EMS system only. It is all too easy to try to extrapolate research findings to other systems. However, field conditions vary so greatly that this type of board generalization usually does not work. Presentation of Results Finally, it is time to write up the results of all this hard work. If all the preparation and data compilation were done correctly, the report writing should be relatively painless. Most reports follow a simple structure:
Other Points to Consider Once a study is complete, it is a good idea to keep an active file containing anecdotes from participants or subjects as well as any obvious biases that may have developed despite a careful project protocol. These will be useful for the final report and in the development of the next research project. Researchers are also reminded to provide feedback about the study to the research team members. There is nothing worse than working on a project only to never hear about its outcome. The team members comprise an important part of the project and should be informed of the final results. Conclusion Research is necessary for EMS to grow and evolve, and involvement in the process is important for all EMS personnel. Developing projects within an agency to evaluate system performance is just as important to EMS as is evaluating prehospital use of thrombolytic therapy. Growing to a comfort level with this process takes time, but if people stay with it, work with it and – most of all – enjoy it, the results will be worth it. Recommended Reading Stapezynski JS, Paris DM: "Conducting prehospital research." In Roush WR (Ed). Principles of EMS Systems: A Comprehensive Text for Physicians. Dallas; American College of Emergency Physicians, 1990. This article was reprinted from JEMS, March (Supplement) 1993.
(INSERT) Reading Smart: Discovering What the Data Do and Don’t Say By Elizabeth A. Criss, RN, MEd When a commercial says "four out of five" people agree, what does that mean? The advertiser is hoping you think it means 80 percent of all people support that particular product or idea. But couldn’t it also mean something else? For instance, what if they had only asked five people for their opinions or mailed out only 10 surveys and received five responses – four people for, one against. There are many other possible combinations that could produce these numbers and still not represent 80 percent of the population. Is this wrong? It’s hard to say. The best response is probably that results, like beauty, are in the eyes of the beholder. That’s all well and good for TV commercials, but this same "data torturing" can occur in medical research. Raw data generated by a project really doesn’t mean anything until it’s analyzed, and the tools used to analyze this information and the way data is compared determine what conclusions can be drawn. That can leave a lot of room for interpretation. Let’s say you’re interested in finding the latest research on the pneumatic anti-shock garment (PASG). Flipping through the journals, you find a study evaluating the effect of PASGs on nontrauma patients. The abstract states this is a prospective study done on 300 patients during a 12-month period. The findings of the study indicate that PASGs are of little value in the treatment of nontrauma patients in the prehospital environment.
Intrigued by these findings, you read the article. The results section describes the 300 patients. You note that the study divided the patients into two groups: blood pressure (BP) > 60 mmHg and BP < 60 mmHg. To assist in understanding the results, the authors include Tables 1 through 3. Moving on to the discussion, you note the authors’ conclusion: "For the majority of nontraumatic patients, the PASG is not beneficial and possibly increases mortality." To support their conclusion, there is a more lengthy and detailed explanation than you found in the abstract. Looking back over the information in Tables 1 and 2, you believe this to be a reasonable conclusion. But what about Table 3? Didn’t it demonstrate that PASG use in these patients reduced mortality? It did, but the authors’ conclusions are still valid. It’s important to note that the authors said "in the majority of "patients," not that the results applied to all patients. So, why didn’t the authors make more reference to the group in Table 3? Table 3 highlights a subgroup, patients, with a BP of < 60 mmHg that was positively affected by PASG use. Sometimes groups like this are left out due to the small number of patients in the subgroup; a small sample size does not allow the authors to calculate meaningful statistics or draw any significant conclusions. Without statistics, the most the authors can do is discuss the result as a possible trend. Nevertheless, the authors should at least mention this group as a potential area for future research. Another possibility for leaving subgroups out of a discussion is that they did not support the author’s original hypothesis. Although not entirely ethical, this has been done. The point of all this is that it is important to understand that data can be manipulated. Researchers will sometimes drop patients who don’t fit the desired hypothesis or support a certain position. It is important for you, the reader, to scrutinize the literature and account for all the patients. If the authors say "majority," instead of "all," find out where the rest of the population went. Be suspicious. Ask yourself if these patients were deliberately left out, or if the sample was just too small to be meaningful. Most of the research published today is well-controlled and scrutinized by professional review panels. However, it doesn’t hurt to become critical reader and ask questions. Elizabeth Criss, RN, MEd, serves on the Prehospital Care Research Forum Board of Advisers, is a senior research associate at the University of Arizona in Tucson and a base hospital coordinator at University medical Center in Tucson. This article was reprinted from JEMS, March 1994. Table 1 All Study Participants
Table 2 Patients with BP>60 mmHg
Table 3 Patients with BP<60 mmHg
|
|