2 Table of Contents NATURE OF THE PROBLEM.......................................................................................................5 Literature Review ........................................................................................................................7 Medical Education Process ..................................................................................................7 Supplemental Materials Available .......................................................................................9 Accessibility of Medical Information ................................................................................10 Confidence in Institutional Education ...............................................................................12 Competition in Medical Education ....................................................................................13 Stress Experienced by Medical Students ...........................................................................14 Summary ..................................................................................................................................16 PURPOSE .....................................................................................................................................18 METHOD .....................................................................................................................................19 Participants .........................................................................................................................19 Instrumentation ..................................................................................................................20 Procedure ...........................................................................................................................21 Data Analysis .....................................................................................................................21 Biases ................................................................................................................................22 RESULTS & DISCUSSION..........................................................................................................22 LIMITATIONS & RECOMMENDATIONS ................................................................................31 REFERENCES ..............................................................................................................................32 APPENDICES ...............................................................................................................................37 Appendix A ...............................................................................................................................37 Appendix B ...............................................................................................................................40 3 List of Tables TABLE 1.0 ....................................................................................................................................23 TABLE 1.1 ....................................................................................................................................24 TABLE 1.2 ....................................................................................................................................25 TABLE 2.0 ....................................................................................................................................28 TABLE 3.0 ....................................................................................................................................29 4 Abstract Medical students and medical education institutions both put a significant amount of effort toward the completion of a traditional medical education. Research and analysis of student study habits regarding the use of supplemental study materials not suggested or provided by their medical education institution revealed that the majority of students do utilize this material. Not only do many students utilize the material, but they pay for the materials and use them on a daily basis. To set the foundation for future research into student study habits regarding these materials, this study was completed by surveying second-, third-, and fourth-year medical students. In addition to surveying students about their use of supplemental materials, students were also surveyed regarding several potential risk factors that may be driving students to use these supplemental materials. Results indicated that students utilize supplemental materials for the sake of efficiency, the use of memory devices and practice question banks, finding material that fit their learning style, and more. As the educational institution furthers their understanding of students’ study habits and needs, they will be better positioned to cater to those needs. 5 NATURE OF THE PROBLEM According to the Association of American Medical Colleges (AAMC), the average in-state tuition at public medical schools during the 2018-2019 academic year was $31,905. This figure does not include student fees, mandatory health insurance, books, licensing fees, and other hidden costs that medical students must pay during their four years of medical education. Mowery (2015) stated that the median cost of completing 4 years of medical school at a public institution was $226,447 for the class of 2015. In just a few years, that figure has grown to about $300,000 (Asch, Grischkan, & Nicholson, 2020). Given that students are expected to incur debt of roughly $200,000 on average for their medical education, it is reasonable that they would expect to be sufficiently prepared for the next steps in their career (typically a residency) following the completion of their Doctor of Medicine. Medical students not only expect to be prepared for their residencies, but they also expect to be competitive applicants. Some of the most highly considered factors that contribute to the competitiveness of an applicant for residency include licensing exam scores, grades in courses related to the residency specialty, and class rank. In working toward becoming a competitive and sufficiently prepared applicant, students do not always feel that their medical education provided by the institution is sufficient on its own. Though medical educators work hard to prepare students for in-class and licensing exams, some students feel that they must use additional resources to supplement their study materials to achieve the highest score possible or to have an “edge” over the competition of class rank. Students may also feel that the resources that are readily available to them through their medical education institution do not fit their learning style. Each of these factors may be leading to medical students 6 spending extra time as well as extra money on supplemental materials that have not been suggested or provided by their medical education institution. Medical education institutions put a significant amount of resources into providing supplemental materials for their students to reduce the overall cost and improve the quality of medical education for their students. In addition to providing supplemental materials, medical schools have begun to upload lectures and other resources online for students to access outside of class for additional study time. Despite this effort being made by educational institutions, students are still putting a significant amount of time, money, and effort into the use of supplemental materials not suggested or provided by their medical education institution. This behavior could potentially be impacting the education and future abilities of medical students. In the creation of unnecessary expenditures, many students who rely on loans are pushing themselves further into debt by purchasing or licensing these supplemental materials on a regular basis throughout their educational tenure. In spending time studying these supplemental materials, medical students are reducing the amount of time spent on valuable study materials meticulously selected by the knowledgeable faculty and staff members of their medical education institution. Medical students may also be compromising their education by studying less reputable resources online. Though the Internet can be extremely beneficial in finding information, that information is not always reliable (Alsafi, Kaya, Patel, & Hamady, 2013). Research should be done to determine the extent and cause of this use of supplemental material as it may be a symptom of a larger systemic problem in the medical education field. Research would better situate medical education professionals to provide the most useful materials for students via their health sciences libraries, reducing the cost of medical education for students and ensuring quality materials for study. 7 Literature Review Many studies regarding learning style and supplemental material use by students have been completed, but few have explored students’ use of materials that have not been suggested or provided by their institution. Medical education institutions generally have extensive libraries that provide study materials for their students, including licensing question banks, exam preparation materials, and more (O’Hanlon & Laynor, 2019). Despite the availability of these materials to students at their health sciences library, many students are still utilizing supplemental materials that have not been provided or suggested by their medical education institution to study for exams. This literature review will explore the medical education process, supplemental materials available to students, and the accessibility of medical information. It will then review factors in students’ confidence in their medical education institution, levels of competitiveness in their institution, and stress experienced by students during their medical education. Medical Education Process Completion of a Doctor of Medicine degree is perceived as an arduous, expensive, and extremely rewarding process. In the United States, there are more than 140 medical education institutions in addition to thousands of graduate training programs (Dezee, Artino, Elnicki, Hemmer, & Durning, 2012). The standard educational process for becoming a licensed physician in the United States requires a minimum of 7 to 11 years of training (including a bachelor’s degree, M.D., etc.), numerous standardized examinations, and completion of at least one residency program. Traditional undergraduate medical education requires the completion of a four-year program following the completion of a bachelor’s degree. This four-year medical education program consists of two years of preclinical basic science education and two years of clinical 8 clerkships (Mowery, 2015). According to Dezee, Artino, Elnicki, Hemmer, and Durning (2012), medical students in the United States receive direct instruction for an average of 21 hours per week during the first two years of their medical education. During students’ fourth year in medical school, they will begin applying for residency programs. Medical students must also complete three United States Medical Licensing Examination (USMLE) “step” exams in order to become licensed practicing physicians. The Step 1 and Step 2 examinations are completed during medical school, and the Step 3 examination is completed during the residency program. Giordano, Hutchinson, and Peppler (2016) reported that students spent a significant amount of time studying for the USMLE Step 1 examination alone, not including the time students spend studying for specific courses as they continue their medical education while preparing to take their USMLE examinations. This shows the amount of time that students are willing to dedicate toward not only succeeding but excelling in their medical education. Not only do students put a large amount of time into their medical education, but they also spend a large amount of money on their medical education. Since the 1960s, the cost of medical education has increased by 750%-- from about $40,000 to about $300,000 (Asch, Grischkan, & Nicholson, 2020). In addition to paying basic tuition and fees for their medical education program, students must also pay application fees, purchase books, pay for certification and licensure exams, and lose a significant amount of potential income with uncompensated training. Medical students, like most students in any program of study, have a reasonable expectation upon entrance into a medical education institution that their program will sufficiently prepare them for the next step in their career. Medical education institutions do have and strive to meet their responsibility to sufficiently prepare their students, however, Morrow et al. (2012) 9 explained that some medical students may have the unreasonable expectation that they will be completely prepared to step into the role of a resident upon the completion of their M.D. The medical education experience does not stop at the end of the fourth year of medical school, and some students will not be fully confident in their own experience and education until they have had the opportunity to work, and truly succeed, in residency programs (Morrow et al., 2012). Supplemental Materials Available Medical education institutions typically have thousands of resources available for students to utilize for supplementing their traditional education. This can include “LibGuides” created by medical education librarians as guides to study resources available in the library. Despite efforts made to advertise the resources available, students may not be utilizing the resources provided at no cost as much as librarians and other medical educators would prefer (O’Hanlon & Laynor, 2019). Additionally, as the world of proprietary medical education study materials expands, librarians have been running into issues with licensing for some study resources-- some companies have study resources available for costs that are too high to be feasibly affordable for medical education institutions while some others do not allow institutions to purchase subscriptions for their students at all. A major goal for medical education librarians is to make as much helpful information as accessible as possible to students, and the lack of accessibility of some proprietary study resources can be an obstacle to providing what their students want or prefer for studying (O’Hanlon & Laynor, 2019). O’Hanlon and Laynor (2019) listed some of the available proprietary study resources frequently used by medical students to supplement their education that were not provided by their medical education institution, though whether these materials are suggested by medical education 10 institutions will certainly vary between institutions and is not addressed by the article. Some of these proprietary resources are: ● Lecturio (not available at institutional pricing) ● SketchyMedical ● Pathoma (not available at institutional pricing) ● Picmonic (not available at institutional pricing) ● Physeo ● Doctors in Training (not available at institutional pricing) The above listed resources and many more are available for use in studying for the USMLE Step 1 and Step 2 exams, class-specific information, and more. Medical students are already spending a substantial amount of money on their institutional medical education, but are willing to pay even more for these supplemental resources. For example, the resource SketchyMedical is available at the cost of $229.99 for 6 months or $369.99 for 12 months, while the resource Doctors in Training can range in cost anywhere from $250.00 to $825.00. Though some resources have free versions, these resources do require a subscription to gain access to the full list of features. Librarians have suggested that a possible explanation for students’ willingness to spend extra money on these supplemental resources may be that medical education libraries are not providing students with the types and styles of study materials that they want to utilize to supplement their education. Though students will need continued prep materials through residency to continue studying for future USMLE Step exams, the overall burden of finding and paying for supplemental materials could be reduced through their medical education institution during their traditional education process. O’Hanlon and Laynor (2019) suggested that students would not resort to the 11 use of unsuggested, unprovided, and costly materials if the libraries were providing the materials that students need at no cost and if students were aware that these materials are being offered. Accessibility of Medical Information Before the Internet was widely used, access to medical information was dependent upon published books and journals. Within the last two decades, the Internet has become a part of the daily life of the average first-world citizen. With this change has come access to a plethora of information, including medical information. The number of medical journals available online rose from just 35 to over 4,000 in the years between 1998 and 2001 (Tao, Demiris, Graves, & Sievert, 2003). The switch to a digital medical information age has introduced “unprecedented access to biomedical knowledge resources” (Friedman, Donaldson, & Vantsevich, 2016, p. 504). This influx of medical information through technology has made changes in medical education curriculum a necessity. Medical educators must now decide what information medical students must have memorized for daily use and what information would better serve a future physician when accessed through technology (Ward et al., 2001). Understanding how to access and utilize that medical information has also become an important part of the medical education curriculum. More than ever, medical educators must prepare medical students for managing the abundance of information that is available at their fingertips (Ward et al., 2001). Students who entered medical school in 2015 entered a completely or almost completely digitized medical practice upon their graduation, and it is the responsibility of medical educators to foresee the necessary skills required for that digital world (Friedman et al., 2016). Medical educators must now assist their students in becoming competent in judging their own personal knowledge base in order to assess the necessary use of information technology. 12 Medical educators must also assist their students in understanding how to best use information technology and interpret the results of searches (Friedman et al., 2016). In 2011, up to 75 percent of adults in the United States used the Internet to access medical information (Alsafi, Kaya, Patel, & Hamady, 2013). Though the Internet can be extremely valuable in finding information, that information is not always reliable and can sometimes even be harmful (Alsafi et al., 2013). During the period between 1991 and 2001, the number of students entering the on-campus health science library at the University of Missouri-Columbia dropped dramatically, which suggested that students were using the Internet rather than the library to access information (Thao et al., 2003). As medical students supplement their education with medical information on the Internet, they should be well-prepared by their institution for sifting through the information for accuracy and reliability. Alsafi et al. (2013) found that an average of only 54.5 percent of the top 50 search results regarding interventional radiology, vascular surgery, and cardiology were deemed “reliable” by the Minerva validation instrument, also known as the LIDA instrument, a validation tool used to assess healthcare information websites. As the Internet continues to become more popular for finding information, medical students may be using this to supplement their education without the guidance of their medical education institution. Recently, proprietary study resources pertinent to medical education have become more accessible, leading students to spend valuable study time on these resources rather than those that have been provided or suggested by their medical education institution (O’Hanlon & Laynor, 2019). The danger of students’ use of materials that have not been provided or suggested by their medical education institution is that they may lay an unstable educational foundation on inaccurate information or may apply accurate information incorrectly to their studies. With an accurate 13 picture of students’ study needs, medical education institutions may be better equipped to provide materials that students need while also assessing for accuracy and reliability. Confidence in Institutional Education This self-supplementation of medical students’ education may be due to a lack of confidence in their institutional education. In a study completed by Lempp and Seale (2004), over 60 percent of medical students reported that they considered their educators to be teaching in a ‘haphazard’ manner, describing some educators as portraying a lack of commitment and poor teaching skills. Dahlin et al. (2005) found that an average of 80 percent of medical students in their first and third year of medical school felt that they were not receiving enough feedback from their educators. These reports of medical students’ lack of confidence in the education that they are receiving may be leading them to turn to the Internet for supplementation of their studies. Some students are even beginning to question the necessity of a traditional style of medical education (O’Hanlon & Laynor, 2019). Another factor affecting the confidence that medical students have in their medical education institution is the perception of the competence of and effort put forth by medical education faculty. In recent years, universities have encouraged educators to increase the amount of research completed. Some universities have even made the completion of research a requirement for continued educational employment. As this change has been happening, the number of medical students enrolled has also been rising (Parry et al., 2008). With research demanding much of educators’ time, the increasing number of students has also contributed to the decreased amount of interaction that educators spend on each student. Parry et al. (2008) suggested that these conflicting demands on the time of educators may be causing the confidence of medical students in their institution to suffer. 14 Competition in Medical Education Following the four years of traditional medical school, students enter residency in a specific specialty to complete their education by learning more in-depth information and practical knowledge about that specialty. Students apply to their preferred residency programs during their fourth year of medical school and go through a competitive process in order to be matched with a residency program. Residency programs tend to be more or less competitive depending on the specialty. For example, 91 percent of applicants were accepted into otolaryngology residency programs in 2018, while only 8 percent of applicants were accepted into preventative medicine residency programs the same year (2018 Main Residency Match, Results and Data, 2018). Different residency programs take different criteria into account when choosing students for their program. Surveyed residency program directors reported that some of the most important criteria were number of honors grades, class rank, and academic awards (Green et al., 2009). In order to meet these criteria and be selected for their choice residency specialty, medical students may feel that they must outshine their peers and go above and beyond to impress their educators and administrators. This attitude creates a competitive atmosphere in medical education institutions. Students interviewed by Lempp and Seale (2004) reported that “competition rather than cooperation is the defining characteristic of medicine” (p.772). In the same study, students interviewed by the researchers often made subtle comments about advantages that they had over other students, including mentions of previously earned degrees or professional qualifications (Lempp & Seale, 2004). Though competition can benefit students in their medical education by serving as motivation, Van Nuland et al. (2015) explained that competition can cause an increase in student anxiety and divide their attention. In cultivating the atmosphere of a medical education institution, 15 faculty and staff must take into consideration how competitiveness may influence students’ behavior, including potentially increasing the amount of supplemental materials utilized by students in study. Stress Experienced by Medical Students Though medical education institutions work hard to provide wellness programs and interventions for their students, medical students experience a significant amount of stress. In a recent study reviewed by Jordan et al. (2020), 57.7% of medical students surveyed reported high levels of fatigue, and 44.6% scored high on the burnout index. In completing the requirements to become licensed practicing physicians, students must cope with a significant amount of stress due to the pressure to constantly perform well. How students perform in the four years of their traditional medical school education will significantly affect their postgraduate prospects for residency and job placement. USMLE Step 2 scores were listed as the fifth most important criteria considered when determining whether or not to accept a student into a residency program (Green et al., 2009). Certain specialties’ residency programs are more demanding regarding qualifications than other specialties, including accepting applicants on average with a higher USMLE score (Gauer & Jackson, 2017). These data suggest that those residency programs are more competitive than others, putting pressure on students to perform better than their peers or watch their residency program options dwindle. Gauer and Jackson (2017) found a significant relationship between USMLE scores on the first two step exams and the specialty of residency programs for students. Students with lower USMLE scores were more likely to be accepted into specialties such as family medicine or psychiatry (Gauer & Jackson, 2017). For this reason, students may feel pressured to go above and beyond the call of traditional education in order to score higher on each of the USMLE exams and gain acceptance into their first-choice residency program, particularly if their 16 first-choice residency program has traditionally accepted students with higher USMLE scores. The drive to score as high as possible on the USMLE examinations puts pressure on students to find the best possible study resources, adding to existing pressure and creating a significant amount of stress (O’Hanlon & Laynor, 2019). Additionally, medical students feel general pressure regarding performance during their medical education experience. Medical students experience a higher level of stress than the general population (Firth, 1986; Reddy et al., 2016). Aktekin et al. (2001) found that medical students presented prevalence of emotional disturbance at a higher rate than students in other areas of study. An average of 79% of first- and third-year medical students at a medical education institution felt that studies controlled their life (Dahlin et al., 2005). A higher rate of medical students experienced emotional exhaustion, suicidal ideation, and depression in comparison to their residents and early-career physician counterparts (Dyrbye et al., 2014). Some of this stress may be attributed to students adjusting to the rigorous curriculum of medical education institutions (Dahlin et al., 2005). The first two years of medical education are focused on learning information that will assist students in their transition to the clinical phase of their education. Students may feel overwhelmed at the amount of information that they feel they must learn in order to adequately prepare them for clinical experiences, leading students to supplement their education outside of institutional direction in order to ensure that they are learning as much as possible. With the scores and grades received on their in-class and USMLE examinations having such a significant impact on their future, students’ stress and anxiety levels regarding this subject may be a driving factor for their use of supplemental materials not suggested or provided by their educational institution. An increase in student loans driven by the high price of materials not 17 provided by their medical education institution may also be increasing the stress levels of medical students. Summary Being aware of the study methods of their students will allow medical education faculty and staff to have a better understanding of where medical education should improve in the future. Medical educators are already interested in and aware of students’ use of supplemental materials that have not been provided or suggested by the medical education institution, but formal research will solidify the foundation of their knowledge and provide a base for future research. This study will allow the medical education institution to understand roughly how many of their students use supplemental materials, how frequently they use these materials, which materials they use, and how much their students spend on these materials. Additionally, this study will address the different possible factors that may correlate with students’ use of these supplemental materials. This knowledge can be useful in the future addressing students’ concerns leading to their use of supplemental materials as well as allowing library staff to better understand which materials students are currently utilizing. Though a significant number of the most frequently used supplemental materials are not available through institutional licensing, the few that are available for institutional licensing may be worth the expenditure to the university to reduce students’ overall cost of medical education. The only people who know what it is like to go through medical school in the current educational climate are the people who are currently doing it. The best people to tell us why students are using these materials, which materials they are using, and what is affecting their medical school experience are the students. 18 PURPOSE The purpose of this study is to understand the driving force behind medical students’ use of supplementary study materials not suggested or provided by their educational institution. This study seeks to determine if students use supplemental materials, why they do so, how much money and time is dedicated to these supplemental materials, and which supplemental materials they are using. This research answers the following questions: 1. Do medical students use supplemental materials not suggested or provided by their medical education institution? 2. How often do medical students use supplemental materials not suggested or provided by their medical education institution? 3. Do medical students spend money on supplemental materials not suggested or provided by their medical education institution? If so, how much do they spend per semester? 4. Why do medical students use supplemental materials not suggested or provided by their medical education institution? 5. Which supplemental materials do medical students use that are not suggested or provided by their medical education institution? The information gathered by this study will be distributed to the University of Utah, School of Medicine Curriculum Committee as well as the Spencer S. Eccles Health Sciences Library staff with the purpose of assisting staff in understanding the current study habits of the current cohort of students in an effort to improve medical education in the future. 19 METHOD Research Type To address the purpose of this study, electronic surveys were conducted between Friday, July 9th, 2021, and Friday, July 23rd, 2021. Surveys were used to gather both quantitative and qualitative data regarding students’ use of outside resources to supplement their studies as well as students’ perceived levels of confidence in their institution, perception of institutional competitiveness, and stress experienced. Participants Participants for this study were current students of the University of Utah, School of Medicine, based on convenience sampling. All currently enrolled students completing their second, third, and fourth year of medical school (Class of 2022, Class of 2023, and Class of 2024, respectively) were identified for potential participation in the research. This particular set of students was chosen because they have completed at least one full year of their medical education and therefore are in the preparation stages for at least one USMLE Step exam. To protect anonymity, students’ demographic information was not collected beyond their current year in their medical education as of Fall 2021. As of end-of-term Spring 2021, the University of Utah School of Medicine M.D. program consisted of 479 students. The demographic makeup of the entire program was as follows: - Female: 46%; Male: 54% - Domestic Students of Color: 19%; White: 78%; Unknown: 2% This data set does not include in the incoming class for Fall 2021, therefore, the incoming class was not included in the survey regarding supplemental materials. However, due to the aggregated nature of this data, the demographic data set includes the Class of 2021 who were at 20 the end of their program and not included in the survey regarding supplemental materials. Assuming that each of the four classes are roughly the same size, the survey invitation was sent to over 300 students that make up the second-, third-, and fourth-year classes for the School of Medicine in the Fall 2021 semester. Instrumentation A researcher-developed main survey was created to gather data regarding students’ use of supplemental materials not suggested or provided by their medical education institution, with specific questions addressing level of usage, frequency used, and money spent on these materials. In addition to questions about students’ use of supplemental materials, the survey explored students’ levels of stress, perception of competitiveness, and confidence in their medical education institution. In order to protect students’ anonymity, the only demographic question included on the survey was the students’ current year in medical education beginning in Fall 2021. The survey was created using the Qualtrics survey tool (see Appendix A). Following review of the survey by the research committee and Institutional Review Board (IRB) approval, the survey was distributed to the students through email using a list-serve provided by the students’ medical education institution. The email that was sent out contained an introduction to the survey as well as a link to the Qualtrics survey tool. The main survey consisted of ten questions and had an estimated response time of two minutes. Throughout the survey, students were asked a series of questions regarding their use of outside resources to supplement their education, their level of confidence in their medical education institution, the perceived amount of competitiveness in the medical education institution, and the students’ stress level. Students who indicated that they do not use supplemental materials were routed directly to questions about their level of confidence in their medical education institution, the perceived amount of competitiveness in the medical 21 education institution, and the student’s stress level. Likert scale questions were converted to multiple choice questions to improve the level of accessibility of the survey. The final version of the questionnaire can be found in Appendix A. Procedure The approved survey was emailed to students who were identified for the study. The survey was open for a two-week period beginning on Friday, July 9th, 2021. Students were asked to fill out the 11-question, two-minute survey by 12:00PM on Friday, July 23rd, 2021. A reminder email was sent on Wednesday, July 21st, 2021. Students participating in the survey were asked to complete a letter of consent at the beginning of the survey. Students who did not give consent were directed to the end of the survey and given a thank you message. No data was collected for these students. Data Analysis The data gathered from this survey were intended to be used to compare the stress, competitiveness, and confidence levels between students who did and do not use supplemental materials, as well as provide a needs analysis for the University of Utah, School of Medicine regarding their students’ study habits. Both quantitative and qualitative methods were used to analyze data from the survey. Data were analyzed using descriptive statistics, cohort analysis, and aspect-based sentiment analysis. Data was exported from Qualtrics Survey Tool to a CSV file for analysis in Excel to compare data between submissions. Analysis was performed to determine how many students used supplemental materials, how often they used supplemental materials, how much money students spent on supplemental materials, and which supplemental materials students used. Cohort analysis was used to determine students’ average levels of stress, confidence in their institution, and perceived level of competitiveness in their institution. Descriptive and thematic 22 analysis was used to analyze students’ responses to open-ended questions regarding the reasons behind their use of supplemental materials. Data received from students who do not report using supplemental materials was also compared to data received from students who do report using supplemental materials as much as was possible with the small sample size. Incomplete responses were removed from the data set. Biases The researcher was employed in an administrative capacity at a medical education institution for one year. During that time, administrators had discussions regarding students’ use of supplemental resources that had not been suggested or provided by the medical education institution. RESULTS & DISCUSSION Providing medical educators with a deeper understanding of their students’ study habits will allow them to improve the services that they provide to their students. Survey-based research was conducted to examine students’ use of supplemental materials, the details of that use, and some possible risk factors that may be affecting students’ use of supplemental materials. Data collected through the survey were examined to determine the answer to each of the research questions driving the purpose of this study. A total of 147 students responded to the survey before the stated deadline of July 23rd, 2021 at 12:00PM MST. Of the 147 respondents, 122 students completed the survey by answering each question required of them. Incomplete responses were excluded from analysis. Though the initial aim of the study was to compare the stress levels, perception of competitiveness, and confidence in institutions between students who did use supplemental materials not suggested or 23 provided by their medical education institution, this was not feasible based on feedback of the survey. However, all research questions were sufficiently answered. Use of Supplemental Materials Of respondents, 97.5% of students indicated that they do use supplemental materials not suggested or provided by their medical education institution for in-class exams, USMLE exams, or for both types of exams, sufficiently addressing the first research question of this study (see Table 1.0). Students at the University of Utah School of Medicine do, overwhelmingly, utilize supplemental materials that are not suggested or provided by their medical education institution. Over ninety-three percent of respondents indicate use of supplemental materials for both types of exams. Responses do not vary significantly between class-levels regarding use of supplemental materials. Students who indicated that they do not use supplemental materials were routed directly to questions regarding potential risk factors. Answering this question is the first step to assisting medical educators with understanding the study habits of their students and how they can better serve their students in the future. The high percentage of responding students indicating that they do use supplemental materials not suggested or provided by their medical education institution lays the foundation for further research into this phenomenon. As students spend a significant amount of money on their medical education and are highly selective of the educational institutions they apply for, the fact that they are regularly using materials that have not been sanctioned or provided by their traditional education is something that medical educators need to explore. 24 Table 1.0 Do medical students use supplemental materials not suggested or provided by their medical education institution? 2nd Year 3rd Year 4th Year No 1 1 1 Yes, only for USMLE exams 2 1 1 Yes, only for in-class exams 0 0 1 Yes, for both types of exams 48 36 30 Regarding the second research question, this study sought to address the frequency of which students use supplemental materials. The majority of responding students indicated that they use supplemental materials that are not suggested or provided by their institution on a daily basis. Just under 30 percent of respondents selected a frequency of use option less than ‘Daily.’ None of the respondents indicated that they only used these supplemental materials once a semester, and three respondents indicated that they used the supplemental materials once a month, the lowest frequency available after once a semester. One hundred and ten out of the 122 respondents selected the two highest frequencies available -- a few times a week and daily, with the results skewing heavily toward daily use. These data suggest that students do dedicate a significant amount of time toward the study of these materials that have not been suggested or provided by their medical education institution. As this intimates an impact on the amount of time students are spending on studying materials that are suggested or provided by their educational institution, understanding which materials their students are relying on is vital for educational professionals. Responses suggest that the listed supplemental materials are some of the primary study materials being utilized by medical students. An understanding of these study habits and the materials that students are relying on may affect the way a medical education institution operates in the future. If educators are familiar with the materials that their students 25 are digesting to supplement their materials, they may be better equipped to integrate these materials into the curriculum or provide students with the supplemental materials that best integrate into the material students are covering in their classes. Table 1.1 How often do medical students use supplemental materials not suggested or provided by their medical education institution? 2nd Year 3rd Year 4th Year Once a semester 0 0 0 Once a month 2 0 1 A few times a month 2 1 0 Once a week 1 2 0 A few times a week 15 4 8 Daily 30 30 23 For the third research question, this study sought to determine how much money students were spending on these supplemental materials. The initial and outright costs of obtaining a medical education are already significant, considering application fees, basic tuition, fees, and the purchase of required textbooks and materials. Understanding the hidden costs of medical education born by the students in their private study habits is vital to understanding the true cost of medical education. Additionally, medical education institutions may be more appealing to potential candidates for admission if the institution is willing to review students’ study habits and ideally provide the frequently used supplemental materials at no cost to their students through the Health Sciences Library or directly through the curriculum. 26 Table 1.2 Do medical students spend money on supplemental materials not suggested or provided by their medical education institution? If so, how much do they spend per semester? 2nd Year 3rd Year 4th Year None, I only use free materials 1 1 0 $1 - $100 4 0 3 $101 - $200 13 5 7 $201 - 300 14 7 10 $301+ 18 24 12 Continuing to address students’ use of supplemental materials, the study requested that students address the reason behind their use of supplemental materials. Each of the 122 students provided some kind of answer to this question, with the majority of the students responding with detail. The researcher was able to group together the responses into groups of significance. The answer most frequently given in response to the question of why students use supplemental materials not suggested or provided by their medical education institution was simple: efficiency. Almost 44 percent of respondents who indicated using supplemental materials not suggested or provided by their educational institution mentioned the efficiency and format of these materials in their reasons for using the materials. Medical students feel that the information given during lectures can lack efficiency in deliverance, and these supplemental materials fit the format that they are looking for. Respondents indicated that these materials were formatted in a more condensed manner and allowed students to learn more information in a shorter amount of time than the traditional materials that they had been assigned in their classes. The second-most frequently mentioned reason was memory devices and practice questions offered by these supplemental materials. Students may not be aware of the practice question banks available to them through their medical education institution, or they may feel 27 that these question banks are insufficient for their needs. The third and fourth-most frequently cited reasons for using supplemental materials not suggested or provided by their institution were learning style and clarification or continuity of material. Respondents suggested that they were more easily able to digest the information when it was specifically catered to their learning style, and many specifically mentioned that the lecture format was not productive for someone of their learning style. Regarding clarification or continuity of material, respondents said that the materials were useful for gaining another perspective on material that they had already learned through their traditional classrooms. Respondents also suggested that this other perspective was helpful in connecting material and information that may otherwise seem unconnected between different lecturers in their traditional education. Respondents explained that there was some information for USMLE step exams that was not and would not be addressed in their traditional classroom, and they felt that they could fill in those gaps of knowledge using supplemental materials prior to taking the exam. Some students expressed their need for tailor-made materials regarding the USMLE exams and felt that they could not get the study material necessary through their medical education institution. Others felt that the information provided by their medical education institution or information provided by the assigned materials was out-of-date. The last significant driving forces behind students’ use of supplemental materials were suggestions from peers or mentors, establishing a basic knowledge foundation not gained through traditional education, and access. Regarding access, students felt that the ability to access information at any time was most convenient for them. This may be a significant factor in students’ study habits leaning toward supplemental materials not suggested or provided by the 28 institution rather than utilizing the materials that are provided through the curriculum or the Health Sciences Library. Addressing the final research question, the survey requested that students list the supplemental materials that they use that are not suggested or provided by their medical education institution. A few of the supplemental materials were mentioned by a significant number of students. These materials are Sketchy, mentioned by 87 students, Pathoma, mentioned by 65 students, UWorld, mentioned by 51 students, and USMLE Rx, mentioned by 48 students. Other supplemental materials listed by students were, in order of most to least mentioned, First Aid, Physeo, Online MedEd, Osmosis, Kaplan, Pixorize, NBME self-assessments, ZygMed, Board Review Course books, Costanzo Physiology, Dr. High Yield notes, Firecracker, Doctors in Training, Goljan Lectures, Khan Academy, Picmonic, Up to Date, Washington University in St. Louis Manuals, and White Coat Coaching. Respondents also listed YouTube channels, podcasts, and Google as sources of study information. One respondent declined to list supplemental materials used. While some of the more frequently used supplemental materials do not have institutional licensing available, this information can still be valuable to medical educators to assist them in understanding the material that their students are utilizing to supplement their education. With an understanding of these study habits, those who work directly with the curriculum may be able to integrate some parts of these study materials into the classroom to create a seamless medical education experience. Other materials that were listed are available for institutional licensing and could be integrated into the Health Sciences Library. 29 Potential Risk Factors In an attempt to gather data that may support potential causes of students’ use of supplemental materials, the researcher included survey questions regarding students’ level of stress, confidence in their medical education institution, and perceived level of competitiveness. The intention behind collecting risk factor data was to compare the data between students who did use supplemental materials and students who do not use supplemental materials. However, the number of students who do not use supplemental materials was so sparse that it is impossible to draw a conclusion from the comparison of the data. Despite the intention of the collection of this data not being met, the researcher has included the data as it may be useful to those who review this study in the future (see Table 2.0). The risk factors have also been compared using cohort analysis (see Table 3.0). Table 2.0 Do you use supplemental materials not suggested or provided by your institution? Average level of confidence in institution re: USMLE exams Average level of confidence in institution re: residency Average level of competitiveness Average level of stress Do not use (3) 5.333 7 6 6.667 Yes, for USMLE (4) 4.75 7.75 6.25 7 Yes, for in-class exams (1) 5 7 8 7 Yes, for both (114) 6.72 7.66 5.04 6.84 30 Table 3.0 Year in School Regardless of Use of Supplemental Materials Average level of confidence in institution re: USMLE exams Average level of confidence in institution re: residency Average level of competitiveness Average level of stress 2nd Year (51) 6.88 7.35 5.04 6.86 3rd Year (38) 6.5 7.89 5.32 6.61 4th Year (33) 6.3 7.79 5.03 7.09 This needs analysis study determined that the medical students at the University of Utah School of Medicine do frequently use supplemental materials not suggested or provided by their medical education institution. The students are spending a significant amount of time studying these materials, spending a significant amount of money per semester on these materials, and feel that they are necessary to supplement the traditional education that is provided by their medical education institution. Through continued research, the institution will be able to more clearly understand the study habits and needs of their students as they pursue their medical education. Continuing this research will also allow the institution to provide an even higher quality of education that may attract better candidates and propel the university’s reputation moving forward. 31 Limitations and Recommendations A limitation of the study was the scale used for questions regarding the level of competitiveness in the institution and the level of stress experienced by students. The scale 1-10 was used, however, many students may have different definitions of those numbers between 1 and 10, which were given specific definitions. The recommendation to be made regarding this limitation would be to reduce the scale and attach specific definitions of each option. An additional limitation of the study was the choice options offered to students regarding the amount of money that they spend per semester. With pricing of some supplemental study materials reaching almost $1,000, offering a max option of $301+ did not allow the researcher to accurately assess the amount of money that students are spending on their supplemental materials. A respondent asserted in the open-ended section of the survey that they had spent over $1,800 in the first two years of their medical education on supplemental materials alone, addressing the highest option for selection, “$301+” as inadequate. For future studies, the researcher would recommend asking how much students have spent on supplemental materials to date during their medical education, offering choices up to $10,000+. Recommendations resulting from the study include study material integration into the curriculum and increased promotion of provided materials to medical students. The data from this data suggest that students are using supplemental materials not suggested or provided by their medical education institution as a significant part of their studies. Understanding which materials they are using and what information is provided by those materials can assist professors and medical educators with better matching the material provided in class to the learning styles of their students, alleviating some of the need for those materials. The 32 recommended action is for faculty and relevant staff to review the materials that students are using and consider integrating the format or material into their traditional curriculum. Data from the survey also indicate that students are not fully aware of the materials that are provided to them by their Health Sciences library at no cost. While the university and School of Medicine put a significant amount of time, effort, and money into curating a specialized collection of study materials for their students, not all students are aware of this. This is specifically true for question banks and memory device materials. These materials are provided in some form through the Health Sciences library, but students may feel that they need to seek them out on their own if they are not aware that they are offered. The recommended action is to emphasize the marketing of the availability of these materials to medical students. Additionally, some of the materials listed by students are available for institutional licensing and may be beneficial to add to the Health Sciences’ collection of materials available through the School of Medicine. Further research should also be done regarding the nature of these supplemental materials and the motive behind companies offering “student-only” licenses rather than institutional licenses, forcing the cost upon the student. Additionally, future research should address students’ motives behind specifically choosing supplemental materials over those provided by the library. With a further in-depth look into competitiveness, stress, etc., there may be a connection between students’ drive to succeed above others in their class and their use of materials not readily available to the rest of their cohort. Lastly, USMLE Step 1 will only release Pass/Fail outcomes on and after administrations given on January 26, 2022. This may affect the results of similar research in the future. 33 References 2018 Main Residency Match, Results and Data. (2018). Retrieved from http://www.nrmp.org/wp-content/uploads/2018/04/Main-Match-Result-and-Data- 2018.pdf. Aktekin, M., Karaman, T., Senol, Y. Y., Erdem, S., Erengin, H., & Akaydin, M. (2001). Anxiety, depression and stressful life events among medical students: a prospective study in Antalya, Turkey. Medical Education, 35(1), 12-17. Alsafi, A., Kaya, G., Patel, H., & Hamady, M. S. (2013). A comparison of the quality of the information available on the Internet on interventional radiology, vascular surgery, and cardiology. Journal of Postgraduate Medicine, 59(1), 69-75. doi: 10.4103/0022- 3859.109509 Asch, D. A., Grischkan, J. & Nicholson, S. (2020). The cost, price, and debt of medical education. New England Journal of Medicine, 383(1), 6-9. doi: 10.1056/NEJMp1916528 Dahlin, M., Joneborg, N., & Runeson, B. (2005). Stress and depression among medical students: a cross-sectional study. Medical Education, 39(6), 594-604. doi: 10.1111/j.1365- 2929.2005.02176.x Dezee, K. J., Artino, A. R., Elnicki, D. M., Hemmer, P. A., & Durning, S. J. (2012). Medical education in the United States of America. Medical Teacher, 34. 521-525. Dyrbye, L. N., West, C. P., Satele, D., Boone, S., Tan, L., Sloan, J., & Shanafelt, T. D. (2014). Burnout among US medical students, residents, and early career physicians relative to the general US population. Academic Medicine, 89(3), 443-451. Firth, J. (1986). Levels and sources of stress in medical students. British Medical Journal (Clinical Research Ed.), 292(6529), 1177-1180. 34 Friedman, C. P., Donaldson, K. M., & Vantsevich, A. V. (2016). Educating medical students in the era of ubiquitous information. Medical Teacher, 38(5), 504-509. doi: 10.3109/0142159X.2016.1150990 Gauer, J. L., & Jackson, J. B. (2017). The association of USMLE Step 1 and Step 2 CK scores with residency match specialty and location. Medical Education Online, 22(1), N.PAG. doi: 10.1080/10872981.2017.1358579 Gavali, M. Y., Khismatrao, D. S., Gavali, Y. V., & Patil, K. B. (2017). Smartphone, the new learning aid amongst medical students. Journal of clinical and diagnostic research: JCDR, 11(5), JC05. Giordano, C., Hutchinson, D., & Peppler, R. (2016). A predictive model for USMLE Step 1 scores. Cureus, 8(9), e769. doi: 10.7759/cureus.769 Green, M., Jones, P., & Thomas, J. X., Jr. (2009). Selection criteria for residency: results of a national program directors survey. Academic Medicine, 84(3), 362-367. Jordan, R. K., Shah, S. S., Desai, H., Tripi, J., Mitchell, A., & Worth, R. G. (2020). Variation of stress levels, burnout, and resilience throughout the academic year in first-year medical students. PloS One, 15(10), e0240667. doi: 10.1371/journal.pone0240667 Lempp, H., & Seale, C. (2004). The hidden curriculum in undergraduate medical education: a qualitative study of medical students’ perceptions of teaching. BMJ, 329(7469), 770-773. doi: 10.1136/bmj.329.7469.770 Morrow, G., Johnson, N., Buford, B., Rothwell, C., Spencer, J., Peile, E., … Illing, J. (2012). Preparedness for practice: the perceptions of medical graduates and clinical teams. Medical Teacher, 34(2), 123-135. doi: 10.3109/0142159X.2012.643260 35 Mowery, Y. M. (2015). A primer on medical education in the United States through the lens of a current resident physician. Annals of translational medicine, 3(18), 270. doi: 10.3978/j.ssn.2305-5839.2015.10.19 O’Hanlon, R., & Laynor, G. (2019). Responding to a new generation of proprietary study resources in medical education. Journal of the Medical Library Association: JMLA, 107(2), 251-257. Parry, J., Mathers, H., Thomas, H., Lilford, R., Stevens, A., & Spurgeon, P. (2008). More students, less capacity? An assessment of the competing demands on academic medical staff. Medical Education, 42(12), 1155-1165. doi: 10.1111/j.1365-2923.2008.03234.x Reddy, E. A., Srijampana, V. V. G. R., Rajana, B., & Muddana, S. S. T. (2016). Perceived stress and prevalence of depression among first-year medical students. Journal of Dr. NTR University of Health Sciences, 4(4), 209-213. doi: 10.4103/2277-8632.171689 Scott, K., Morris, A., & Marais, B. (2018). Medical student use of digital learning resources. Clinical Teacher, 15(1), 29-33. doi: 10.1111/tct.12630 Tao, D., Demiris, G., Graves, R. S., & Sievert, M. (2003). Transition from in-library use of resources to outside library use: the impact of the Internet on information seeking behavior of medical students and faculty. In AMIA Annual Symposium Proceedings (Vol. 2003, p. 1027). American Medical Informatics Association. Tuition and student fees. (n.d.). Retrieved from aamc.org/data/tuitionandstudentfees/ Van Nuland, S. E., Roach, V. A., Wilson, T. D., & Belliveau, D. J. (2015). Head to head: The role of academic competition in undergraduate anatomical education. Anatomical Sciences Education, 8(5), 404-412. 36 Ward, J. T., Gordon, J., Field, M. J., & Lehmann, H. P. (2001). Communication and information technology in medical education. Lancet, 357(9258), 792-796. 37 Appendix A Questionnaire Instrument Use of Supplemental Materials by Medical Students Survey Question 1: Please select your level in medical education (Fall 2021): __2nd Year __3rd Year __4th Year Question 2: When studying for class-specific AND/OR USMLE exams, do you use supplemental materials not suggested or provided by your medical education institution? __Yes, only for USMLE exams __Yes, only for class-specific exams __Yes, for both types of exams __No ***Students who answer “No” to Question 2 are directly routed to Question 7*** Question 3: How often do you use supplemental materials not suggested or provided by your medical education institution? __Once a semester __Once a month __A few times a month __Once a week __A few times a week __Daily Question 4: How much money do you spend PER SEMESTER on supplemental materials not suggested or provided by your medical education institution? __None, I only use free materials 38 __$1 - $100 __$101 - $200 __$201 - $300 __$301+ Question 5: Please list any supplemental materials not suggested or provided by your medical education institution that you currently use or have used in the past: ______________________________________________________________________________ ______________________________________________________________________________ Question 6: Why do you use supplemental materials not suggested or provided by your medical education institution? ______________________________________________________________________________ ______________________________________________________________________________ Question 7: On a scale of 1-10, how would you rate your confidence that your medical education institution prepares you sufficiently for your USMLE exams? (1 being not confident at all and 10 being extremely confident) __1 (Not confident at all) __2 __3 __4 __5 __6 __7 __8 __9 __10 (Extremely confident) Question 8: On a scale of 1-10, how would you rate your confidence that your medical education institution prepares you sufficiently for residency? (1 being not confident at all and 10 being extremely confident) 39 __1 (Not confident at all) __2 __3 __4 __5 __6 __7 __8 __9 __10 (Extremely confident) Question 9: On a scale of 1-10, how would you rate the level of competitiveness between peers at your medical education institution? (1 being no competitiveness and 10 being intense competitiveness) __1 (No competitiveness) __2 __3 __4 __5 __6 __7 __8 __9 __10 (Intense competitiveness) Question 10: On a scale of 1-10, how would you rate the stress level of your overall experience in medical school? (1 being not stressful and 10 being extremely stressful) __1 (Not stressful at all) __2 __3 __4 __5 __6 __7 __8 __9 __10 (Extremely stressful) 40 Appendix B Recruitment Email July 23, 2021 Dear University of Utah Medical Student, In an effort to improve the supplemental materials provided to you by your institution and help your medical education institution better understand your study habits, please take this 10- question, 2-minute anonymous survey: https://weber.co1.qualtrics.com/jfe/form/SV_3NInqORxcPsAGLb This survey is academic research being conducted by Alexa Arnold, under the supervision of Dr. Melina Alexander, as a candidate for the Master of Education program at Weber State University. The survey will be available until 12pm on Friday, July 23rd. This study aims to better understand the study habits of medical students in regard to the use of supplemental materials. Your responses are critical to this study, whether or not you regularly use supplemental materials. Anonymous data will be presented to the University of Utah, School of Medicine, with an aim of improving your medical education. Please note that your participation in this survey is voluntary, and you can withdraw or refuse to answer any question at any time without penalty. The data provided to University of Utah, School of Medicine staff will not have any identifying information. If you have any questions or concerns about this research project, please contact the student researcher Alexa Arnold at alexaarnold@mail.weber.edu or by phone at 801-390-4605. Thank you! Alexa Arnold