8   Identifying Impact

In recent years, much debate has surrounded the definition of doctoral impact and its measurement in both traditional and professional doctorates (Halse & Mowbray, 2011). Professional doctorates differ according to the discipline and university—some consist solely of work-based projects or research supervised by university faculty, and others include coursework and discipline-specific dissertations. Notwithstanding the format and structure, professional doctorates can have a substantial impact for professionals and their work environments by connecting professionals, universities, and institutions or organizations. Online professional doctorates in which participants remain in their professional contexts while pursuing an academic degree online provide even more opportunities for such connections, especially if they are purposefully designed to connect theory, research, and practice. In addition to maintaining quality in online teaching and learning, an important aspect of quality management in an online professional doctorate is assessing to what extent it is meeting the needs of researching professionals and merging theory, research, and practice to effect change. The impact of online professional doctorates might differ from that of traditional doctoral education; it is therefore necessary to ensure that what is being measured as impact is both envisioned and facilitated during an online professional doctorate, providing alignment among program goals, program design, and the research conducted to assess impact.

In this chapter, we discuss the types of impact that an online professional doctorate can have and the ways in which they can be documented. We then share the types of impact we have identified in the EdD in educational technology at the University of Florida (UF EdD EdTech) and describe key considerations for identifying and documenting impact in an online professional doctorate.

DEFINING IMPACT IN AN ONLINE PROFESSIONAL DOCTORATE

We subscribe to Halse and Mowbray’s (2011) conceptualization of the doctorate “as both a process and a product” and their call to “think anew about the impact of the doctorate from a more critical frame,” one “that attends to the diversity of individuals, organizations and institutions participating in various phases of the doctorate” (p. 514). We thus believe that a program’s impact must be assessed throughout the phases of the student journey in addition to after program completion. Furthermore, the impact of an online professional doctorate and the methods used to measure it depend on the goals of a specific program, the discipline, and the design of the program (e.g., coursework and a dissertation, only a dissertation, a portfolio of artifacts and projects). It is, therefore, important for program leaders to identify the outcomes they will measure and the impact that program designers and the institution value before making decisions about instruments and data collection.

Impact in doctoral education has been defined as the “outcomes, benefits and returns that include, but are not limited to, economic returns” (Halse & Mowbray, 2011, p. 514) and has been conceived of as twofold: impact for the learner and impact for the organization (Halse & Mowbray, 2011; Lester & Costley, 2010). Researchers have reported that professionals have gained increased expertise, confidence, recognition, responsibility, and stature in the workplace as a result of professional graduate programs (Costley & Stephenson, 2008; Lester & Costley, 2010; Rhodes & Shiel, 2007). For organizations, “increased professionalism and motivation” (Lester & Costley, 2010, p. 568) and organizational changes in terms of professionals’ job roles or responsibilities within the organization has followed from professional doctorates. Therefore, the impact of an online professional doctorate can be broadly categorized in terms of changes for researching professionals and for their professional environments.

In our model for an online professional doctorate that connects theory, research, and practice, we assert that the participating researching professionals should combine foundational and theoretical knowledge in their disciplines with knowledge of research in their contexts to conduct research that generates knowledge to improve their professional contexts. We consider it valuable to document if and how knowledge acquired in academic environments during the online doctorate is applied to professional contexts and what changes in professional environments are effected by the research conducted. While the primary purpose of knowledge produced in the online professional doctorate is to address a problem of practice in a professional environment, the process of fostering scholarly thinking (i.e., reading, writing, reflection, and enculturation) leads to the dissemination of such knowledge in contexts beyond the immediate professional environment of the researching professional. Researching professionals in the UF EdD EdTech have conducted research and acquired and shared knowledge about educational technology that is applicable not only in their immediate professional contexts but also in other contexts in the discipline or across disciplines. Thus, online professional doctorates lead to both Mode 1 and Mode 2 knowledge production, where Mode 1 is largely disciplinary and Mode 2 is largely interdisciplinary (Gibbons et al., 1994). Based on the goals of an online professional doctorate and the types of activities in a curriculum, different types of knowledge may be generated and disseminated, making it important for program leaders or those wishing to identify impact to determine the type(s) of knowledge being generated and the impact of that knowledge on professional environments and the discipline.

The knowledge “formed and performed” (Tennant, 2004, p. 431) during doctoral education is not always visible because the doctoral student—in this case, a researching professional—both acquires and generates knowledge. To assess how knowledge is formed and performed, it is essential to understand (a) whether students are experiencing transformation and demonstrating changes in approach, behaviour, and action as these pertain to their professional practice or disciplines; (b) if and how students’ thinking, perceptions, and identities have changed; and (c) if and how students are growing professionally and contributing to changes in their professional contexts. Furthermore, in terms of individual development, some researchers have highlighted as impact the development of students as leaders and experts in their professional contexts or disciplines (Costley & Lester, 2012).

Transformational learning aims for changes in an individual’s perspectives (beliefs, attitudes, and behaviour) and can be fostered through critical reflection and critical discourse (Mezirow, 1998). “A transformed way of understanding, or interpreting, or viewing something” (Meyer & Land, 2003, p. 4), or the crossing of a threshold, leads to a changed outlook on a discipline, changed approaches, and/or changes in personal identity. Land and Meyer (2010) describe transformation as a journey through preliminal, liminal, and postliminal stages. Students go through a “state of liminality” (Meyer & Land, 2003, p. 10), where they struggle with old perspectives and the integration of new knowledge, before they can cross the threshold to reconfigure their understanding, which they demonstrate by changes in discourse, actions, and behaviour. Identifying what and when these changes occurred within researching professionals would shed light on the ways in which their doctorate work contributed to the transformation of their thinking. In order to document transformational learning, which is not sequential in nature, information would need to be gathered from students at regular intervals throughout and after their program. Further complicating such documentation would be the opportunities professionals have to apply acquired knowledge or changed understanding, approaches, and behaviours on the job; therefore, professional growth and the impact for professional environments is ongoing and should be documented throughout the online professional doctorate program.

MEASURING IMPACT IN AN ONLINE PROFESSIONAL DOCTORATE

Researchers have used a wide range of quantitative and qualitative methods to assess changes in students’ identity, thinking, approaches, behaviour, and action during doctoral education, including surveys, interviews, biographies, journals, concept maps, open-ended survey questions, and portfolios (Kiley, 2009; Land & Meyer, 2010; Lee, 2008; Wisker, 2015). Although several of these methods relate to individual courses and only occasionally to complete programs, in an online professional doctorate, it is important to study students’ development of scholarly thinking, transformation, and professional growth across courses, seminars, individual work, and the dissertation.

The impact of a program and its accompanying experiences is often assessed by collecting data and analyzing dissertations by graduates of the program. For example, Costley (2010) describes the individual and organizational impact of a transdisciplinary Doctor of Professional Studies program in the United Kingdom based on interviews with program graduates. She categorizes impact as direct changes made in the graduates’ organizations as a result of projects from the doctorate, enhanced credibility experienced by graduates, graduates’ perceptions of increased capabilities, and graduates’ continued interest in ongoing learning and professional development following the doctorate. In addition to analyzing interviews with students who had completed or were writing their dissertations, Burgess et al. (2013), in order to identify whether knowledge and skills were transferred to professional practice, analyzed students’ dissertations in three professional doctorates that contained reflections on the doctoral process and on the impact of their research. They reported the impact of the doctorate in terms of motivation; changes to students’ values, perceptions, and behaviour; student perceptions of research processes; and the ways in which students applied in the workplace the knowledge and skills they acquired during the doctorate. Dissertations in professional doctorates have also been analyzed for impact in the form of different types of knowledge or different types of generated outputs (Costley & Lester, 2012; Lester, 2012).

Going beyond graduates’ perspectives and dissertations, Fox and Slade (2014) interviewed peers and senior colleagues at graduates’ work organizations, as well as the graduates themselves, to determine what changes had taken place in the graduates and in their workplaces and how these changes had impacted their workplaces. They found that graduates were showing greater confidence and were engaging in multiple ways within and beyond their professional contexts, leading the authors to conclude that graduates’ changed self-perceptions had led to increased interactions and a larger network. This reinforces the research of Sanders, Kuit, Smith, Fulton, and Curtis (2011), who found that the perceived professional identities of students in professional doctoral programs changed during their studies and were impacted by their broader networks.

The above examples illustrate the data sources and types of impact in professional doctorates as reported in the literature; however, the research was not conducted in online doctorates. Discourse, reflection, articulation of perspectives, and implementation of research occur in electronic form in an online professional doctorate, resulting in data that is available and analyzable. We are fully cognizant that such data sources serve as external reflections of cognitive processes and ontological change and that much of this change is internal. Attempts can be made through prompts, activities, and research instruments to make such change transparent during online doctorates. Land and Meyer (2010) called such data sources “jewels in the curriculum” that help “externalize” students’ thought processes (p. 75). Changes in students’ thinking, perceptions, and identities; in their disciplines; and in their professional contexts can be documented in the form of journals, text, audio or video reflections, other multimedia artifacts, or portfolios (Kumar & Arnold, 2014). For example, in the UF EdD EdTech, at the end of their first year students write a reflection about their growth as researching professionals that program leaders subsequently analyze to determine the transformation and changes in self-perceptions that have occurred.

Professional doctoral students interact not only in academia with professors and peers but also in professional organizations and at their workplaces, making it difficult to identify exactly where and when such changes occur. Although it is possible to survey online students and interview them about their changed behaviours in environments outside of the online professional doctorate, collecting data from students is challenging at a distance. Students’ self-reports of changes in behaviour or action can be confirmed with up-to-date curriculum vitae (CVs) and professional websites, which can provide concrete information on professional growth, including changes in roles and responsibilities and actions such as leading workshops and making presentations. Data can also be collected in the form of surveys or interviews that target areas such as application of knowledge to practice, participation in professional organizations, and new initiatives and projects. An analysis of the networks and communities in which professional doctoral students participate can also provide insight into whether they are interacting and disseminating knowledge in new professional and academic communities, organizations, and spaces. As we mentioned earlier, data from students’ professional environments (e.g., from colleagues and supervisors) can also provide information on how theory and research from their academic experiences are interacting with and changing their practice.

In addition to impacting doctoral students and their professional environments, online professional doctorates can impact the institution offering the doctorate in at least four ways. First, online professional doctorates can impact other programs at the institution. At the University of Florida, for example, an online professional doctorate in teacher education was developed and offered soon after the UF EdD EdTech was introduced. Although the program is very different from ours, faculty members initially held several discussions with us to understand how our program is structured and the challenges that we have faced. Second, an online doctorate can impact the institution in terms of the availability or development of both online student services and online information-literacy resources. When we first offered the UF EdD EdTech, library resources for online students were not well developed at our institution. The resources available and the understanding of professional doctoral students have greatly increased with the offering of three online professional doctorates in our college and of multiple professional online programs at the university. Third, an online doctorate can impact how faculty members mentor students in on-campus programs. Ten of the eleven faculty members who teach in two online professional doctorates at our institution reflected during interviews that their mentoring of online professional students has led to their adoption of additional strategies when mentoring on-campus doctoral students (Kumar & Johnson, 2017). A fourth impact of an online doctoral program on the wider institution is in faculty members’ professional growth and learning about professional environments and other disciplines. During online interviews, faculty in our program who had not been professional doctoral students and whose research had not been conducted in professional environments reported that their work with online professional doctoral students had increased their awareness and knowledge of research in specific professional environments (Kumar et al., 2013).

The goals and purpose of impact measurement can thus encompass not only the impact for participating professionals and their professional environments but also the impact for faculty members, the program, and the institution offering the online professional doctorate. As faculty members in an online professional doctorate, we have focused on the measurement of impact for professional doctoral students and their contexts in the UF EdD EdTech, which we describe in the next section.

MEASURING IMPACT IN THE UF EDD EDTECH

In our online doctoral program, we emphasize the relevance of instructional content to students’ professional goals and the applicability of program activities to real-world environments. We encourage students to focus on a problem of practice and to produce artifacts that are relevant and useful in the context of those identified problems. We have found that when connections among theory, research, and practice are consistently scaffolded throughout an online professional doctorate with the aims of fostering scholarly habits of mind, conducting research that improves professional contexts, and helping professionals achieve their learning and professional goals, the measurement of impact reveals not only changes in researching professionals, including in their behaviours and their professional environments, but also dissemination of knowledge in various forms. In the following sections, we describe the data sources we used to assess impact with different cohorts during different stages of the UF EdD EdTech and the types of impact that we identified. Although not all of these sources of data were used with all cohorts, they provide an overview of various data-collection methods. We hope that our presentation of impact in our program can help others consider possible areas of impact in their online professional doctorates and methods of documenting these impacts.

We collected data from our first two cohorts during each year using a survey, semi-structured interviews, and focus groups. Our goal was to determine how students were applying knowledge acquired from the doctorate in their practice, enculturating into the discipline, experiencing growth, and contributing to changes in their professional environments during the online professional doctorate (see figure 3). Doctoral students or faculty members not involved in the design or implementation of the program collaborated to collect these data. Students’ CVs were analyzed to triangulate students’ self-reported impact.

Figure 3. Sources of impact data in UF EdD EdTech.

Primary Data Sources

Survey at the end of Year 1. At the end of Year 1, we used a survey to assess the quality of online teaching and learning in the UF EdD EdTech. The survey had an internal consistency reliability of 0.88 (Kumar et al., 2011) and featured three sections: Faculty Instruction and Feedback; Support, Learning Environments, and Community Building; and Application of Learning. The Application of Learning items helped identify students’ application of knowledge and professional growth as a result of the online professional doctorate.

Focus groups at the end of Year 1. To further probe for the changes students experienced regarding identity and approach to professional practice, faculty members not connected with our program conducted focus groups with one of our cohorts. Focus-group questions probed for changes in students’ approaches to research, their professional practice, and their disciplines. The aim was to identify whether students’ perceptions of themselves as scholars and professionals in their contexts had changed during their first year in the professional doctorate.

Student interviews during Year 2. To identify specific areas of impact, we conducted telephone or in-person semi-structured interviews during Year 2 about students’ professional growth and the impact of the professional doctorate on their practice. The interviewer, a new faculty member, probed for tacit changes as well as specific applications of content to professional contexts during the first one and a half years of the program. Because students entered the program with considerable experience and expertise, follow-up questions focused on impact specific to learning from the program: for example, Was this an initiative you had planned before you began the doctorate or one you would have developed regardless of your participation in it? Had you presented at similar conferences before? (Kumar & Dawson, 2012a, 2012b). Students were encouraged to provide specific names of initiatives and links to resources as evidence of their claims. Open-ended questions were also posed to provide students with opportunities to highlight other areas of impact. It was not possible to conduct interviews with all cohorts; therefore, we used an open-ended survey to collect this data.

Students’ curriculum vitae. Although students were asked to provide resources as evidence, the interview, focus group, and survey data in the UF EdD EdTech were often based on students’ self-reports; therefore, triangulation was achieved by analyzing students’ CVs for achievements, professional activities in educational technology, and changes in job roles. All students had professional websites in the public domain that provided information to corroborate their statements. Both the CVs and websites served as sources of data.

Focus groups during Year 3. Because of our inability to conduct individual interviews with one cohort during the first two years, faculty members from another program conducted focus groups for us between the completion of qualifying exams and the beginning of the dissertation phase. The questions focused on students’ experiences during the first two years in the program and how those experiences had influenced their professional growth and practice.

Post-program student interviews. Given the highly individual nature of the dissertation process, despite small-group mentoring and peer support, we consider it important to conduct interviews after program completion about students’ experiences with the dissertation and online mentoring. In the UF EdD EdTech, a final question at the end of this interview probed for the impact of the dissertation and the research process on students’ identities, their professional growth, and their professional environments. A researcher who did not contribute to the design or implementation of the professional doctorate conducted these interviews.

Other Sources of Data

Artifacts and reflections submitted by students during the doctorate have informed decisions about curriculum and program activities in the UF EdD EdTech but have not been analyzed to document student growth and impact. We hope to analyze these rich data sources moving forward. We have, however, analyzed all dissertations completed by the end of 2014 to identify the types of research and research methodologies being used in professional environments and will follow up with graduates about the impact of their research on those environments. In addition to collecting data from students and their artifacts, we would have liked to interview or survey the leadership in graduates’ professional contexts about their professional growth and the impact of their research on those contexts. However, we lacked the time and resources in our program to do so. We would highly recommend this additional source of data to program leaders.

The faculty working in an online professional doctorate and mentoring individual students can also provide insight into student growth. Interviews with faculty about online mentoring of dissertations included questions about students’ professional growth and yielded additional perspectives about impact. These data have to be triangulated with the students’ CVs or their self-reports in order to confirm the impact reported by faculty members.

AREAS OF IMPACT IDENTIFIED IN THE UF EDD EDTECH

Professionals in our program work in different disciplines (e.g. mathematics, science, art, foreign language education, nursing, social studies), professional contexts (e.g., higher education, K-12, nonprofit), and job roles (e.g., instructional designer, teacher, administrator, instructor). In this section, we briefly present our results from research about the impact of the program on researching professionals themselves, on their changed approaches to practice, on their application of knowledge and skills to their professional environments, and on their enculturation into the discipline and dissemination of knowledge. In addition, we report here the impact of students’ dissertation research on their professional environments.

Self-Perceptions and Professional Growth

During interviews at the end of the first year, students stated that the professional doctorate had increased their confidence in their abilities and that their interactions with peers within the doctorate and with experts in professional organizations had increased their morale and broadened their horizons. The process of reflecting on their professional goals and areas of specialization, reading research critically and relating it to their professional practice, and engaging in academic discussions with professionals from multiple contexts led to new insight and knowledge, changed their perceptions of themselves as professionals and of their professional contexts, and helped them formulate new professional goals (Kumar, 2014a).

Students felt that new doors had opened because of their participation in the professional doctorate, and they had begun considering new roles beyond their contexts. Students had quickly assumed leadership in their organizations, advising and making decisions about technology acquisition, integration, and professional development; participating on committees; and leading initiatives. Within their first year after graduation, thirteen of the fourteen graduates in our first cohort had assumed new responsibilities, and twelve had taken on new roles (Kumar, 2014a). For example, teachers had assumed responsibilities interacting with the school board and participating in school- and district-level technology acquisition and professional development, and instructional designers had led faculty development and other initiatives related to e-learning. While several students changed jobs during the professional doctorate, many assumed new roles or started new jobs after they graduated.

Changed Approach to Professional Practice

Exposure to research and scholarly articles during doctoral coursework resulted in students making research-based decisions about buying, adopting, or implementing new technologies; bringing literature to meetings (Kumar & Dawson, 2012a); and implementing research and evaluations in their professional contexts even before they began their dissertations. The most prominent impact cited by students was related to their use of educational technology research and a data-driven approach. Students provided several examples of their new capability to read research critically, communicate research results during meetings, use research to justify their decisions, and conduct small-scale evaluation and research projects in their professional environments (i.e., K-12, higher education, corporate environments, and virtual schools). Students’ growing comfort with research increased their confidence in working with colleagues and leaders on grants and policy decisions, and students were also increasingly invited to participate in grants or provide advice on decisions.

Application of New Knowledge in Professional Environments

In chapter 7, we described in detail the CoI survey that contains items on a scale of 1 to 5 about the application of learning to professional practice. Table 3 lists the mean ratings, cited from previous articles (Kumar et al., 2011; Kumar & Ritzhaupt, 2014), for these items in our first three cohorts. These ratings not only reveal the impact of the program but also support the fact that the consistent improvement measures that we used to focus on the quality of online teaching and learning and the connections among theory, research, and practice succeeded in improving the program for subsequent cohorts.

Table 3. Application of Learning in Cohorts 1, 2, and 3 at the End of Year 1

In addition to the CoI survey results, student interviews, open-ended surveys, and students’ CVs after Year 1 indicated that a majority of our students had applied learning from the program to integrate new technologies, create new curriculum, and write technology grants in their professional practice within the first two years of their participation in the online professional doctorate (see table 4; Kumar & Dawson, 2012a). Although the UF EdD EdTech emphasizes not the use of new technologies but rather the thoughtful and research-based integration of any technology in education, students were exposed to new technologies and to multiple ways of integrating and teaching with these technologies, which they applied to their professional environments.

Table 4. Application of Program Content in Professional Practice, Cohorts 1 and 2

Students used new technologies such as a learning management system, interactive whiteboards, social media, and simulations in innovative ways in their teaching, organizations, or districts. Several students adopted research-based strategies and created theory- and research-based online materials, modules, courses, and curricula in higher education, high school science and math, middle school social studies, and elementary classrooms (Kumar & Dawson, 2012a).

Enculturation into the Discipline and Dissemination of Knowledge

Doctoral students in educational technology come from different disciplines and backgrounds; therefore, we consider the development of scholarly thinking in the discipline an important impact of the professional doctorate. Several students joined professional organizations in educational technology and presented at regional, national, and international conferences in both educational technology and other disciplines (e.g., Online Learning Consortium; International Society for Technology in Education; American Educational Research Association; Association of Educational Communications and Technology; Campus Technology; Virtual School Symposium). Students continued to be active in professional organizations in their respective disciplines (e.g., nursing, library sciences, business education, math) during their doctoral work and also expanded their interdisciplinary and educational technology networks (Kumar & Dawson, 2014).

Increased knowledge and confidence resulted in students sharing their expertise in educational technology in the form of presentations and blended and online professional development offerings at their institutions, counties, or districts. They reported that in addition to the content of the online professional doctorate, which exposed them to research in educational technology implementation, the pedagogy used by faculty influenced the way they taught or designed curriculum in their professional contexts. During interviews, faculty members provided examples of how students were integrating research and theories in educational technology into their teaching, professional development workshops, or initiatives at their institutions. The new knowledge they had acquired and the resulting activities led to students perceiving themselves differently and to others perceiving them as experts in educational technology. Students who had published their projects or research in scholarly journals emphasized the role of the professional doctorate in helping them do so.

Impact of Dissertations in Professional Environments

Students in the UF EdD EdTech completed dissertations that were grounded in theory, research, and practice and that addressed the integration of educational technology into disciplines such as nursing, information literacy, public health, and academic advising, as well as into teaching and learning at all levels of education (e.g., K-12, nonprofit, higher education; Dawson & Kumar, 2014). The implementation of dissertation research in professional contexts has had significant impact for these contexts. Several of the dissertations completed in our program involved the implementation of new initiatives such as an e-learning guide, professional development for teachers and faculty, and new technologies such as iPads. Dissertations documented the outcomes of interventions and implementations, and students elaborated, during postgraduation interviews, on the ways in which these outcomes had influenced their professional contexts. For example, one student studied the implementation of a student information system, and, based on her findings and suggestions, professional development was implemented in her district. Another student designed online professional development for faculty members at her institution; researched faculty learning and application of content to their teaching; and, in order to apply her findings to other professional development offerings, improved online professional development based on her results. She also presented her research at a leading international conference. A third student studied the use of interactive whiteboards in the teaching of English in elementary classrooms and, based on his dissertation research, compiled best practices for professional development and the design of instructional materials for his context. This student later applied the knowledge he had gained in a new school context in a different country.

KEY CONSIDERATIONS

Over the three cohorts that have completed dissertations in the UF EdD EdTech, we have found different types of impact for researching professionals, both in their application of knowledge and skills to their professional environments and in the ways in which they enculturate into the discipline and disseminate their research and knowledge. Based on our experiences, we suggest some key considerations for measuring impact in an online professional doctorate.

Defining areas of impact and purpose of impact assessment. The impact of an online professional doctorate is influenced by several factors that can also serve as a starting point for defining types of impact: the goals of the program; the discipline in which it is offered; institutional and accreditation requirements for the format of the curriculum and the culminating research project; and the curriculum design. The impacts on context, discipline, profession, and student can take different forms during the various phases of an online professional doctorate. Consequently, those measuring impact in online professional doctorates must adopt an open-ended approach, identifying predetermined areas of impact while allowing for unanticipated impact. Some areas of impact found in the literature that can be used for impact assessment are identity or self-perception; attitudes, beliefs and approaches; application of content to the workplace; student professional growth; and changes in the professional environment. Some additional areas of impact are the hosting institution, the faculty members involved in the doctorate, and professional communities in which students participate. External and internal requirements—such as program reports, data for accreditation, and institutional guidelines—often influence the purpose of impact assessment. We have found it valuable to articulate clearly the purpose of data collection (e.g., to determine if the program is achieving its goals, to provide justification for funding) while attempting to document impact.

Collecting data related to program impact. It is important to explore how the impacts of a professional doctorate are perceived by the participants in the program, but these self-perceptions should be triangulated with other sources of data (e.g., student curricular artifacts, projects, products created by students in their professional contexts). One of the challenges we faced in the UF EdD EdTech with sources of data such as curricular artifacts or projects was that these artifacts were not designed with the goals of impact assessment in mind. Rather, they were designed to achieve program goals and were completed by students hoping to achieve approval, and they had already been graded or reviewed by faculty members in that context. The impact analysis of these products by faculty members in the program was therefore problematic. For this reason, as mentioned above, we have found it valuable to partner with researchers and faculty members not involved in the design of the program for the purposes of the data collection and analysis.

Formative and summative data collection is needed to document changes from the standpoints of process and product. Additionally, these different types of data collection allow program designers to make revisions to an online professional doctorate during a program offering (formative data collection) and for subsequent program offerings (summative data collection). Both formative and summative data collection require time and resources not always available to program leaders or faculty. They present several additional challenges in an online professional doctorate: for example, data are usually from participants’ self-reports and require triangulation using other sources, and data must be collected using technologies such as Skype and telephone. Although students and graduates might be comfortable using these technologies to talk about their experiences, their employers or colleagues might not. Furthermore, collecting data about impact on the professional environment can sometimes require gathering sensitive information that cannot be shared by those in the student’s professional context.

Ensuring program fit and student engagement. In addition to aligning impact metrics and purpose with the goals of the specific online professional doctorate, clearly communicating program goals to prospective students can contribute to program impact. In the UF EdD EdTech, we require program applicants to read the articles we have published about the program during the admissions process so that they can determine how our program can further their professional goals, since professional students are usually quite clear about why they would like to pursue a terminal degree. Based on their personal goals, identity, and stage of career, students are asked to define their own goals for their doctoral studies and to develop a plan to achieve those goals. Notwithstanding the value of purposeful design, the impact of an online professional doctorate is largely dependent on the extent to which professional students engage with the curriculum and peers and combine theory, research, and practice for application in their professional contexts.

Examining student professional development and partnerships with faculty. In our program, the focus on student professional growth, student research interests, the relevance of program content to professional environments, student enculturation into a discipline, and student reflection on professional goals and learning have contributed to positive impact for students, according to data collected during student interviews (Kumar & Dawson, 2014). Throughout the program, we scaffold the connections among theory, research, and practice and conduct research-relation-oriented mentoring in which dissertations are driven by students’ research interests, leading to multiple opportunities for impact on professionals and their work environments. These practices do not always align with the research agendas or grants of faculty members, but they have, for the most part, aligned with faculty areas of expertise and research interests. Regardless of the degree of alignment, graduates of our program have partnered with faculty to present at conferences, to publish, and to participate in decision making in professional organizations. Faculty members have found that students who are motivated to share their scholarship beyond their professional contexts tend to do so despite the constraints of time and multiple commitments. Although research has shown that graduates of professional doctorates typically remain within their professional community (Costley, 2013), we have found that our model for the online professional doctorate has largely resulted in graduates engaging in the academic community or professional organizations in the discipline even if they have not done so before. An open-minded approach to the kinds of partnerships, research, and scholarship that may result from dissertations in an online professional doctorate is essential while measuring impact.

Creating a climate for impact in the professional context. Most students, while completing the UF EdD EdTech, have worked in professional contexts that support their doctoral endeavours and that encourage the application of theory and research. However, if a professional student’s institution or professional context is not conducive to that kind of encouragement, the impact of an online professional doctorate can be limited. For example, a professional environment with strict confidentiality agreements may encourage the implementation of research but not allow the sharing of those results outside of that context. When students are being admitted to our program, we alert them to the importance of a professional environment that is supportive, and we ask them to inquire about the possibilities of implementing projects and research in their contexts. Informing their organization or institution about their participation in a terminal degree gives employers and colleagues the opportunity to involve students in projects in which their expertise may be useful.

CONCLUSION

Our discussion of impact in this chapter is largely based on our model for the online professional doctorate—one that provides students with opportunities to connect theory, research, and practice. Nevertheless, our suggestions for the definition, categorization, and measurement of impact in an online professional doctorate can be useful to those engaged in other types of professional doctorates and even in doctorates that are not offered online. Several students in the UF EdD EdTech have emphasized in their interviews and open-ended survey responses that the online nature of the doctorate significantly contributed to its impact. It provided them with opportunities to immediately and continually apply their learning throughout the doctorate, to reflect on their learning with peers, and to revise and share their approaches and research as they progressed.