Tracking and Evaluation
The Center for Translational Medicine (CTM) relies on systematic evaluation of programs, resources and services, and translational activities to assess its progress toward improving efficiency and effectiveness along the translational science spectrum: moving research concepts from basic science to clinical studies and population health outcomes. The objective of CTM Tracking and Evaluation is to assess the overall impact and accomplishments of the Center, building a robust composite of performance indicators to continuously improve research translation at UT Southwestern.
Methods and Measuring Outcomes
Meaningful measurements and data collection are important steps in program evaluation. The Tracking and Evaluation team utilizes mixed methods to monitor success, including data surveillance, surveys, and specific aim reports (key performance indicators). In addition, we collect qualitative feedback from program faculty, staff, and investigators to identify areas needing immediate or drastic improvement. Tracking and Evaluation employs classic methodologies to assess CTM effects on investigators and trainees (customers), program components (contributors), and the larger system (clinical and translational science as a whole).
Tracking and Evaluation methods and measures are two-fold. First, there is a focus on each individual CTM component to assist in promoting user-defined specific aims and metrics to determine success. The Tracking and Evaluation team and CTM component leaders work together to create metrics used to demonstrate measurable progress and success of individual specific aims. These metrics are revised year-to-year as the grant progresses based on their effectiveness and usefulness. Metrics and progress reports on specific aims are collected using a quarterly tracking system, which allows for data maintenance throughout the year.
The second focus is in seeking to bridge internal gaps between components to facilitate dynamic improvement, analyzing performance data from different sources to inform cross-functional metrics.
The CTM is required to report to the NIH all publications associated with projects that receive services, resources, or other funding from the CTM. The Tracking and Evaluation team assists investigators with properly citing CTM grants when a manuscript is submitted to a journal as well as ensuring that papers accepted for publication are deposited in the NIH Manuscript Submission system for their release to PubMed Central.
Research Launchpad is the web-based system used to track requests for CTM resources by the UT Southwestern research community. This system allows researchers with a UTSW ID to login and request a service on behalf of themselves or their PI. The request is then routed to appropriate CTM personnel.
Annual Progress Report
The CTM is required to submit an Annual Progress Report (APR) to the NIH to continue receiving the CTSA funds each year. The Tracking and Evaluation team assists in this process by summarizing data collected throughout the year, including, but not limited to, all of the methods and measurements mentioned above. Progress report summaries of the overall CTM and the individual components are a central part of the APR, demonstrating the activities and direction of the Center during the previous grant year.
Regional and National Consortium Involvements
Texas Regional CTSA Consortium (TRCC)
The Tracking and Evaluation team actively participates in the Texas Regional CTSA Consortium (TRCC), including ongoing working groups as well as breakout sessions at TRCC quarterly meetings. These activities keep the Tracking and Evaluation team current on activities of other Texas CTSA institutions and their evaluation methods and concerns. Outcomes from these involvements will improve translational research infrastructure for all Texas CTSA institutions, providing benchmarks and strategic alignment between CTSAs for translational investigators. Tracking and Evaluation also collaborates with the TRCC in participating in cross-CTSA data collection efforts using common metrics to assess outcomes across CTSA sites.
National CTSA Consortium
Tracking and Evaluation is also actively involved in National CTSA Consortium efforts, including Evaluation Key Function Committee projects and membership in multiple work groups. The Tracking and Evaluation team is working closely with the national level to implement Common Metrics, as mandated by NCATS. The Common Metrics will provide uniformity in data collection across all CTSAs and will advance our Center to better serve the research community in the future.
CTSA Evaluation Methodology Publications and Important Links
- Factor Analysis of the Clinical Research Appraisal Inventory (CRAI)
- Institute Of Medicine Report - The CTSA Program at NIH: Opportunities for Advancing Clinical and Translational Research
- Evaluation Guidelines for the Clinical and Translational Science Awards (CTSAs)
- Evaluating Translational Research: A Process Marker Model
- Common Metrics to Assess the Efficiency of Clinical Research
- Evaluation Metrics for Biostatistical and Epidemiological Collaborations
- Evaluating and Giving Feedback to Mentors: New Evidence-Based Approaches
- Assessing Research Self-Efficacy in the Physician-Scientists: The Clinical Research Appraisal Inventory
- A Shortened Version of the Clinical Research Appraisal Inventory: CRAI-12
- Google Wave: Have CTSA-Minded Institutions Caught It?
- Using Research Metrics to Improve Timelines: Proceedings from the 2nd Annual CTSA Clinical Research Management Workshop