{"ID":107471,"post_author":"9412100","post_date":"2023-05-01 14:49:12","post_date_gmt":"0000-00-00 00:00:00","post_content":"","post_title":"LIMSjournal - Spring 2023","post_excerpt":"","post_status":"draft","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"","to_ping":"","pinged":"","post_modified":"2023-05-01 14:49:12","post_modified_gmt":"2023-05-01 18:49:12","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.limsforum.com\/?post_type=ebook&p=107471","menu_order":0,"post_type":"ebook","post_mime_type":"","comment_count":"0","filter":"","holland":null,"_ebook_metadata":{"enabled":"on","private":"0","guid":"A85B8358-BAAA-49C8-AC9F-5857EB36A9F8","title":"LIMSjournal - Spring 2023","subtitle":"Volume 9, Issue 1","cover_theme":"nico_7","cover_image":"https:\/\/www.limsforum.com\/wp-content\/plugins\/rdp-ebook-builder\/pl\/cover.php?cover_style=nico_7&subtitle=Volume+9%2C+Issue+1&editor=Shawn+Douglas&title=LIMSjournal+-+Spring+2023&title_image=https%3A%2F%2Fs3.limswiki.org%2Fwww.limswiki.org%2Fimages%2Fc%2Fc1%2FFig2_Prahladh_EgyptJofForSci22_12.png&publisher=Lablynx+Press","editor":"Shawn Douglas","publisher":"Lablynx Press","author_id":"26","image_url":"","items":{"0f0e755c0f65e74ef81e3430d5482779_type":"article","0f0e755c0f65e74ef81e3430d5482779_title":"From months to minutes: Creating Hyperion, a novel data management system expediting data insights for oncology research and patient care (Snyder et al. 2022)","0f0e755c0f65e74ef81e3430d5482779_url":"https:\/\/www.limswiki.org\/index.php\/Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care","0f0e755c0f65e74ef81e3430d5482779_plaintext":"\n\nJournal:From months to minutes: Creating Hyperion, a novel data management system expediting data insights for oncology research and patient careFrom LIMSWikiJump to navigationJump to searchFull article title\n \nFrom months to minutes: Creating Hyperion, a novel data management system expediting data insights for oncology research and patient careJournal\n \nPLOS Digital HealthAuthor(s)\n \nSnyder, Eric; Rivers, Thomas; Smith, Lisa; Paoni, Scott; Cunliffe, Scott; Patel, Arpan; Ramsdale, ErikaAuthor affiliation(s)\n \nJames P. Wilmot Cancer InstitutePrimary contact\n \nErika underscore ramsdale at urmc dot rochester dot eduEditors\n \nShah, RutwikYear published\n \n2022Volume and issue\n \n1(11)Article #\n \ne0000036DOI\n \n10.1371\/journal.pdig.0000036ISSN\n \n2767-3170Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/journals.plos.org\/digitalhealth\/article?id=10.1371\/journal.pdig.0000036Download\n \nhttps:\/\/journals.plos.org\/digitalhealth\/article\/file?id=10.1371\/journal.pdig.0000036&type=printable (PDF)\n\nContents \n\n1 Abstract \n2 Background and significance \n3 Materials and methods \n\n3.1 James P. Wilmot Cancer Institute \n3.2 Medical informatics challenges \n\n3.2.1 Challenge 1: Lowering the skill floor \n3.2.2 Challenge 2: Reducing the cost of technology \n3.2.3 Challenge 3: Enhancing user autonomy \n3.2.4 Challenge 4: Optimizing data governance \n3.2.5 Challenge 5: Changing technological team structure \n\n\n\n\n4 Results \n\n4.1 Hyperion data manager \n4.2 Hyperion front end \n4.3 Data governance \n4.4 Security \n4.5 Usage metrics \n\n\n5 Discussion \n\n5.1 Drawbacks and considerations \n\n\n6 Conclusion and future work \n7 Supporting information \n8 Acknowledgements \n\n8.1 Funding \n8.2 Data availability \n8.3 Competing interests \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nEnsuring timely access to accurate data is critical for the functioning of a cancer center. Despite overlapping data needs, data are often fragmented and sequestered across multiple systems (such as the electronic health record [EHR], state and federal registries, and research databases), creating high barriers to data access for clinicians, researchers, administrators, quality officers, and patients. The creation of integrated data systems also faces technical, leadership, cost, and human resource barriers, among others. The University of Rochester's James P. Wilmot Cancer Institute (WCI) hired a small team of individuals with both technical and clinical expertise to develop a custom data management software platform\u2014Hyperion\u2014 addressing five challenges: lowering the skill level required to maintain the system, reducing costs, allowing users to access data autonomously, optimizing data security and utilization, and shifting technological team structure to encourage rapid innovation.\nThe Hyperion data management platform was designed to meet these challenges in addition to usual considerations of data quality, security, access, stability, and scalability. Implemented between May 2019 and December 2020 at the WCI, Hyperion includes a sophisticated custom validation and interface engine to process data from multiple sources, storing it in a database. Graphical user interfaces (GUIs) and custom wizards permit users to directly interact with data across operational, clinical, research, and administrative contexts. The use of multi-threaded processing, open-source programming languages, and automated system tasks (normally requiring technical expertise) minimizes costs. An integrated ticketing system and active stakeholder committee support data governance and project management. A co-directed, cross-functional team with flattened hierarchy and integration of industry software management practices enhances problem solving and responsiveness to user needs. Access to validated, organized, and current data is critical to the functioning of multiple domains in medicine. Although there are downsides to developing in-house customized software, we describe a successful implementation of custom data management software in an academic cancer center.\nKeywords: data management, data security, data visualization, data analysis, data integration, cancer centers\n\nBackground and significance \nAcademic cancer centers, particularly those embracing the learning health systems (LHS) model to dynamically generate and utilize high-quality evidence for patient decision making[1], require integration and maintenance of data systems offering intuitive access and manipulation of valid, ordered, and up-to-date knowledge informing clinical operations, clinical decision support, and research. Electronic health record (EHR) systems do not offer the data curation nor the user experience required to fully meet these needs. EHR systems were primarily designed to improve billing and revenue capture, requiring very different design decisions which often result in clunky, burdensome, and disorganized systems from the perspectives of many end-users. Moreover, useful data are typically not exclusively stored in a single location like the EHR, but across dozens of databases utilizing disparate (and often incompatible) technologies. \nAddressing the data needs of an academic cancer center introduces many challenges, including recruitment and retention of the appropriate technical expertise, while adhering to already thin financial budgets.[2] Technical difficulties include seamless integration of multiple data sources, enhancement of user buy-in for the data system (including mitigation of technology burnout), and rapidly changing technical and data landscapes.[2] Leadership challenges implicate the dominant paradigm for vertical, clinician-centric decision-making: current organizational leadership structures may be ill-suited to devising technical data solutions that inherently require systems thinking and rapid adaptation\/iteration. Importing organizational processes, systems thinking approaches, and technical domain expertise from other industries could help academic cancer centers around the country surmount many issues impeding data utilization.\nAttempts to optimally balance data currency, access, validation, and integrity in the healthcare setting typically involve data or research warehouses.[3][4] Given the disparate nature of the data sets to be merged, as well as the heightened security and privacy concerns involved in storing patient data, barriers include large capital outlays and difficulties accommodating potentially competing design aims such as efficiency, timeliness of reports, user experience, data validity, and accuracy.[5] Different health systems\u2014or even different groups within an individual health system\u2014may prioritize different design aims, such that a one-size-fits-all technical solution is inadvisable and obviates the ability to purchase a ready-made (commercial off-the-shelf) software solution. Even if a ready-made solution is available, the shifting data landscape could quickly make it obsolete; maintenance of data architecture, not only its initial build and deployment, requires significant ongoing technical skill and time. Throughout the processes of data architecture design and implementation, ongoing user feedback is critical, and clinical domain expertise is required at every step to maximize utility, comprehensiveness, and validity.\nThis paper discusses a novel design and successful implementation of a sophisticated data architecture\u2014Hyperion\u2014to address the data needs of an academic cancer center. Although each center has individualized needs embedded in idiosyncratic circumstances, a few principles may be derived to guide other centers hoping to implement and maintain a customized data architecture that users can employ confidently, productively, and adaptively to facilitate rapid answers to quality and research questions, and ultimately to improve patient outcomes at the point of care.\n\nMaterials and methods \nJames P. Wilmot Cancer Institute \nThe James P. Wilmot Cancer Institute (WCI) at the University of Rochester Medical Center (URMC, a large tertiary care academic medical center) is the largest cancer care provider in upstate New York, with a catchment area of 3.4 million people across a 27-county region. Prior to mid-2019, patient and research data were distributed across multiple unconnected systems, including the EHR (Epic), Research Electronic Data Capture (REDCap)[6] databases, clinical trial management system, individually maintained disease databases, laboratory information management systems (LIMS), shared resource databases, and billing applications. Additionally, useful data outside of the institution existed in a variety of formats, including behind web portals (e.g., clinicaltrials.gov)[7], in private company databases (e.g., externally performed molecular tumor testing and nonprofit organization public health databases), and within various externally maintained registries (e.g., New York State cancer registry). Gathering, merging, and validating data were tedious, time- and resource-intensive procedures performed largely manually, resulting in high expenditures for human abstractors and significant delays in implementing data insights. Data users lacked the ability to interact with real-time data; while static reports could be generated, the reliance on manual effort led to outdated, delayed reports. Intuitive and interactive data visualization was unavailable, limiting data exploration necessary to develop a research question or protocol, review clinical data, probe quality issues, or refine operations.\nIn 2019, a small informatics team was assembled to address WCI\u2019s data challenges, consisting of two software architects with expertise in custom healthcare software, a business intelligence software developer, a project manager, and a clinician with expertise in data science and quantitative research methods. The initial primary aim was to improve data availability to WCI faculty and staff, but in collaboration with the data architects it was clear that other aims should be equally prioritized, including data security, access to near-real-time data, data validity, flexibility in integrating data sources, and a platform for interacting with and visualizing data. Between May 2019 and December 2020, a complete data management and analytics platform was built, iterated, and deployed for WCI users: the Hyperion Centralized Medical Analytics (Hyperion) platform. The design and development process was conducted using the Project Management Institute's (PMI's) Project Management Body of Knowledge (PMBOK) best practices under the guidance of the project manager, including scoping, planning, identifying and meeting with stakeholders for each design phase, and communicating requested changes to the development team.[8]\n\nMedical informatics challenges \nAlthough well-maintained data warehouses are critical to the functioning of academic institutions, they do not address all informatics needs. All data management solutions share the common goal of consolidating data and ensuring its quality (ensuring, for example, data validity, availability, and completeness). In addition to meeting this goal for WCI, we sought to overcome five main challenges with the implementation of Hyperion. These medical informatics challenges\u2014abstracted from our iterative design process, and which included input from all stakeholders\u2014are distilled principles which could be considered across diverse implementations rather than a comprehensive description of discrete implementation challenges we faced.\n\nChallenge 1: Lowering the skill floor \nSetting up a data warehouse and maintaining complex interfaces, dashboards, and ad-hoc reporting often require significant time and large teams of information technology professionals. In one instance, developing a single-purpose, straightforward data management system to study antimicrobial resistance required two years and 4,000 person-hours among four skilled personnel.[9] The personnel costs of creating such systems become the primary challenge in implementation.[10] Beyond deployment, data management systems must respond to constantly updating data sets and sources, as well as updated user needs; for optimal functioning, maintenance typically requires continued involvement of highly educated (and high-cost) professionals such as software developers and data architects to design changes, as well as a team of programmers to directly implement changes to the software code. Budgets for these activities can quickly become unsustainable. Resilient software design, automation of projected maintenance activities, and creation of interfaces to translate programming and auditing activities could potentially \u201clower the skill floor,\u201d allowing a smaller team of non-experts to substitute for many of the activities of much larger (and more costly) information technology teams.\n\nChallenge 2: Reducing the cost of technology \nThe technical architecture to achieve data storage and processing can be very expensive, whether it exists on-premises or in cloud-based systems. On-premises solutions have high start-up costs for processing and storage hardware. Operating a cloud-based system similar in size to Hyperion would be expected to cost about $35,000 per month, using Amazon Web Services as a benchmark.[11]\n\nChallenge 3: Enhancing user autonomy \nData users exist on a spectrum of familiarity and sophistication with data manipulation, ranging from users who want simple, intuitive visual summaries to those capable of sophisticated analysis of raw data. A data system needs to be accessible and usable across the spectrum of potential users within WCI, ideally without the need for manual creation of curated datasets and visuals. Users should be able to autonomously access the data and tools they need to answer their own queries as much as possible.\n\nChallenge 4: Optimizing data governance \nPatient data are subject to laws and policies\u2014such as provisions within the Health Insurance Portability and Accountability Act (HIPAA)\u2014enforcing high scrutiny and standards to maintain patient privacy rights. Integration of data governance policies and activities into the data management platform would facilitate adherence to the strict confidentiality guidelines of the health system. Additionally, a well-designed ticketing system can help users clarify their requests, ensure appropriate data usage, streamline ticket approval and completion, and permit ongoing user needs assessment.\n\nChallenge 5: Changing technological team structure \nThe culture of academic medical centers often promotes top-down decision-making, prioritizing a particular paradigm centered around clinicians. Although clinicians bring invaluable perspective to the design and usage of data systems, they are not typically trained in (or necessarily aware of) the specific technical skillset required to design sophisticated data management systems. Leadership structures with clinicians or administrators at the \u201ctop\u201d can hinder technical teams, limiting their autonomy to implement technical best practice decisions, and delaying even simple architectural and programming tasks. Flattening the decisional hierarchy and implementing a transdisciplinary team approach could optimize functioning and increase development speed, simulating the pace of industry teams.\n\nResults \nThe custom-built data management solution for WCI, Hyperion, consists of a back-end relational Structured Query Language (SQL) database linked to a front-end interface platform accommodating multiple user types and tasks, including database administration, ticketing, reporting, and user dashboards. Table 1 compares Hyperion features to those of the most commonly available clinical data management software packages. Table 2 summarizes how Hyperion's design addresses the five identified challenges above.\n\r\n\n\n\n\n\n\n\n\n\n\nTable 1. Comparison of features: Hyperion versus other commercially available, commonly used clinical data management software systems.\n\n\n\n\n\n\n\n\n\n\n\n\nTable 2. Design elements supporting identified challenges.\n\n\n\nHyperion data manager \nHyperion\u2019s \u201cnervous system\u201d is the Hyperion Data Manager (HDM), an interface and validation engine built utilizing the Python programming language. Via a front-end system utilizing a Flask-based graphical user interface (GUI), approved users access point-and-click tasks to import data sources; initiate and manage extract, transform, and load (ETL) procedures; create sandboxes; copy table structures; re-run interfaces; and check for errors. A built-in custom wizard permits users to set up data interfaces without any programming knowledge for most data sources; Hyperion's bidirectional interfaces can be easily configured for all common data formats and models (HL7 FHIR, XML, application programming interface [API], etc.).\nFor integrating any new data source, HDM reads the database schema for the new data source and presents approved administrative users with a field listing. Via the GUI, users can select what to import into the Hyperion platform, create table names, rename data elements, and select a data update interval. HDM creates a scheduled interface pull at the selected interval and can accommodate near-real-time intervals. HDM can read in the most utilized database technologies such as Microsoft\u2019s SQL Server and Oracle, and it also supports token authentication, which is utilized by multiple governmental data sources and common medical database software products such as REDCap. For EHR data, HDM has custom code to simplify and semantically define tables and interpret names for commonly used fields. It translates field names into a more readable form based on clinical naming conventions and uploads data directly at the specified intervals. This streamlines table design and reduces time delay for downstream needs, including ad-hoc report development, sandbox creation, and full-scale applications. As all data tables are normalized and explicitly defined within our system, data reusability across the system is also facilitated.\nEase of use, combined with robust procedures to ensure data quality, has permitted rapid integration of multiple data sources for a variety of clinical, quality, operational, and research purposes (Fig. 1). To increase oversight of data validity and currency, HDM incorporates a complete validation and metrics monitoring package for data uploaded to Hyperion. HDM stores and displays metrics including number of new records per interface, timing and number of updates, and deletions or modifications to records since the prior interface update. Automated dashboards allow non-expert users to monitor metrics (Fig. 2); the dashboard displays are interactive, allowing point-and-click activities (e.g., adding or removing metrics to the display) and hovering over data points to get more information (see Figures A, B, and C in Supporting information, S1 Text for further examples).\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 1. Current architecture of Wilmot Cancer Institute (WCI) data warehouse.\n\n\n\n\n\n\n\n\n\n\n\n\nFigure 2. HDM monitoring dashboard.\n\n\n\nHDM\u2019s validation routines regularly evaluate for data consistency issues such as field and data type mismatches and extreme shifts in table sizes that would indicate mass data deletion or insertion. Some validation routines are specifically applied to certain data sources. For example, correct address information is critical for analyses or visualizations involving geospatial elements (Fig 3). Incoming address data are validated against a database of extant addresses, and a \u201ccertainty percentage\u201d is calculated, which permits a user-defined accuracy threshold for certifying the address as correct. Validation routines are linked to a wizard, and programmers are not needed for review of their output. The wizard allows users to examine items or processes flagged as potentially incorrect by the validation routines, and it presents them with a point-and-click menu of options (e.g., flag for further review, manually correct item, or approve upload).\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 3. CANcer Visual Analytic System (CANVAS) visual.\n\n\n\nTo improve both computational and cost efficiency, HDM uses multi-threaded best practice approaches to parallelize processes and increase processing speed. Job duration calculations allow for efficient scheduling of multiple processor cores prior to task execution. Utilizing the combination of scheduling plus multi-threaded processing increased speed by 50\u201377% compared to non-optimized methods. Cost efficiency is further optimized using open-source technology (e.g., Python\/Flask) for all code. Hyperion imports millions of rows of data every few hours, with a total implementation cost of $1,500 for hardware and software (in addition to already existing enterprise licensure costs). This design also facilitates future implementation of analytic pipelines, including machine learning pipelines.\nHDM sandboxes offer researchers temporary, partitioned access to curated datasets. Researchers can autonomously handle and analyze their data within a secure environment. Access is completely separated from other Hyperion architecture, and access privileges have an associated, pre-specified timeframe. Upon lapse of access privileges, HDM will terminate access, and archive and lock the data.\n\nHyperion front end \nThe front-end platform also utilizes open-source programming technology (HTML5 and JavaScript) to limit cost and align with common programming skillsets. Via a secure web portal, each user has access to a curated set of dashboards and activities based upon their access profile. Dashboards are highly interactive and customized to user groups based on iterative feedback. Users can view and securely sign documents (such as the data governance policy) and submit requests and ideas via an integrated ticketing system.\nHyperion administration and security activities are accessible via the web portal and HTML5 interface for approved users; point-and-click tools enable addition of new users and assignment of access privileges. The system has been tested and functions with all major web browsers. System administrators can view applications utilization data, including granular data by user and by clicks. Real-time data monitoring in conjunction allows for all data use to be precisely tracked and audited.\nHyperion\u2019s analytics-processing framework enables real-time analytics across any data element in the system, including those accessed via APIs, e.g., imaging, pathology, and molecular\/genomic data. Process efficiencies as described above permit rapid turnaround of ad-hoc reports, and scheduled reports are automatically generated and delivered at set intervals. A key principle guiding design of the user experience, however, prioritizes user autonomy in data access, analysis, and visualization. To this end, multiple customized dashboards permit users to directly interact with curated datasets and visualizations; Table 3 gives examples of some dashboards developed for various user groups in WCI.\n\r\n\n\n\n\n\n\n\n\n\n\nTable 3. Example user dashboards.\n\n\n\nCross-referencing and validation of patient addresses (described above), as well as integrated data crosswalks between various geographic levels (e.g., zip code and census tract levels), facilitate geospatial visualization and analysis at any geographic level specified by the user. These capabilities enhance the ability to examine data stratified by area within the WCI catchment area, map changes over time, or link to other datasets to analyze disparities across the region (e.g., nutritional access or socioeconomic disadvantage. Hyperion\u2019s CANcer Visual Analytic System (CANVAS) allows users to create regional map overlays for data elements such as patient visits, diagnostic categories, and clinical trial accruals which can be toggled on\/off or superimposed (Fig. 3). Animations permit direct visualization of changes over time.\n\nData governance \nData governance processes follow a centralized data governance model with a decentralized execution standard: this allows centralized control and authority to reside with the informatics team but permits the individual users and groups to be able to execute queries and analytics on curated sandboxed data sets or dashboards. This method of governance ensures data reliability and consistency via a single data baseline, validated daily.\nUpon initial login, Hyperion users are presented with the current WCI data governance policy, which they must electronically read and sign prior to interacting further with the system (Fig. 4). The integrated governance platform requires new signatures at login when the policy is updated. All signatures are encrypted and stored with the documents in Hyperion. The platform can support multiple documents with multiple versions, allowing for customized documentation for different user types as required.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 4. Custom electronic signature platform presenting the data governance policy at login.\n\n\n\nHyperion has an integrated custom ticketing platform for users to submit requests for additional development, reporting, or other requests. This is the primary method for initial communication of needs to the WCI Informatics Team and the Data Governance Committee. This method limits user cognitive load, streamlines development process, and facilitates automated tracking and reporting of requests. The ticketing platform collects all relevant info from users and populates a document for committee discussion, triage, and project management. Tickets are automatically routed to individuals on the governing committee for initial approval; if more information is needed, the committee member can submit questions back to the system via embedded email links. For items with limited time and scope requirements, the team can independently triage the request and place it appropriately within the project queue. Larger requests (e.g., >40 person-hours, complexity, and\/or multiple stakeholder groups affected) are flagged for committee discussion. Users can track the progress and actions on their individual tickets, including the ability to view key details entered by the informatics team (such as committee approvals, relevant dates, and requests for further information).\nThe Data Governance Committee consists of the WCI Informatics team and representatives from clinical operations, quality, administration, and research, all of whom have separate leadership roles within their respective domains. The Committee meets monthly and determines overall mission and priorities, discusses and triages large project requests, and reviews data usage and security.\n\nSecurity \nHyperion is only accessible via a secure networked computer or institutional virtual private network (VPN). Hyperion\u2019s custom security module augments the institution\u2019s Active Directory (AD) authentication and allows for user auditing and access control down to the individual application level. This permits user access to be customized by role and job function, and allows for the capture of full details on each user action, which Hyperion logs and monitors. Secure sandboxes are provided to users for data analytics purposes, without the ability to download data; this method of viewing and analyzing Hyperion reports is strongly encouraged. Otherwise, requested data documents are sent via either end-to-end encrypted email or secure file transfer protocol (SFTP). All documents able to be downloaded by the user include metadata tags to identify the source of download, the individual user, and date\/time stamps. In the event of a data consistency issue, the document can be compared to the logged SQL audit calls stored within Hyperion to ensure data was not altered. Request fulfillment for patient-identifiable data meets all institutional data security policies, including review by the Data Governance Committee.\n\nUsage metrics \nFrom January 2020 through December 2021, Hyperion has processed >41 billion records. More than 174 million records are currently stored, and 791 million records have been updated since January 2020. Hyperion currently has 148 unique users (52% physicians, 21% nursing, 17% IT, 10% administrators) accessing an average of 27 pages per day through December 2021. Table 4 shows the most accessed dashboards. There are currently 25 customized real-time updating dashboards in Hyperion, as well as 13 automated reports that distribute throughout the week to various departments and individuals via automated email distribution. The average turnaround time for an ad-hoc reporting request is three days. The average time to deploy a new dashboard is about four weeks.\n\r\n\n\n\n\n\n\n\n\n\n\nTable 4. Most accessed dashboards, January 2020\u2013December 2021.\n\n\n\nDiscussion \nCreating and maintaining complex, secure, and high-volume data warehouses, processing and assembling data views, and interfacing new data sources represent significant challenges for academic healthcare organizations, even those with adequate information technology (IT) staffing and budget. In typical practice, the team (or vendor) that creates the data warehousing software is distinct from the team tasked with maintaining it and supporting data access and visualization activities. Even after deployment of the data warehousing software, basic IT support for data warehousing typically includes interface developers, application developers, business analysts, business intelligence developers, security professionals, and help desk staff. Iterative adaptation of the software to meet changing data needs can be challenging or impossible, leading to the accrual of \u201cworkarounds\u201d and technical debt.\nIn WCI, we assembled a small, transdisciplinary team to develop a customized, adaptable, and scalable data management approach, supported by senior leadership and enterprise IT structures. Beginning with the definition and elaboration of key challenges we wished to address, we designed and built a modular and scalable software package addressing data storage, validation, and processing needs as well as data monitoring, access, and visualization. These structures were designed to permit rapid iteration and adaptation by a small but highly skilled technical team when needed, but to allow basic administration and continued maintenance to be performed by non-technical staff. The Hyperion database architecture, Hyperion Data Manager, security module, and governance modules were designed and deployed with a six-month timeline, and the entire package can be maintained by a single full-time equivalent (FTE). The modular extensible approach significantly reduces enhancement and update times. Limiting the continuous need for high-level technical skillsets to maintain software and data integrity frees a smaller team to work \u201cat the top of their cognitive skillsets,\u201d iterating solutions in response to user feedback, and creatively generating new solutions to complex data problems in WCI. This approach maximizes cost-effectiveness in addition to overall efficiency. Furthermore, Hyperion aligns with FAIR data principles (findability, accessibility, interoperability, and reusability).[12]\nHyperion has high user uptake, with many faculty members, staff, administrators, and others logging in daily to independently access data for clinical, operational, administrative, and research purposes. Although much of the data in Hyperion originates within the institutional EHR, it has previously been sequestered within individual patient records and only slightly more accessible to automated extraction methods than paper charts. Hyperion makes validated, curated, organized, near-real-time datasets automatically accessible and easily explorable via an interactive suite of analysis and visualization tools. Quality initiatives, program development, physician decision-making, clinical trial planning and management, research (grant applications and peer-reviewed analyses), clinical operations management, and more are all supported within a single management platform.\nIn addition to the design and building of the software elements, implementation of Hyperion required careful consideration and design of support structures to address data governance, security, and project management. Intrinsic software elements such as the data governance policy tracking, security audits, and the sophisticated ticketing system are embedded in a set of processes overseen by a supporting committee structure consisting of key stakeholders and meeting at least monthly. Users are not competing with requests external to WCI for attention, and two-way communication is streamlined and efficient due to the embedded nature of the informatics team.\nBeyond embedding a responsive team within and among the software users, the composition and functioning of the team are substantial revisions to the usual model of academic IT teams. A co-directed model shares formal authority for external interactions between a technical expert with extensive background in healthcare IT, and a practicing clinician with training in data science and informatics. Internally, the team is structured with a flattened decisional hierarchy, a deliberate emphasis on diversity of opinion and perspective, and a rapidly iterative approach to problem solving adapted from industry. Projects are managed through a combination of agile-based approaches and more traditional project management philosophies such as milestone phase gating. Although this structure is not able to completely shield the team from institutional bureaucracy, it has helped to create and sustain space for innovation and creativity within a traditionally cautious and even inert system.\n\nDrawbacks and considerations \nThere are several potential disadvantages to our strategy for addressing data needs within an academic cancer center. Academic medical centers may not be able or willing to support in-house software development for patient data, relying on outsourced software to guarantee robustness, security, and technical support. In the current market, the technical skillsets to achieve customized software solutions for managing patient data are rare and expensive, and achieving buy-in to budget for these positions may be very challenging. We have mitigated personnel costs with a smaller team comprised of individuals with cross-functional skillsets, but this poses its own difficulty: there is limited redundancy within the system to accommodate team member absences or departures. Although the software design offloads much of the technical maintenance, allowing it to be done by nontechnical personnel, the sustainability of the software still relies on a core highly-skilled technical team. However, stable teams are also required to internally support many vendor products within the technical ecosystem.\n\nConclusion and future work \nIn summary, Hyperion has surmounted large challenges in working with healthcare data to merge, organize, validate, and package data for use in multiple applications, including interactive user dashboards. Additional design considerations included lowering the skill floor for interaction with and maintenance of the software, reducing costs, and encouraging user autonomy. Development of data governance and other support structures, as well as discussions about team functioning and structure, occurred in parallel with the software build. Today, Hyperion provides a flexible, reliable, and scalable foundation for the development of responsive and innovative applications to support the mission and goals of an academic cancer center.\nFuture work includes turning our attention to further supporting data analytics, including machine learning (ML). An ML pipeline is being developed to allow users to explore their data using advanced techniques while also receiving hands-on education about the potential and pitfalls of these methods. Novel data visualization methods, including augmented and virtual reality methods (AR\/VR) are also in initial development and testing. \n\nSupporting information \nS1 Text (.docx): Fig A. Nursing dashboard. Fig B. Example of Provider Dashboard landing page, with interactive features (hover-over pop-up text and ability to click on bar graphs to \u201cdrill down\u201d on data). Fig C. Clinical Trial dashboard main page, with interactive features.\nAcknowledgements \nWe acknowledge the continued support of the Wilmot Cancer Institute\u2019s leadership in encouraging innovation, as well as the support of the physicians, nurses, and staff.\n\nFunding \nER is supported by the National Cancer Institute (K08CA248721) and the National Institute on Aging (R03AG067977). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.\n\nData availability \nAll data are in the manuscript and\/or supporting information files.\n\nCompeting interests \nThe authors have declared that no competing interests exist.\n\nReferences \n\n\n\u2191 Olsen, LeighAnne; Aisner, Dara; McGinnis, J. Michael et al., eds. (2007). \"Institute of Medicine Roundtable on Evidence-Based Medicine\". The learning healthcare system: workshop summary. Washington, DC: National Academies Press. ISBN 978-0-309-10300-8.   \n \n\n\u2191 2.0 2.1 Chaudhry, Basit; Wang, Jerome; Wu, Shinyi; Maglione, Margaret; Mojica, Walter; Roth, Elizabeth; Morton, Sally C.; Shekelle, Paul G. (16 May 2006). \"Systematic review: impact of health information technology on quality, efficiency, and costs of medical care\". Annals of Internal Medicine 144 (10): 742\u2013752. doi:10.7326\/0003-4819-144-10-200605160-00125. ISSN 1539-3704. PMID 16702590. https:\/\/pubmed.ncbi.nlm.nih.gov\/16702590 .   \n \n\n\u2191 Lyman, Jason A.; Scully, Kenneth; Harrison, James H. (1 March 2008). \"The development of health care data warehouses to support data mining\". Clinics in Laboratory Medicine 28 (1): 55\u201371, vi. doi:10.1016\/j.cll.2007.10.003. ISSN 0272-2712. PMID 18194718. https:\/\/pubmed.ncbi.nlm.nih.gov\/18194718 .   \n \n\n\u2191 Sheta, Osama El-Sayed; Eldeen, Ahmed Nour (2012). \"Building a health care data warehouse for cancer diseases\". arXiv. doi:10.48550\/ARXIV.1211.4371. https:\/\/arxiv.org\/abs\/1211.4371 .   \n \n\n\u2191 Ewen, Edward F.; Medsker, Carl E.; Dusterhoft, Laura E. (1 November 1998). \"Data warehousing in an integrated health system: building the business case\" (in en). Proceedings of the 1st ACM international workshop on Data warehousing and OLAP (Washington D.C. USA: ACM): 47\u201353. doi:10.1145\/294260.294271. ISBN 978-1-58113-120-8. https:\/\/dl.acm.org\/doi\/10.1145\/294260.294271 .   \n \n\n\u2191 Harris, Paul A.; Taylor, Robert; Thielke, Robert; Payne, Jonathon; Gonzalez, Nathaniel; Conde, Jose G. (1 April 2009). \"Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support\". Journal of Biomedical Informatics 42 (2): 377\u2013381. doi:10.1016\/j.jbi.2008.08.010. ISSN 1532-0480. PMC 2700030. PMID 18929686. https:\/\/pubmed.ncbi.nlm.nih.gov\/18929686 .   \n \n\n\u2191 \"ClinicalTrials.gov\". U.S. National Library of Medicine. https:\/\/clinicaltrials.gov\/ . Retrieved 11 July 2022 .   \n \n\n\u2191 Project Management Institute, ed. (2017). A guide to the project management body of knowledge \/ Project Management Institute. PMBOK guide (Sixth edition ed.). Newtown Square, PA: Project Management Institute. ISBN 978-1-62825-184-5.   \n \n\n\u2191 Wisniewski, Mary F.; Kieszkowski, Piotr; Zagorski, Brandon M.; Trick, William E.; Sommers, Michael; Weinstein, Robert A. (2003). \"Development of a clinical data warehouse for hospital infection control\". Journal of the American Medical Informatics Association: JAMIA 10 (5): 454\u2013462. doi:10.1197\/jamia.M1299. ISSN 1067-5027. PMC PMC212782. PMID 12807807. https:\/\/pubmed.ncbi.nlm.nih.gov\/12807807 .   \n \n\n\u2191 Rubin, Daniel L.; Desser, Terry S. (1 March 2008). \"A data warehouse for integrating radiologic and pathologic data\". Journal of the American College of Radiology: JACR 5 (3): 210\u2013217. doi:10.1016\/j.jacr.2007.09.004. ISSN 1558-349X. PMID 18312970. https:\/\/pubmed.ncbi.nlm.nih.gov\/18312970 .   \n \n\n\u2191 \"AWS Pricing Calculator\". Amazon Web Services. https:\/\/calculator.aws\/ . Retrieved 12 November 2021 .   \n \n\n\u2191 Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, I. Jsbrand Jan; Appleton, Gabrielle; Axton, Myles; Baak, Arie; Blomberg, Niklas; Boiten, Jan-Willem et al. (15 March 2016). \"The FAIR Guiding Principles for scientific data management and stewardship\". Scientific Data 3: 160018. doi:10.1038\/sdata.2016.18. ISSN 2052-4463. PMC 4792175. PMID 26978244. https:\/\/pubmed.ncbi.nlm.nih.gov\/26978244 .   \n \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and spelling. In some cases important information was missing from the references, and that information was added. The original includes both an abstract and an author summary, which is confusing since the purpose of the asbtract is to act as a summary of the text; for this version, the two were combined to form an intelligible, coherent abstract.\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\">https:\/\/www.limswiki.org\/index.php\/Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care<\/a>\nCategories: LIMSwiki journal articles (added in 2023)LIMSwiki journal articles (all)LIMSwiki journal articles on clinical informaticsLIMSwiki journal articles on softwareNavigation menuPage actionsJournalDiscussionView sourceHistoryPage actionsJournalDiscussionMoreToolsIn other languagesPersonal toolsLog inNavigationMain pageEncyclopedic articlesRecent changesRandom pageHelp about MediaWikiSearch\u00a0 ToolsWhat links hereRelated changesSpecial pagesPermanent linkPage informationPopular publications\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\nPrint\/exportCreate a bookDownload as PDFDownload as PDFDownload as Plain textPrintable version This page was last edited on 22 February 2023, at 01:03.Content is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.This page has been accessed 102 times.Privacy policyAbout LIMSWikiDisclaimers\n\n\n\n","0f0e755c0f65e74ef81e3430d5482779_html":"<body class=\"mediawiki ltr sitedir-ltr mw-hide-empty-elt ns-206 ns-subject page-Journal_From_months_to_minutes_Creating_Hyperion_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care rootpage-Journal_From_months_to_minutes_Creating_Hyperion_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care skin-monobook action-view skin--responsive\"><div id=\"rdp-ebb-globalWrapper\"><div id=\"rdp-ebb-column-content\"><div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\"><a id=\"rdp-ebb-top\"><\/a>\n<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:From months to minutes: Creating Hyperion, a novel data management system expediting data insights for oncology research and patient care<\/h1><div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\"><!-- start content --><div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\"><div class=\"mw-parser-output\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Ensuring timely access to accurate data is critical for the functioning of a <a href=\"https:\/\/www.limswiki.org\/index.php\/Cancer\" title=\"Cancer\" class=\"wiki-link\" data-key=\"fcd6751ee9aef5d96b8448c082d5e582\">cancer<\/a> center. Despite overlapping data needs, data are often fragmented and sequestered across multiple systems (such as the <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health record<\/a> [EHR], state and federal registries, and research <a href=\"https:\/\/www.limswiki.org\/index.php\/Database\" title=\"Database\" class=\"wiki-link\" data-key=\"ac630f0b5e30cbe7fed1422999c2baad\">databases<\/a>), creating high barriers to data access for clinicians, researchers, administrators, quality officers, and patients. The creation of <a href=\"https:\/\/www.limswiki.org\/index.php\/System_integration\" title=\"System integration\" class=\"wiki-link\" data-key=\"65c80aae9714da73905a2a343d8e6a81\">integrated data systems<\/a> also faces technical, leadership, cost, and human resource barriers, among others. The University of Rochester's James P. Wilmot Cancer Institute (WCI) hired a small team of individuals with both technical and clinical expertise to develop a custom <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a> software platform\u2014Hyperion\u2014 addressing five challenges: lowering the skill level required to maintain the system, reducing costs, allowing users to access data autonomously, optimizing <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_security\" title=\"Information security\" class=\"wiki-link\" data-key=\"9eff362d944224ff1d4ffe3a149d7cff\">data security<\/a> and utilization, and shifting technological team structure to encourage rapid innovation.\n<\/p><p>The Hyperion data management platform was designed to meet these challenges in addition to usual considerations of <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_quality\" title=\"Data quality\" class=\"wiki-link\" data-key=\"7fe43b05eae4dfa9b5c0547cc8cfcceb\">data quality<\/a>, security, access, stability, and scalability. Implemented between May 2019 and December 2020 at the WCI, Hyperion includes a sophisticated custom validation and interface engine to process data from multiple sources, storing it in a database. Graphical user interfaces (GUIs) and custom wizards permit users to directly interact with data across operational, clinical, research, and administrative contexts. The use of multi-threaded processing, open-source programming languages, and automated system tasks (normally requiring technical expertise) minimizes costs. An integrated ticketing system and active stakeholder committee support data governance and project management. A co-directed, cross-functional team with flattened hierarchy and integration of industry software management practices enhances problem solving and responsiveness to user needs. Access to validated, organized, and current data is critical to the functioning of multiple domains in medicine. Although there are downsides to developing in-house customized software, we describe a successful implementation of custom data management software in an academic cancer center.\n<\/p><p><b>Keywords<\/b>: data management, data security, data visualization, data analysis, data integration, cancer centers\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background_and_significance\">Background and significance<\/span><\/h2>\n<p>Academic <a href=\"https:\/\/www.limswiki.org\/index.php\/Cancer\" title=\"Cancer\" class=\"wiki-link\" data-key=\"fcd6751ee9aef5d96b8448c082d5e582\">cancer<\/a> centers, particularly those embracing the <a href=\"https:\/\/www.limswiki.org\/index.php\/Learning_health_systems\" title=\"Learning health systems\" class=\"wiki-link\" data-key=\"67847e5755f19310c2c0c7ec5c8f6e8e\">learning health systems<\/a> (LHS) model to dynamically generate and utilize high-quality evidence for patient decision making<sup id=\"rdp-ebb-cite_ref-1\" class=\"reference\"><a href=\"#cite_note-1\">[1]<\/a><\/sup>, require <a href=\"https:\/\/www.limswiki.org\/index.php\/System_integration\" title=\"System integration\" class=\"wiki-link\" data-key=\"65c80aae9714da73905a2a343d8e6a81\">integration<\/a> and maintenance of data systems offering intuitive access and manipulation of valid, ordered, and up-to-date knowledge informing clinical operations, <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support<\/a>, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_research\" title=\"Medical research\" class=\"wiki-link\" data-key=\"0ee7e4e2a32a422d78fe6bd1ab0d1cbc\">research<\/a>. <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">Electronic health record<\/a> (EHR) systems do not offer the data curation nor the user experience required to fully meet these needs. EHR systems were primarily designed to improve billing and revenue capture, requiring very different design decisions which often result in clunky, burdensome, and disorganized systems from the perspectives of many end-users. Moreover, useful data are typically not exclusively stored in a single location like the EHR, but across dozens of <a href=\"https:\/\/www.limswiki.org\/index.php\/Database\" title=\"Database\" class=\"wiki-link\" data-key=\"ac630f0b5e30cbe7fed1422999c2baad\">databases<\/a> utilizing disparate (and often incompatible) technologies. \n<\/p><p>Addressing the data needs of an academic cancer center introduces many challenges, including recruitment and retention of the appropriate technical expertise, while adhering to already thin financial budgets.<sup id=\"rdp-ebb-cite_ref-:0_2-0\" class=\"reference\"><a href=\"#cite_note-:0-2\">[2]<\/a><\/sup> Technical difficulties include seamless integration of multiple data sources, enhancement of user buy-in for the data system (including mitigation of technology burnout), and rapidly changing technical and data landscapes.<sup id=\"rdp-ebb-cite_ref-:0_2-1\" class=\"reference\"><a href=\"#cite_note-:0-2\">[2]<\/a><\/sup> Leadership challenges implicate the dominant paradigm for vertical, clinician-centric decision-making: current organizational leadership structures may be ill-suited to devising technical data solutions that inherently require systems thinking and rapid adaptation\/iteration. Importing organizational processes, systems thinking approaches, and technical domain expertise from other industries could help academic cancer centers around the country surmount many issues impeding data utilization.\n<\/p><p>Attempts to optimally balance data currency, access, validation, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">integrity<\/a> in the healthcare setting typically involve <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_warehouse\" title=\"Data warehouse\" class=\"wiki-link\" data-key=\"ca506499cdf544371c0a0d549ff0e9ee\">data or research warehouses<\/a>.<sup id=\"rdp-ebb-cite_ref-3\" class=\"reference\"><a href=\"#cite_note-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-4\" class=\"reference\"><a href=\"#cite_note-4\">[4]<\/a><\/sup> Given the disparate nature of the data sets to be merged, as well as the heightened security and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_privacy\" title=\"Information privacy\" class=\"wiki-link\" data-key=\"185f6d9f874e48914b5789317408f782\">privacy<\/a> concerns involved in storing patient data, barriers include large capital outlays and difficulties accommodating potentially competing design aims such as efficiency, timeliness of reports, user experience, data validity, and accuracy.<sup id=\"rdp-ebb-cite_ref-5\" class=\"reference\"><a href=\"#cite_note-5\">[5]<\/a><\/sup> Different health systems\u2014or even different groups within an individual health system\u2014may prioritize different design aims, such that a one-size-fits-all technical solution is inadvisable and obviates the ability to purchase a ready-made (commercial off-the-shelf) software solution. Even if a ready-made solution is available, the shifting data landscape could quickly make it obsolete; maintenance of data architecture, not only its initial build and deployment, requires significant ongoing technical skill and time. Throughout the processes of data architecture design and implementation, ongoing user feedback is critical, and clinical domain expertise is required at every step to maximize utility, comprehensiveness, and validity.\n<\/p><p>This paper discusses a novel design and successful implementation of a sophisticated data architecture\u2014Hyperion\u2014to address the data needs of an academic cancer center. Although each center has individualized needs embedded in idiosyncratic circumstances, a few principles may be derived to guide other centers hoping to implement and maintain a customized data architecture that users can employ confidently, productively, and adaptively to facilitate rapid answers to quality and research questions, and ultimately to improve patient outcomes at the point of care.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"James_P._Wilmot_Cancer_Institute\">James P. Wilmot Cancer Institute<\/span><\/h3>\n<p>The James P. Wilmot Cancer Institute (WCI) at the University of Rochester Medical Center (URMC, a large tertiary care academic medical center) is the largest cancer care provider in upstate New York, with a catchment area of 3.4 million people across a 27-county region. Prior to mid-2019, patient and research data were distributed across multiple unconnected systems, including the EHR (Epic), Research Electronic Data Capture (REDCap)<sup id=\"rdp-ebb-cite_ref-6\" class=\"reference\"><a href=\"#cite_note-6\">[6]<\/a><\/sup> databases, <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_trial_management_system\" title=\"Clinical trial management system\" class=\"wiki-link\" data-key=\"69c3d457afb8e96412b08403b7bfcccb\">clinical trial management system<\/a>, individually maintained disease databases, <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> (LIMS), shared resource databases, and billing applications. Additionally, useful data outside of the institution existed in a variety of formats, including behind web portals (e.g., clinicaltrials.gov)<sup id=\"rdp-ebb-cite_ref-7\" class=\"reference\"><a href=\"#cite_note-7\">[7]<\/a><\/sup>, in private company databases (e.g., externally performed <a href=\"https:\/\/www.limswiki.org\/index.php\/Molecular_diagnostics\" title=\"Molecular diagnostics\" class=\"wiki-link\" data-key=\"8fc14cae7a6fbac9a53fae1394fae7ee\">molecular tumor testing<\/a> and nonprofit organization <a href=\"https:\/\/www.limswiki.org\/index.php\/Public_health\" title=\"Public health\" class=\"wiki-link\" data-key=\"81092e25c0bd359cedd1b9f9dc350c86\">public health<\/a> databases), and within various externally maintained registries (e.g., New York State cancer registry). Gathering, merging, and validating data were tedious, time- and resource-intensive procedures performed largely manually, resulting in high expenditures for human abstractors and significant delays in implementing data insights. Data users lacked the ability to interact with real-time data; while static reports could be generated, the reliance on manual effort led to outdated, delayed reports. Intuitive and interactive <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_visualization\" title=\"Data visualization\" class=\"wiki-link\" data-key=\"4a3b86cba74bc7bb7471aa3fc2fcccc3\">data visualization<\/a> was unavailable, limiting data exploration necessary to develop a research question or protocol, review clinical data, probe quality issues, or refine operations.\n<\/p><p>In 2019, a small <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics_(academic_field)\" title=\"Informatics (academic field)\" class=\"wiki-link\" data-key=\"0391318826a5d9f9a1a1bcc88394739f\">informatics<\/a> team was assembled to address WCI\u2019s data challenges, consisting of two software architects with expertise in custom healthcare software, a <a href=\"https:\/\/www.limswiki.org\/index.php\/Business_intelligence\" title=\"Business intelligence\" class=\"wiki-link\" data-key=\"cb8c6c58a636a17adacd11b7617cba5e\">business intelligence<\/a> software developer, a project manager, and a clinician with expertise in data science and quantitative research methods. The initial primary aim was to improve data availability to WCI faculty and staff, but in collaboration with the data architects it was clear that other aims should be equally prioritized, including <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_security\" title=\"Information security\" class=\"wiki-link\" data-key=\"9eff362d944224ff1d4ffe3a149d7cff\">data security<\/a>, access to near-real-time data, data validity, flexibility in integrating data sources, and a platform for interacting with and visualizing data. Between May 2019 and December 2020, a complete <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analytics<\/a> platform was built, iterated, and deployed for WCI users: the Hyperion Centralized Medical Analytics (Hyperion) platform. The design and development process was conducted using the Project Management Institute's (PMI's) Project Management Body of Knowledge (PMBOK) best practices under the guidance of the project manager, including scoping, planning, identifying and meeting with stakeholders for each design phase, and communicating requested changes to the development team.<sup id=\"rdp-ebb-cite_ref-8\" class=\"reference\"><a href=\"#cite_note-8\">[8]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Medical_informatics_challenges\">Medical informatics challenges<\/span><\/h3>\n<p>Although well-maintained data warehouses are critical to the functioning of academic institutions, they do not address all informatics needs. All data management solutions share the common goal of consolidating data and ensuring its quality (ensuring, for example, data validity, availability, and completeness). In addition to meeting this goal for WCI, we sought to overcome five main challenges with the implementation of Hyperion. These <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_informatics\" class=\"mw-redirect wiki-link\" title=\"Medical informatics\" data-key=\"f89ecb3b26617b8c6e09bc5e050cfd5d\">medical informatics<\/a> challenges\u2014abstracted from our iterative design process, and which included input from all stakeholders\u2014are distilled principles which could be considered across diverse implementations rather than a comprehensive description of discrete implementation challenges we faced.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Challenge_1:_Lowering_the_skill_floor\">Challenge 1: Lowering the skill floor<\/span><\/h4>\n<p>Setting up a data warehouse and maintaining complex interfaces, dashboards, and ad-hoc reporting often require significant time and large teams of information technology professionals. In one instance, developing a single-purpose, straightforward data management system to study antimicrobial resistance required two years and 4,000 person-hours among four skilled personnel.<sup id=\"rdp-ebb-cite_ref-9\" class=\"reference\"><a href=\"#cite_note-9\">[9]<\/a><\/sup> The personnel costs of creating such systems become the primary challenge in implementation.<sup id=\"rdp-ebb-cite_ref-10\" class=\"reference\"><a href=\"#cite_note-10\">[10]<\/a><\/sup> Beyond deployment, data management systems must respond to constantly updating data sets and sources, as well as updated user needs; for optimal functioning, maintenance typically requires continued involvement of highly educated (and high-cost) professionals such as software developers and data architects to design changes, as well as a team of programmers to directly implement changes to the software code. Budgets for these activities can quickly become unsustainable. Resilient software design, automation of projected maintenance activities, and creation of interfaces to translate programming and auditing activities could potentially \u201clower the skill floor,\u201d allowing a smaller team of non-experts to substitute for many of the activities of much larger (and more costly) information technology teams.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Challenge_2:_Reducing_the_cost_of_technology\">Challenge 2: Reducing the cost of technology<\/span><\/h4>\n<p>The technical architecture to achieve data storage and processing can be very expensive, whether it exists on-premises or in <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud-based<\/a> systems. On-premises solutions have high start-up costs for processing and storage hardware. Operating a cloud-based system similar in size to Hyperion would be expected to cost about $35,000 per month, using Amazon Web Services as a benchmark.<sup id=\"rdp-ebb-cite_ref-11\" class=\"reference\"><a href=\"#cite_note-11\">[11]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Challenge_3:_Enhancing_user_autonomy\">Challenge 3: Enhancing user autonomy<\/span><\/h4>\n<p>Data users exist on a spectrum of familiarity and sophistication with data manipulation, ranging from users who want simple, intuitive visual summaries to those capable of sophisticated analysis of raw data. A data system needs to be accessible and usable across the spectrum of potential users within WCI, ideally without the need for manual creation of curated datasets and visuals. Users should be able to autonomously access the data and tools they need to answer their own queries as much as possible.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Challenge_4:_Optimizing_data_governance\">Challenge 4: Optimizing data governance<\/span><\/h4>\n<p>Patient data are subject to laws and policies\u2014such as provisions within the <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Insurance_Portability_and_Accountability_Act\" title=\"Health Insurance Portability and Accountability Act\" class=\"wiki-link\" data-key=\"b70673a0117c21576016cb7498867153\">Health Insurance Portability and Accountability Act<\/a> (HIPAA)\u2014enforcing high scrutiny and standards to maintain patient privacy rights. Integration of data governance policies and activities into the data management platform would facilitate adherence to the strict confidentiality guidelines of the health system. Additionally, a well-designed ticketing system can help users clarify their requests, ensure appropriate data usage, streamline ticket approval and completion, and permit ongoing user needs assessment.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Challenge_5:_Changing_technological_team_structure\">Challenge 5: Changing technological team structure<\/span><\/h4>\n<p>The culture of academic medical centers often promotes top-down decision-making, prioritizing a particular paradigm centered around clinicians. Although clinicians bring invaluable perspective to the design and usage of data systems, they are not typically trained in (or necessarily aware of) the specific technical skillset required to design sophisticated data management systems. Leadership structures with clinicians or administrators at the \u201ctop\u201d can hinder technical teams, limiting their autonomy to implement technical best practice decisions, and delaying even simple architectural and programming tasks. Flattening the decisional hierarchy and implementing a transdisciplinary team approach could optimize functioning and increase development speed, simulating the pace of industry teams.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>The custom-built data management solution for WCI, Hyperion, consists of a back-end relational Structured Query Language (SQL) database linked to a front-end interface platform accommodating multiple user types and tasks, including database administration, ticketing, reporting, and user dashboards. Table 1 compares Hyperion features to those of the most commonly available clinical data management software packages. Table 2 summarizes how Hyperion's design addresses the five identified challenges above.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Snyder_PLOSDigHlth22_1-11.png\" class=\"image wiki-link\" data-key=\"9aecfd75d58e0e5acbcda113372df3c4\"><img alt=\"Tab1 Snyder PLOSDigHlth22 1-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/02\/Tab1_Snyder_PLOSDigHlth22_1-11.png\" decoding=\"async\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tbody><tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tbody><tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><p><b>Table 1.<\/b> Comparison of features: Hyperion versus other commercially available, commonly used clinical data management software systems.<\/p><\/blockquote>\n<\/td><\/tr>\n<\/tbody><\/table>\n<\/td><\/tr><\/tbody><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab2_Snyder_PLOSDigHlth22_1-11.png\" class=\"image wiki-link\" data-key=\"2c4745e5d84e3980d05c1b84180a4705\"><img alt=\"Tab2 Snyder PLOSDigHlth22 1-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8e\/Tab2_Snyder_PLOSDigHlth22_1-11.png\" decoding=\"async\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tbody><tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tbody><tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><p><b>Table 2.<\/b> Design elements supporting identified challenges.<\/p><\/blockquote>\n<\/td><\/tr>\n<\/tbody><\/table>\n<\/td><\/tr><\/tbody><\/table>\n<h3><span class=\"mw-headline\" id=\"Hyperion_data_manager\">Hyperion data manager<\/span><\/h3>\n<p>Hyperion\u2019s \u201cnervous system\u201d is the Hyperion Data Manager (HDM), an interface and validation engine built utilizing the <a href=\"https:\/\/www.limswiki.org\/index.php\/Python_(programming_language)\" title=\"Python (programming language)\" class=\"wiki-link\" data-key=\"ef6905a29cbb75d3c71e6bdf6e2915dd\">Python programming language<\/a>. Via a front-end system utilizing a Flask-based graphical user interface (GUI), approved users access point-and-click tasks to import data sources; initiate and manage extract, transform, and load (ETL) procedures; create sandboxes; copy table structures; re-run interfaces; and check for errors. A built-in custom wizard permits users to set up data interfaces without any programming knowledge for most data sources; Hyperion's bidirectional interfaces can be easily configured for all common data formats and models (<a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7#Fast_Healthcare_Interoperability_Resources_.28FHIR.29\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"ce348e672e47598e4b42cbd03c292ec4\">HL7 FHIR<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Extensible_Markup_Language\" title=\"Extensible Markup Language\" class=\"wiki-link\" data-key=\"f7c17028e7fb39d8b39c6d31504411a8\">XML<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Application_programming_interface\" title=\"Application programming interface\" class=\"wiki-link\" data-key=\"36fc319869eba4613cb0854b421b0934\">application programming interface<\/a> [API], etc.).\n<\/p><p>For integrating any new data source, HDM reads the database schema for the new data source and presents approved administrative users with a field listing. Via the GUI, users can select what to import into the Hyperion platform, create table names, rename data elements, and select a data update interval. HDM creates a scheduled interface pull at the selected interval and can accommodate near-real-time intervals. HDM can read in the most utilized database technologies such as Microsoft\u2019s SQL Server and Oracle, and it also supports token authentication, which is utilized by multiple governmental data sources and common medical database software products such as REDCap. For EHR data, HDM has custom code to simplify and semantically define tables and interpret names for commonly used fields. It translates field names into a more readable form based on clinical naming conventions and uploads data directly at the specified intervals. This streamlines table design and reduces time delay for downstream needs, including ad-hoc report development, sandbox creation, and full-scale applications. As all data tables are normalized and explicitly defined within our system, data reusability across the system is also facilitated.\n<\/p><p>Ease of use, combined with robust procedures to ensure <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_quality\" title=\"Data quality\" class=\"wiki-link\" data-key=\"7fe43b05eae4dfa9b5c0547cc8cfcceb\">data quality<\/a>, has permitted rapid integration of multiple data sources for a variety of clinical, quality, operational, and research purposes (Fig. 1). To increase oversight of data validity and currency, HDM incorporates a complete validation and metrics monitoring package for data uploaded to Hyperion. HDM stores and displays metrics including number of new records per interface, timing and number of updates, and deletions or modifications to records since the prior interface update. Automated dashboards allow non-expert users to monitor metrics (Fig. 2); the dashboard displays are interactive, allowing point-and-click activities (e.g., adding or removing metrics to the display) and hovering over data points to get more information (see Figures A, B, and C in Supporting information, S1 Text for further examples).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Snyder_PLOSDigHlth22_1-11.png\" class=\"image wiki-link\" data-key=\"1019c23e3f597faeddf8174057158738\"><img alt=\"Fig1 Snyder PLOSDigHlth22 1-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/09\/Fig1_Snyder_PLOSDigHlth22_1-11.png\" decoding=\"async\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tbody><tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tbody><tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><p><b>Figure 1.<\/b> Current architecture of Wilmot Cancer Institute (WCI) data warehouse.<\/p><\/blockquote>\n<\/td><\/tr>\n<\/tbody><\/table>\n<\/td><\/tr><\/tbody><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Snyder_PLOSDigHlth22_1-11.png\" class=\"image wiki-link\" data-key=\"62e9ba3e51d46b4e7638ca431d96d3ce\"><img alt=\"Fig2 Snyder PLOSDigHlth22 1-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/36\/Fig2_Snyder_PLOSDigHlth22_1-11.png\" decoding=\"async\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tbody><tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tbody><tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><p><b>Figure 2.<\/b> HDM monitoring dashboard.<\/p><\/blockquote>\n<\/td><\/tr>\n<\/tbody><\/table>\n<\/td><\/tr><\/tbody><\/table>\n<p>HDM\u2019s validation routines regularly evaluate for data consistency issues such as field and data type mismatches and extreme shifts in table sizes that would indicate mass data deletion or insertion. Some validation routines are specifically applied to certain data sources. For example, correct address information is critical for analyses or visualizations involving geospatial elements (Fig 3). Incoming address data are validated against a database of extant addresses, and a \u201ccertainty percentage\u201d is calculated, which permits a user-defined accuracy threshold for certifying the address as correct. Validation routines are linked to a wizard, and programmers are not needed for review of their output. The wizard allows users to examine items or processes flagged as potentially incorrect by the validation routines, and it presents them with a point-and-click menu of options (e.g., flag for further review, manually correct item, or approve upload).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Snyder_PLOSDigHlth22_1-11.png\" class=\"image wiki-link\" data-key=\"1f477608fdf4dec471ca332b90c5b94b\"><img alt=\"Fig3 Snyder PLOSDigHlth22 1-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0c\/Fig3_Snyder_PLOSDigHlth22_1-11.png\" decoding=\"async\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tbody><tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tbody><tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><p><b>Figure 3.<\/b> CANcer Visual Analytic System (CANVAS) visual.<\/p><\/blockquote>\n<\/td><\/tr>\n<\/tbody><\/table>\n<\/td><\/tr><\/tbody><\/table>\n<p>To improve both computational and cost efficiency, HDM uses multi-threaded best practice approaches to parallelize processes and increase processing speed. Job duration calculations allow for efficient scheduling of multiple processor cores prior to task execution. Utilizing the combination of scheduling plus multi-threaded processing increased speed by 50\u201377% compared to non-optimized methods. Cost efficiency is further optimized using open-source technology (e.g., Python\/Flask) for all code. Hyperion imports millions of rows of data every few hours, with a total implementation cost of $1,500 for hardware and software (in addition to already existing enterprise licensure costs). This design also facilitates future implementation of analytic pipelines, including machine learning pipelines.\n<\/p><p>HDM sandboxes offer researchers temporary, partitioned access to curated datasets. Researchers can autonomously handle and analyze their data within a secure environment. Access is completely separated from other Hyperion architecture, and access privileges have an associated, pre-specified timeframe. Upon lapse of access privileges, HDM will terminate access, and archive and lock the data.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Hyperion_front_end\">Hyperion front end<\/span><\/h3>\n<p>The front-end platform also utilizes open-source programming technology (HTML5 and JavaScript) to limit cost and align with common programming skillsets. Via a secure web portal, each user has access to a curated set of dashboards and activities based upon their access profile. Dashboards are highly interactive and customized to user groups based on iterative feedback. Users can view and securely sign documents (such as the data governance policy) and submit requests and ideas via an integrated ticketing system.\n<\/p><p>Hyperion administration and security activities are accessible via the web portal and HTML5 interface for approved users; point-and-click tools enable addition of new users and assignment of access privileges. The system has been tested and functions with all major web browsers. System administrators can view applications utilization data, including granular data by user and by clicks. Real-time data monitoring in conjunction allows for all data use to be precisely tracked and audited.\n<\/p><p>Hyperion\u2019s analytics-processing framework enables real-time analytics across any data element in the system, including those accessed via APIs, e.g., imaging, pathology, and molecular\/genomic data. Process efficiencies as described above permit rapid turnaround of ad-hoc reports, and scheduled reports are automatically generated and delivered at set intervals. A key principle guiding design of the user experience, however, prioritizes user autonomy in data access, analysis, and visualization. To this end, multiple customized dashboards permit users to directly interact with curated datasets and visualizations; Table 3 gives examples of some dashboards developed for various user groups in WCI.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab3_Snyder_PLOSDigHlth22_1-11.png\" class=\"image wiki-link\" data-key=\"95cf9db75d301333cc93b9470ebf1b32\"><img alt=\"Tab3 Snyder PLOSDigHlth22 1-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0a\/Tab3_Snyder_PLOSDigHlth22_1-11.png\" decoding=\"async\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tbody><tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tbody><tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><p><b>Table 3.<\/b> Example user dashboards.<\/p><\/blockquote>\n<\/td><\/tr>\n<\/tbody><\/table>\n<\/td><\/tr><\/tbody><\/table>\n<p>Cross-referencing and validation of patient addresses (described above), as well as integrated data crosswalks between various geographic levels (e.g., zip code and census tract levels), facilitate geospatial visualization and analysis at any geographic level specified by the user. These capabilities enhance the ability to examine data stratified by area within the WCI catchment area, map changes over time, or link to other datasets to analyze disparities across the region (e.g., nutritional access or socioeconomic disadvantage. Hyperion\u2019s CANcer Visual Analytic System (CANVAS) allows users to create regional map overlays for data elements such as patient visits, diagnostic categories, and clinical trial accruals which can be toggled on\/off or superimposed (Fig. 3). Animations permit direct visualization of changes over time.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_governance\">Data governance<\/span><\/h3>\n<p>Data governance processes follow a centralized data governance model with a decentralized execution standard: this allows centralized control and authority to reside with the informatics team but permits the individual users and groups to be able to execute queries and analytics on curated sandboxed data sets or dashboards. This method of governance ensures data reliability and consistency via a single data baseline, validated daily.\n<\/p><p>Upon initial login, Hyperion users are presented with the current WCI data governance policy, which they must electronically read and sign prior to interacting further with the system (Fig. 4). The integrated governance platform requires new signatures at login when the policy is updated. All signatures are <a href=\"https:\/\/www.limswiki.org\/index.php\/Encryption\" title=\"Encryption\" class=\"wiki-link\" data-key=\"86a503652ed5cc9d8e2b0252a480b5e1\">encrypted<\/a> and stored with the documents in Hyperion. The platform can support multiple documents with multiple versions, allowing for customized documentation for different user types as required.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Snyder_PLOSDigHlth22_1-11.png\" class=\"image wiki-link\" data-key=\"557dd53f4e523f8231150ce86b1b10d4\"><img alt=\"Fig4 Snyder PLOSDigHlth22 1-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3b\/Fig4_Snyder_PLOSDigHlth22_1-11.png\" decoding=\"async\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tbody><tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tbody><tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><p><b>Figure 4.<\/b> Custom electronic signature platform presenting the data governance policy at login.<\/p><\/blockquote>\n<\/td><\/tr>\n<\/tbody><\/table>\n<\/td><\/tr><\/tbody><\/table>\n<p>Hyperion has an integrated custom ticketing platform for users to submit requests for additional development, reporting, or other requests. This is the primary method for initial communication of needs to the WCI Informatics Team and the Data Governance Committee. This method limits user cognitive load, streamlines development process, and facilitates automated tracking and reporting of requests. The ticketing platform collects all relevant info from users and populates a document for committee discussion, triage, and project management. Tickets are automatically routed to individuals on the governing committee for initial approval; if more <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> is needed, the committee member can submit questions back to the system via embedded email links. For items with limited time and scope requirements, the team can independently triage the request and place it appropriately within the project queue. Larger requests (e.g., >40 person-hours, complexity, and\/or multiple stakeholder groups affected) are flagged for committee discussion. Users can track the progress and actions on their individual tickets, including the ability to view key details entered by the informatics team (such as committee approvals, relevant dates, and requests for further information).\n<\/p><p>The Data Governance Committee consists of the WCI Informatics team and representatives from clinical operations, quality, administration, and research, all of whom have separate leadership roles within their respective domains. The Committee meets monthly and determines overall mission and priorities, discusses and triages large project requests, and reviews data usage and security.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Security\">Security<\/span><\/h3>\n<p>Hyperion is only accessible via a secure networked computer or institutional virtual private network (VPN). Hyperion\u2019s custom security module augments the institution\u2019s Active Directory (AD) authentication and allows for user auditing and access control down to the individual application level. This permits user access to be customized by role and job function, and allows for the capture of full details on each user action, which Hyperion logs and monitors. Secure sandboxes are provided to users for data analytics purposes, without the ability to download data; this method of viewing and analyzing Hyperion reports is strongly encouraged. Otherwise, requested data documents are sent via either end-to-end encrypted email or secure file transfer protocol (SFTP). All documents able to be downloaded by the user include <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a> tags to identify the source of download, the individual user, and date\/time stamps. In the event of a data consistency issue, the document can be compared to the logged SQL <a href=\"https:\/\/www.limswiki.org\/index.php\/Audit_trail\" title=\"Audit trail\" class=\"wiki-link\" data-key=\"96a617b543c5b2f26617288ba923c0f0\">audit calls<\/a> stored within Hyperion to ensure data was not altered. Request fulfillment for patient-identifiable data meets all institutional data security policies, including review by the Data Governance Committee.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Usage_metrics\">Usage metrics<\/span><\/h3>\n<p>From January 2020 through December 2021, Hyperion has processed >41 billion records. More than 174 million records are currently stored, and 791 million records have been updated since January 2020. Hyperion currently has 148 unique users (52% physicians, 21% nursing, 17% IT, 10% administrators) accessing an average of 27 pages per day through December 2021. Table 4 shows the most accessed dashboards. There are currently 25 customized real-time updating dashboards in Hyperion, as well as 13 automated reports that distribute throughout the week to various departments and individuals via automated email distribution. The average turnaround time for an ad-hoc reporting request is three days. The average time to deploy a new dashboard is about four weeks.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab4_Snyder_PLOSDigHlth22_1-11.png\" class=\"image wiki-link\" data-key=\"b99c80d9c93d64b58a6e4d0f0261fff2\"><img alt=\"Tab4 Snyder PLOSDigHlth22 1-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/38\/Tab4_Snyder_PLOSDigHlth22_1-11.png\" decoding=\"async\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tbody><tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tbody><tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><p><b>Table 4.<\/b> Most accessed dashboards, January 2020\u2013December 2021.<\/p><\/blockquote>\n<\/td><\/tr>\n<\/tbody><\/table>\n<\/td><\/tr><\/tbody><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>Creating and maintaining complex, secure, and high-volume data warehouses, processing and assembling data views, and interfacing new data sources represent significant challenges for academic healthcare organizations, even those with adequate information technology (IT) staffing and budget. In typical practice, the team (or vendor) that creates the data warehousing software is distinct from the team tasked with maintaining it and supporting data access and visualization activities. Even after deployment of the data warehousing software, basic IT support for data warehousing typically includes interface developers, application developers, business analysts, business intelligence developers, security professionals, and help desk staff. Iterative adaptation of the software to meet changing data needs can be challenging or impossible, leading to the accrual of \u201cworkarounds\u201d and technical debt.\n<\/p><p>In WCI, we assembled a small, transdisciplinary team to develop a customized, adaptable, and scalable data management approach, supported by senior leadership and enterprise IT structures. Beginning with the definition and elaboration of key challenges we wished to address, we designed and built a modular and scalable software package addressing data storage, validation, and processing needs as well as data monitoring, access, and visualization. These structures were designed to permit rapid iteration and adaptation by a small but highly skilled technical team when needed, but to allow basic administration and continued maintenance to be performed by non-technical staff. The Hyperion database architecture, Hyperion Data Manager, security module, and governance modules were designed and deployed with a six-month timeline, and the entire package can be maintained by a single full-time equivalent (FTE). The modular extensible approach significantly reduces enhancement and update times. Limiting the continuous need for high-level technical skillsets to maintain software and data integrity frees a smaller team to work \u201cat the top of their cognitive skillsets,\u201d iterating solutions in response to user feedback, and creatively generating new solutions to complex data problems in WCI. This approach maximizes cost-effectiveness in addition to overall efficiency. Furthermore, Hyperion aligns with FAIR data principles (findability, accessibility, interoperability, and reusability).<sup id=\"rdp-ebb-cite_ref-12\" class=\"reference\"><a href=\"#cite_note-12\">[12]<\/a><\/sup>\n<\/p><p>Hyperion has high user uptake, with many faculty members, staff, administrators, and others logging in daily to independently access data for clinical, operational, administrative, and research purposes. Although much of the data in Hyperion originates within the institutional EHR, it has previously been sequestered within individual patient records and only slightly more accessible to automated extraction methods than paper charts. Hyperion makes validated, curated, organized, near-real-time datasets automatically accessible and easily explorable via an interactive suite of analysis and visualization tools. Quality initiatives, program development, physician decision-making, clinical trial planning and management, research (grant applications and peer-reviewed analyses), clinical operations management, and more are all supported within a single management platform.\n<\/p><p>In addition to the design and building of the software elements, implementation of Hyperion required careful consideration and design of support structures to address data governance, security, and project management. Intrinsic software elements such as the data governance policy tracking, security audits, and the sophisticated ticketing system are embedded in a set of processes overseen by a supporting committee structure consisting of key stakeholders and meeting at least monthly. Users are not competing with requests external to WCI for attention, and two-way communication is streamlined and efficient due to the embedded nature of the informatics team.\n<\/p><p>Beyond embedding a responsive team within and among the software users, the composition and functioning of the team are substantial revisions to the usual model of academic IT teams. A co-directed model shares formal authority for external interactions between a technical expert with extensive background in healthcare IT, and a practicing clinician with training in data science and informatics. Internally, the team is structured with a flattened decisional hierarchy, a deliberate emphasis on diversity of opinion and perspective, and a rapidly iterative approach to problem solving adapted from industry. Projects are managed through a combination of agile-based approaches and more traditional project management philosophies such as milestone phase gating. Although this structure is not able to completely shield the team from institutional bureaucracy, it has helped to create and sustain space for innovation and creativity within a traditionally cautious and even inert system.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Drawbacks_and_considerations\">Drawbacks and considerations<\/span><\/h3>\n<p>There are several potential disadvantages to our strategy for addressing data needs within an academic cancer center. Academic medical centers may not be able or willing to support in-house software development for patient data, relying on outsourced software to guarantee robustness, security, and technical support. In the current market, the technical skillsets to achieve customized software solutions for managing patient data are rare and expensive, and achieving buy-in to budget for these positions may be very challenging. We have mitigated personnel costs with a smaller team comprised of individuals with cross-functional skillsets, but this poses its own difficulty: there is limited redundancy within the system to accommodate team member absences or departures. Although the software design offloads much of the technical maintenance, allowing it to be done by nontechnical personnel, the sustainability of the software still relies on a core highly-skilled technical team. However, stable teams are also required to internally support many vendor products within the technical ecosystem.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion_and_future_work\">Conclusion and future work<\/span><\/h2>\n<p>In summary, Hyperion has surmounted large challenges in working with healthcare data to merge, organize, validate, and package data for use in multiple applications, including interactive user dashboards. Additional design considerations included lowering the skill floor for interaction with and maintenance of the software, reducing costs, and encouraging user autonomy. Development of data governance and other support structures, as well as discussions about team functioning and structure, occurred in parallel with the software build. Today, Hyperion provides a flexible, reliable, and scalable foundation for the development of responsive and innovative applications to support the mission and goals of an academic cancer center.\n<\/p><p>Future work includes turning our attention to further supporting data analytics, including <a href=\"https:\/\/www.limswiki.org\/index.php\/Machine_learning\" title=\"Machine learning\" class=\"wiki-link\" data-key=\"79aab39cfa124c958cd1dbcab3dde122\">machine learning<\/a> (ML). An ML pipeline is being developed to allow users to explore their data using advanced techniques while also receiving hands-on education about the potential and pitfalls of these methods. Novel data visualization methods, including augmented and virtual reality methods (AR\/VR) are also in initial development and testing. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supporting_information\">Supporting information<\/span><\/h2>\n<ul><li><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/doi.org\/10.1371\/journal.pdig.0000036.s001\" target=\"_blank\">S1 Text<\/a> (.docx): <b>Fig A.<\/b> Nursing dashboard. <b>Fig B.<\/b> Example of Provider Dashboard landing page, with interactive features (hover-over pop-up text and ability to click on bar graphs to \u201cdrill down\u201d on data). <b>Fig C.<\/b> Clinical Trial dashboard main page, with interactive features.<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We acknowledge the continued support of the Wilmot Cancer Institute\u2019s leadership in encouraging innovation, as well as the support of the physicians, nurses, and staff.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>ER is supported by the National Cancer Institute (K08CA248721) and the National Institute on Aging (R03AG067977). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_availability\">Data availability<\/span><\/h3>\n<p>All data are in the manuscript and\/or supporting information files.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors have declared that no competing interests exist.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<div class=\"mw-references-wrap mw-references-columns\"><ol class=\"references\">\n<li id=\"cite_note-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-1\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Olsen, LeighAnne; Aisner, Dara; McGinnis, J. Michael <i>et al.<\/i>, eds. (2007). \"Institute of Medicine Roundtable on Evidence-Based Medicine\". <i>The learning healthcare system: workshop summary<\/i>. Washington, DC: National Academies Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 978-0-309-10300-8.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Institute+of+Medicine+Roundtable+on+Evidence-Based+Medicine&rft.atitle=The+learning+healthcare+system%3A+workshop+summary&rft.date=2007&rft.place=Washington%2C+DC&rft.pub=National+Academies+Press&rft.isbn=978-0-309-10300-8&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-:0-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-:0_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-:0_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chaudhry, Basit; Wang, Jerome; Wu, Shinyi; Maglione, Margaret; Mojica, Walter; Roth, Elizabeth; Morton, Sally C.; Shekelle, Paul G. (16 May 2006). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/16702590\" target=\"_blank\">\"Systematic review: impact of health information technology on quality, efficiency, and costs of medical care\"<\/a>. <i>Annals of Internal Medicine<\/i> <b>144<\/b> (10): 742\u2013752. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7326%2F0003-4819-144-10-200605160-00125\" target=\"_blank\">10.7326\/0003-4819-144-10-200605160-00125<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Serial_Number\" data-key=\"a5dec3e4d005e654c29ad167ab53f53a\">ISSN<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.worldcat.org\/issn\/1539-3704\" target=\"_blank\">1539-3704<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16702590\" target=\"_blank\">16702590<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/16702590\" target=\"_blank\">https:\/\/pubmed.ncbi.nlm.nih.gov\/16702590<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Systematic+review%3A+impact+of+health+information+technology+on+quality%2C+efficiency%2C+and+costs+of+medical+care&rft.jtitle=Annals+of+Internal+Medicine&rft.aulast=Chaudhry&rft.aufirst=Basit&rft.au=Chaudhry%2C%26%2332%3BBasit&rft.au=Wang%2C%26%2332%3BJerome&rft.au=Wu%2C%26%2332%3BShinyi&rft.au=Maglione%2C%26%2332%3BMargaret&rft.au=Mojica%2C%26%2332%3BWalter&rft.au=Roth%2C%26%2332%3BElizabeth&rft.au=Morton%2C%26%2332%3BSally+C.&rft.au=Shekelle%2C%26%2332%3BPaul+G.&rft.date=16+May+2006&rft.volume=144&rft.issue=10&rft.pages=742%E2%80%93752&rft_id=info:doi\/10.7326%2F0003-4819-144-10-200605160-00125&rft.issn=1539-3704&rft_id=info:pmid\/16702590&rft_id=https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F16702590&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-3\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lyman, Jason A.; Scully, Kenneth; Harrison, James H. (1 March 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/18194718\" target=\"_blank\">\"The development of health care data warehouses to support data mining\"<\/a>. <i>Clinics in Laboratory Medicine<\/i> <b>28<\/b> (1): 55\u201371, vi. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cll.2007.10.003\" target=\"_blank\">10.1016\/j.cll.2007.10.003<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Serial_Number\" data-key=\"a5dec3e4d005e654c29ad167ab53f53a\">ISSN<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.worldcat.org\/issn\/0272-2712\" target=\"_blank\">0272-2712<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18194718\" target=\"_blank\">18194718<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/18194718\" target=\"_blank\">https:\/\/pubmed.ncbi.nlm.nih.gov\/18194718<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+development+of+health+care+data+warehouses+to+support+data+mining&rft.jtitle=Clinics+in+Laboratory+Medicine&rft.aulast=Lyman&rft.aufirst=Jason+A.&rft.au=Lyman%2C%26%2332%3BJason+A.&rft.au=Scully%2C%26%2332%3BKenneth&rft.au=Harrison%2C%26%2332%3BJames+H.&rft.date=1+March+2008&rft.volume=28&rft.issue=1&rft.pages=55%E2%80%9371%2C+vi&rft_id=info:doi\/10.1016%2Fj.cll.2007.10.003&rft.issn=0272-2712&rft_id=info:pmid\/18194718&rft_id=https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F18194718&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-4\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sheta, Osama El-Sayed; Eldeen, Ahmed Nour (2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/arxiv.org\/abs\/1211.4371\" target=\"_blank\">\"Building a health care data warehouse for cancer diseases\"<\/a>. <i>arXiv<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.48550%2FARXIV.1211.4371\" target=\"_blank\">10.48550\/ARXIV.1211.4371<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/arxiv.org\/abs\/1211.4371\" target=\"_blank\">https:\/\/arxiv.org\/abs\/1211.4371<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+a+health+care+data+warehouse+for+cancer+diseases&rft.jtitle=arXiv&rft.aulast=Sheta&rft.aufirst=Osama+El-Sayed&rft.au=Sheta%2C%26%2332%3BOsama+El-Sayed&rft.au=Eldeen%2C%26%2332%3BAhmed+Nour&rft.date=2012&rft_id=info:doi\/10.48550%2FARXIV.1211.4371&rft_id=https%3A%2F%2Farxiv.org%2Fabs%2F1211.4371&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-5\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ewen, Edward F.; Medsker, Carl E.; Dusterhoft, Laura E. (1 November 1998). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dl.acm.org\/doi\/10.1145\/294260.294271\" target=\"_blank\">\"Data warehousing in an integrated health system: building the business case\"<\/a> (in en). <i>Proceedings of the 1st ACM international workshop on Data warehousing and OLAP<\/i> (Washington D.C. USA: ACM): 47\u201353. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F294260.294271\" target=\"_blank\">10.1145\/294260.294271<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 978-1-58113-120-8<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/dl.acm.org\/doi\/10.1145\/294260.294271\" target=\"_blank\">https:\/\/dl.acm.org\/doi\/10.1145\/294260.294271<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+warehousing+in+an+integrated+health+system%3A+building+the+business+case&rft.jtitle=Proceedings+of+the+1st+ACM+international+workshop+on+Data+warehousing+and+OLAP&rft.aulast=Ewen&rft.aufirst=Edward+F.&rft.au=Ewen%2C%26%2332%3BEdward+F.&rft.au=Medsker%2C%26%2332%3BCarl+E.&rft.au=Dusterhoft%2C%26%2332%3BLaura+E.&rft.date=1+November+1998&rft.pages=47%E2%80%9353&rft.place=Washington+D.C.+USA&rft.pub=ACM&rft_id=info:doi\/10.1145%2F294260.294271&rft.isbn=978-1-58113-120-8&rft_id=https%3A%2F%2Fdl.acm.org%2Fdoi%2F10.1145%2F294260.294271&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-6\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Harris, Paul A.; Taylor, Robert; Thielke, Robert; Payne, Jonathon; Gonzalez, Nathaniel; Conde, Jose G. (1 April 2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/18929686\" target=\"_blank\">\"Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support\"<\/a>. <i>Journal of Biomedical Informatics<\/i> <b>42<\/b> (2): 377\u2013381. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2008.08.010\" target=\"_blank\">10.1016\/j.jbi.2008.08.010<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Serial_Number\" data-key=\"a5dec3e4d005e654c29ad167ab53f53a\">ISSN<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.worldcat.org\/issn\/1532-0480\" target=\"_blank\">1532-0480<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/2700030\/\" target=\"_blank\">2700030<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18929686\" target=\"_blank\">18929686<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/18929686\" target=\"_blank\">https:\/\/pubmed.ncbi.nlm.nih.gov\/18929686<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Research+electronic+data+capture+%28REDCap%29--a+metadata-driven+methodology+and+workflow+process+for+providing+translational+research+informatics+support&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Harris&rft.aufirst=Paul+A.&rft.au=Harris%2C%26%2332%3BPaul+A.&rft.au=Taylor%2C%26%2332%3BRobert&rft.au=Thielke%2C%26%2332%3BRobert&rft.au=Payne%2C%26%2332%3BJonathon&rft.au=Gonzalez%2C%26%2332%3BNathaniel&rft.au=Conde%2C%26%2332%3BJose+G.&rft.date=1+April+2009&rft.volume=42&rft.issue=2&rft.pages=377%E2%80%93381&rft_id=info:doi\/10.1016%2Fj.jbi.2008.08.010&rft.issn=1532-0480&rft_id=info:pmc\/2700030&rft_id=info:pmid\/18929686&rft_id=https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F18929686&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-7\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/clinicaltrials.gov\/\" target=\"_blank\">\"ClinicalTrials.gov\"<\/a>. U.S. National Library of Medicine<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/clinicaltrials.gov\/\" target=\"_blank\">https:\/\/clinicaltrials.gov\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 July 2022<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ClinicalTrials.gov&rft.atitle=&rft.pub=U.S.+National+Library+of+Medicine&rft_id=https%3A%2F%2Fclinicaltrials.gov%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-8\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Project Management Institute, ed. (2017). <i>A guide to the project management body of knowledge \/ Project Management Institute<\/i>. PMBOK guide (Sixth edition ed.). Newtown Square, PA: Project Management Institute. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 978-1-62825-184-5.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=A+guide+to+the+project+management+body+of+knowledge+%2F+Project+Management+Institute&rft.date=2017&rft.series=PMBOK+guide&rft.edition=Sixth+edition&rft.place=Newtown+Square%2C+PA&rft.pub=Project+Management+Institute&rft.isbn=978-1-62825-184-5&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-9\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wisniewski, Mary F.; Kieszkowski, Piotr; Zagorski, Brandon M.; Trick, William E.; Sommers, Michael; Weinstein, Robert A. (2003). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/12807807\" target=\"_blank\">\"Development of a clinical data warehouse for hospital infection control\"<\/a>. <i>Journal of the American Medical Informatics Association: JAMIA<\/i> <b>10<\/b> (5): 454\u2013462. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M1299\" target=\"_blank\">10.1197\/jamia.M1299<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Serial_Number\" data-key=\"a5dec3e4d005e654c29ad167ab53f53a\">ISSN<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.worldcat.org\/issn\/1067-5027\" target=\"_blank\">1067-5027<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC212782\/\" target=\"_blank\">PMC212782<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12807807\" target=\"_blank\">12807807<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/12807807\" target=\"_blank\">https:\/\/pubmed.ncbi.nlm.nih.gov\/12807807<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+of+a+clinical+data+warehouse+for+hospital+infection+control&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association%3A+JAMIA&rft.aulast=Wisniewski&rft.aufirst=Mary+F.&rft.au=Wisniewski%2C%26%2332%3BMary+F.&rft.au=Kieszkowski%2C%26%2332%3BPiotr&rft.au=Zagorski%2C%26%2332%3BBrandon+M.&rft.au=Trick%2C%26%2332%3BWilliam+E.&rft.au=Sommers%2C%26%2332%3BMichael&rft.au=Weinstein%2C%26%2332%3BRobert+A.&rft.date=2003&rft.volume=10&rft.issue=5&rft.pages=454%E2%80%93462&rft_id=info:doi\/10.1197%2Fjamia.M1299&rft.issn=1067-5027&rft_id=info:pmc\/PMC212782&rft_id=info:pmid\/12807807&rft_id=https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F12807807&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-10\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rubin, Daniel L.; Desser, Terry S. (1 March 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/18312970\" target=\"_blank\">\"A data warehouse for integrating radiologic and pathologic data\"<\/a>. <i>Journal of the American College of Radiology: JACR<\/i> <b>5<\/b> (3): 210\u2013217. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jacr.2007.09.004\" target=\"_blank\">10.1016\/j.jacr.2007.09.004<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Serial_Number\" data-key=\"a5dec3e4d005e654c29ad167ab53f53a\">ISSN<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.worldcat.org\/issn\/1558-349X\" target=\"_blank\">1558-349X<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18312970\" target=\"_blank\">18312970<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/18312970\" target=\"_blank\">https:\/\/pubmed.ncbi.nlm.nih.gov\/18312970<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+data+warehouse+for+integrating+radiologic+and+pathologic+data&rft.jtitle=Journal+of+the+American+College+of+Radiology%3A+JACR&rft.aulast=Rubin&rft.aufirst=Daniel+L.&rft.au=Rubin%2C%26%2332%3BDaniel+L.&rft.au=Desser%2C%26%2332%3BTerry+S.&rft.date=1+March+2008&rft.volume=5&rft.issue=3&rft.pages=210%E2%80%93217&rft_id=info:doi\/10.1016%2Fj.jacr.2007.09.004&rft.issn=1558-349X&rft_id=info:pmid\/18312970&rft_id=https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F18312970&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-11\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/calculator.aws\/\" target=\"_blank\">\"AWS Pricing Calculator\"<\/a>. Amazon Web Services<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/calculator.aws\/\" target=\"_blank\">https:\/\/calculator.aws\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 November 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=AWS+Pricing+Calculator&rft.atitle=&rft.pub=Amazon+Web+Services&rft_id=https%3A%2F%2Fcalculator.aws%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<li id=\"cite_note-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-12\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, I. Jsbrand Jan; Appleton, Gabrielle; Axton, Myles; Baak, Arie; Blomberg, Niklas; Boiten, Jan-Willem <i>et al.<\/i> (15 March 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/26978244\" target=\"_blank\">\"The FAIR Guiding Principles for scientific data management and stewardship\"<\/a>. <i>Scientific Data<\/i> <b>3<\/b>: 160018. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fsdata.2016.18\" target=\"_blank\">10.1038\/sdata.2016.18<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Serial_Number\" data-key=\"a5dec3e4d005e654c29ad167ab53f53a\">ISSN<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.worldcat.org\/issn\/2052-4463\" target=\"_blank\">2052-4463<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/4792175\/\" target=\"_blank\">4792175<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26978244\" target=\"_blank\">26978244<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/26978244\" target=\"_blank\">https:\/\/pubmed.ncbi.nlm.nih.gov\/26978244<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+FAIR+Guiding+Principles+for+scientific+data+management+and+stewardship&rft.jtitle=Scientific+Data&rft.aulast=Wilkinson&rft.aufirst=Mark+D.&rft.au=Wilkinson%2C%26%2332%3BMark+D.&rft.au=Dumontier%2C%26%2332%3BMichel&rft.au=Aalbersberg%2C%26%2332%3BI.+Jsbrand+Jan&rft.au=Appleton%2C%26%2332%3BGabrielle&rft.au=Axton%2C%26%2332%3BMyles&rft.au=Baak%2C%26%2332%3BArie&rft.au=Blomberg%2C%26%2332%3BNiklas&rft.au=Boiten%2C%26%2332%3BJan-Willem&rft.au=da+Silva+Santos%2C%26%2332%3BLuiz+Bonino&rft.date=15+March+2016&rft.volume=3&rft.pages=160018&rft_id=info:doi\/10.1038%2Fsdata.2016.18&rft.issn=2052-4463&rft_id=info:pmc\/4792175&rft_id=info:pmid\/26978244&rft_id=https%3A%2F%2Fpubmed.ncbi.nlm.nih.gov%2F26978244&rfr_id=info:sid\/en.wikipedia.org:Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\"><span style=\"display: none;\"> <\/span><\/span>\n<\/span>\n<\/li>\n<\/ol><\/div><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and spelling. In some cases important information was missing from the references, and that information was added. The original includes both an abstract and an author summary, which is confusing since the purpose of the asbtract is to act as a summary of the text; for this version, the two were combined to form an intelligible, coherent abstract.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20230501181627\nCache expiry: 86400\nDynamic content: false\nComplications: []\nCPU time usage: 1.103 seconds\nReal time usage: 1.553 seconds\nPreprocessor visited node count: 13501\/1000000\nPost\u2010expand include size: 109417\/2097152 bytes\nTemplate argument size: 32334\/2097152 bytes\nHighest expansion depth: 25\/40\nExpensive parser function count: 0\/100\nUnstrip recursion depth: 0\/20\nUnstrip post\u2010expand size: 27715\/5000000 bytes\n-->\n<!--\nTransclusion expansion time report (%,ms,calls,template)\n100.00% 639.476 1 -total\n 82.26% 526.013 1 Template:Reflist\n 50.50% 322.912 12 Template:Citation\/core\n 29.56% 189.055 2 Template:Cite_book\n 28.78% 184.013 8 Template:Cite_journal\n 13.99% 89.460 1 Template:Infobox_journal_article\n 12.55% 80.268 1 Template:Infobox\n 9.91% 63.392 10 Template:Date\n 8.93% 57.090 26 Template:Citation\/identifier\n 6.15% 39.348 2 Template:Cite_web\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:14039-0!canonical and timestamp 20230501181626 and revision id 51450. Serialized with JSON.\n -->\n<\/div><\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care\">https:\/\/www.limswiki.org\/index.php\/Journal:From_months_to_minutes:_Creating_Hyperion,_a_novel_data_management_system_expediting_data_insights_for_oncology_research_and_patient_care<\/a><\/div>\n<!-- end content --><div class=\"visualClear\"><\/div><\/div><\/div><div class=\"visualClear\"><\/div><\/div><!-- end of the left (by default at least) column --><div class=\"visualClear\"><\/div><\/div>\n\n\n\n<\/body>","0f0e755c0f65e74ef81e3430d5482779_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/02\/Tab1_Snyder_PLOSDigHlth22_1-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8e\/Tab2_Snyder_PLOSDigHlth22_1-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/09\/Fig1_Snyder_PLOSDigHlth22_1-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/36\/Fig2_Snyder_PLOSDigHlth22_1-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0c\/Fig3_Snyder_PLOSDigHlth22_1-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0a\/Tab3_Snyder_PLOSDigHlth22_1-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3b\/Fig4_Snyder_PLOSDigHlth22_1-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/38\/Tab4_Snyder_PLOSDigHlth22_1-11.png"],"0f0e755c0f65e74ef81e3430d5482779_timestamp":1682966963,"fd0ab2ce71ffc5ec7632fce8449ce6f2_type":"article","fd0ab2ce71ffc5ec7632fce8449ce6f2_title":"Digitalization of calibration data management in the pharmaceutical industry using a multitenant platform (Mustap\u00e4\u00e4 et al. 2022)","fd0ab2ce71ffc5ec7632fce8449ce6f2_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Digitalization_of_calibration_data_management_in_the_pharmaceutical_industry_using_a_multitenant_platform","fd0ab2ce71ffc5ec7632fce8449ce6f2_plaintext":"\n\nJournal:Digitalization of calibration data management in the pharmaceutical industry using a multitenant platformFrom LIMSWikiJump to navigationJump to searchFull article title\n \nDigitalization of calibration data management in the pharmaceutical industry using a multitenant platformJournal\n \nApplied SciencesAuthor(s)\n \nMustap\u00e4\u00e4, Tuukka; Nummiluikki, Juho; Viitala, RaineAuthor affiliation(s)\n \nAalto University, Beamex Oy AbPrimary contact\n \nEmail: tuukka dot mustapaa at aalto dot fiEditors\n \nMendyk, AleksanderYear published\n \n2022Volume and issue\n \n12(15)Article #\n \n7531DOI\n \n10.3390\/app12157531ISSN\n \n2076-3417Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/2076-3417\/12\/15\/7531Download\n \nhttps:\/\/www.mdpi.com\/2076-3417\/12\/15\/7531\/pdf (PDF)\n\nContents \n\n1 Abstract \n2 Introduction \n3 Background \n\n3.1 Metrology infrastructure as a part of the quality infrastructure \n3.2 IoT in pharmaceutical and process industries \n3.3 Requirements for the calibration data management and data formats \n\n3.3.1 General guidelines and good practices \n3.3.2 Mass as an example measurand \n3.3.3 Requirements for data integrity \n3.3.4 Harmonization challenges for developing digital solutions \n\n\n\n\n4 Materials and methods \n\n4.1 Studied cases for the proof of concept \n\n4.1.1 Case 1 \n4.1.2 Case 2 \n\n\n4.2 Calibration and data management processes \n4.3 Digitalizing calibration data exchange and management \n\n4.3.1 Calibration provider \n4.3.2 Calibration customer \n\n\n4.4 Aims and requirements for the POC DCC platform \n\n\n5 Results \n\n5.1 Data transfer utilizing the POC platform \n5.2 Features for a calibration provider \n5.3 Features for a calibration customer \n5.4 Testing and validation of the platform-based calibration data exchange and management processes \n\n\n6 Discussion \n\n6.1 Challenges in advancing the digitalization of the metrology infrastructure \n6.2 Future work and research topics \n\n\n7 Conclusions \n8 Abbreviations, acronyms, and initialisms \n9 Acknowledgements \n\n9.1 Author contributions \n9.2 Funding \n9.3 Conflicts of interest \n\n\n10 References \n11 Notes \n\n\n\nAbstract \nThe global quality infrastructure (QI) has been established and is maintained to ensure the safety of products and services for their users. One of the cornerstones of the QI is metrology, i.e., the science of measurement, as quality management systems (QMS) commonly rely on measurements for evaluating quality. For this reason, calibration procedures and management of the data related to them are of the utmost importance for quality management in the process industry, made a particularly high priority by regulatory authorities. To overcome the relatively low level of digitalization in metrology, machine-interpretable data formats such as digital calibration certificates (DCC) are being developed. \nIn this paper, we analyze the current calibration processes in the pharmaceutical industry, and the requirements defined for them in the relevant standards and regulations. For digitalizing the calibration-related data exchange, a multitenant cloud- and platform-based method is presented. To test and validate the approach, a proof of concept (POC) implementation of the platform is developed with a focus on ease and cost-efficiency of deployment and use, while also ensuring the preservation of traceability and data integrity. The POC is based on two industrial use cases involving organizations with different roles in the metrology infrastructure. From testing this POC, the presented approach proves to be an efficient method for organizing the calibration data exchange in industrial use.\nKeywords: digitalization, quality management, pharmaceutical industry, digital calibration certificate, metrology, traceability, platform\n\nIntroduction \nWhenever a product or service is brought to market, it is crucial that the product or service in question is safe for consumers and the environment. For these purposes, a global quality infrastructure (QI) consisting of both public and private organizations has been established.[1] The global cooperation of the QI is coordinated by the International Network on Quality Infrastructure (INetQI)[2], which defines QI as \"the system comprising the organizations (public and private) together with the policies, relevant legal and regulatory framework, and practices needed to support and enhance the quality, safety and environmental soundness of goods, services and processes.\"[1] The cornerstones of QI are metrology, standardization, accreditation, conformity assessment, and market surveillance, which have their own sub-infrastructures, such as metrology infrastructure (MI), which is tied to its own international organizations.[1][3] Due to the differences in roles of the organizations and regulatory frameworks in the QI, there are organizations and regulatory bodies covering the different parts of the QI on national or regional levels. In turn, due to this division, there are possibilities for overlapping or conflicting regulations, which regulatory bodies ideally aim to avoid. \nCurrently, as significant efforts are being made to digitalize the QI[1][4][5][6][7], the differences in current practices and requirements quickly become apparent, causing challenges in the implementation of digital solutions. For ensuring that the data formats and digital processes meet their respective requirements and are applicable globally in different applications, harmonization is needed, as the current requirements may vary by the domain or region. Otherwise, it could be impossible, or at least highly cost-inefficient, for the organizations participating in the maintenance of the infrastructures, such as calibration laboratories and service providers, to adapt their services to be interoperable with the varying systems used by individual customers.\nIn this paper, we focus on the industrial part of the MI, in which the determining of the measurement uncertainty of individual instruments and traceability of measurements to the measurement unit system are established through calibrations.[8] As calibrations are a confirmative part of the quality management of processes that are based on measurements, they are not directly profitable operations but instead are necessary for avoiding prohibitive expenses and delays or fulfilling compliance requirements, allowing access to markets. For this reason, estimating the total economic benefits of the digital transformation of the MI for all the involved organizations individually is complex. \nThe same also applies to other similar processes in different parts of the QI. As the purpose of the QI, and thus also the MI, is mainly to oversee and support the industry in providing reliable and safe products and services to customers[1], most economical benefits from the digital transformation are formed in the customer end of the infrastructure. In the MI, this includes, e.g., manufacturing companies that can have several thousands of measurement instruments monitoring and controlling production lines and processes. For these companies, a significant part of the value of digitalization comes from the improvements in efficiency and reduced need for manual work through automation, which leads to savings from reduced human resources tied to quality management. In the case of calibrations, where the costs and possible savings from managing instrument data and information are relative to the number of the instruments that need to be regularly calibrated, these benefits will not be as significant for smaller service providers. This means that any investments in the digital transformation become difficult to justify without support and demand from the customers.\nA good example of an industry that is reliant on MI is the pharmaceutical industry, where measurements provide the means for controlling the drug manufacturing processes. Thus, the quality and safety of pharmaceutical products are directly dependent on the reliability of process measurements. For this reason, ensuring the trustworthiness of the process instruments is an essential part of quality management in the pharmaceutical industry, which is why the measurement systems and their maintenance and calibration procedures are highly regulated.[9][10][11] Ensuring the data integrity of the measurements and any calibration-related data in particular is of the utmost importance for this purpose, which is why detailed records and audit trails for the processes are required.[12] Consequently, any processes including human operations, e.g., related to the documentation and handling of calibration data, will typically require several inspection and approval procedures to prevent the occurrence of human errors.\nSince harmonization of data formats is essential for the interoperability and overall efficiency of digital systems, important areas requiring in-depth examination are the established standards, regulations, and guidelines that provide the framework for the current data formats and processes. Thus, a key question for the success of digitalization efforts becomes, \"how compatible and applicable are the requirements defined for the different parties in QI in a fully digital environment?\"\nIn general, the success of industry-wide transition in the digital environment relies on organizational capabilities to adapt to and uptake new technologies and systems when necessary. Thus, an important aspect of the digital transformation is ensuring the ease of adaptation, inexpensiveness, and sufficient scalability of the processes and systems so that the requirements of various types and sizes of organizations can be fulfilled. One possible solution for arranging these kinds of systems and services in a scalable manner is the utilization of a common cloud platform. An example of such a concept in the domain of metrology is the European Metrology Cloud project, aiming to advance the digitalization of legal metrology in Europe.[7][13]\nOptimal digitalization of calibration data management requires a thorough understanding of the underlying processes and workflows. In this paper, we investigate the current practices and general requirements for calibration data management as a part of an overall quality management system (QMS) in the pharmaceutical industry. We analyze the significance and feasibility of the requirements for the harmonization of digital processes. The paper presents ways to optimize calibration data management processes, allowing improved efficiency by reduced manual work and removing the possibility of human errors in calibration data management. Thus, traceability and data integrity can be preserved between the organizations in the calibration chain. Furthermore, the proposed approach enables data analysis of the calibration data on a completely different scale compared to the printed A4\/paper on glass (e.g., PDF) approach. A proof of concept (POC) of the method was developed by prioritizing the ease of adaptation to digital processes, and as such, the POC used systems already widely used in the industry, e.g., for user and access management. Thus, the set-up and operation of the platform required a less complex infrastructure compared to a fully proprietary infrastructure, as used by Thiel[13] and Bruns et al.[14]\nTo summarize, we present the following contributions beyond the state-of-the-art:\n\nInvestigation of calibration procedures in the context of digital calibration certificates (DCC)[15], including the core standards and regulations affecting the pharmaceutical industry;\nAnalysis of the applicability of the standards and regulations for the implementation of machine-proof data formats and application programming interfaces (APIs), acknowledging that the existing procedures, standards, and regulations have been prepared for human interpretation;\nOptimized digital data management processes for preserving traceability and data integrity in calibration chains;\nA concept for a multitenant platform for establishing ecosystems for collaborating organizations within the metrology infrastructure; and\nA proof of concept (POC) realization of points 3 and 4.\nhe paper is organized as follows. The next section provides the relevant background on the digitalization of metrology infrastructure, current developments in the use of internet of things (IoT) in manufacturing and quality management, and requirements for calibration data management in the process and pharmaceutical industries. Next, the harmonization of digitalized calibration data management based on the requirements for different organizations is discussed, followed by the proposed approach for the exchange of digital calibration data within the metrology infrastructure. The possibilities of the digital calibration data management, remaining challenges, and proposed research topics are discussed in the penultimate section, followed by conclusions.\n\nBackground \nMetrology infrastructure as a part of the quality infrastructure \nThe worldwide MI is built upon standards, mutual trust, and recognition among organizations operating around the world.[16] To make the infrastructure as comprehensive as possible, it consists of organizations with different hierarchical roles. At the top of the hierarchy are the national metrology institutes (NMIs) and designated institutes (DI), which maintain the SI system and metrological standards jointly defined by the International Bureau of Weights and Measures (BIPM). Figure 1 illustrates how the MI is established as a part of the QI. The hierarchy of the MI is often depicted in the form of a triangle or pyramid as the amount of measurement instruments and references increases when moving from the NMIs towards the industrial measurement application. (A few examples of how the MI has been implemented nationally in European countries are given in the work of Nikander et al.[17])\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 1. Metrology infrastructure as a part of the quality infrastructure (QI).\n\n\n\nThe measurement standards maintained by the NMIs or DIs are referred to as primary standards, as their purpose is to provide as accurate and precise physical representations of the unit definitions as possible. Between the industrial applications and primary standards are secondary measurement standards, which are maintained by accredited laboratories and calibrated by the NMIs or DIs. Calibrations are essential to maintain the traceability to SI unit definitions, which ensures the comparability and trustworthiness of measurement results given by individual instruments.[8] Depending on the applications, additional levels of references may also be used in the industry. For example, in the pharmaceutical industry, where the number of instruments used for monitoring and controlling the manufacturing processes can be several thousand per manufacturing site, the use of working standards and travelling standards is common as they provide efficiency in terms of time and costs, e.g., by allowing the calibrations of the process instruments to be carried out on-site. Figure 2 shows an example of a calibration chain from the primary standard of an NMI, all the way through to a process instrument of a pharmaceutical manufacturing company.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 2. An illustration of traceability in a calibration chain.\n\n\n\nDue to the complexity of the MI and diversity of the organizations involved, the progress of digitalization has been slow, and in many cases data are still being managed in paper or paper-on-glass formats, requiring human interpretation, e.g., in the case of documenting calibrations in calibration certificates.[15] For field calibrations, digitalized solutions have become more common, as the costs of calibrations are proportional to the efficiency of the processes. However, these solutions tend to be instrument- and system-provider-specific, and due to competition, the willingness of the system providers to collaborate by harmonizing their systems and making them more open has been low. In addition, it is common that larger industrial companies cooperate with several instrument manufacturers and service providers. Consequently, in the worst case for the instrument owner, the data exchange from different organizations has been scattered in several systems using different formats. Because of this, human resources relative to the number of annual calibrations have been tied up in calibration management alone to ensure compliance with quality management regulations, meaning that it has been a significant expense.\nAs digitalization in the industry and other sectors is progressing, and the collection and use of data is growing rapidly, the topics concerning data quality and trustworthiness become increasingly important, which is driving the research and development efforts in digitalization within the metrology community.[7][18] The ongoing efforts include the definition and development of digital formats for presenting metrological information, e.g., data models for presenting metrologically relevant data as semantic metadata[19][20][21], or formats for digital calibration certificates.[15][22][23][24] For this work, the exchange of DCCs was tested using the DCC format originally defined by the Physikalisch-Technische Bundesanstalt (PTB) and further developed in EMPIR SmartCom and Gemimeg research projects.[4][5][25][26][27][28]\nDigitization of the data formats sets new requirements for the development of the infrastructure, as its operation has been based on mutual recognition and trust between organizations.[16] Transition to a digital environment also means that this trust needs to be established digitally to enable its representation in a machine-understandable format.[29][30][31] For this purpose, data security and cryptographic solutions, e.g., the use of digital signatures, are relevant.[15][17]\n\nIoT in pharmaceutical and process industries \nThe digital transformation of the manufacturing industry is typically referred to as Industry 4.0, based on the significance and potential of digitalization in industrial settings. The key technologies of Industry 4.0 include smart IoT devices and sensors that enable the collection of vast amounts of data from different manufacturing processes and operations. Combined with the technological advancements in computing power, analyzing methods for big data and machine learning, Industry 4.0 enables significant improvements in the efficiency of manufacturing processes.[32]\nIn addition to automation, digitalization also provides new possibilities to enhance the work where automation is not feasible or needs to be supported by manual operations, i.e., smart working. Smart-working-enabling technologies include, e.g., wearable IoT devices and augmented reality, which aid the interaction between the operators and manufacturing systems.[33] IoT technologies can also be used to improve worker safety in the working environment.[34]\nOne of the main technologies that is being studied and developed based on Industry-4.0-enabling technologies is cyber-physical systems, i.e., digital twins, in which, e.g., data, simulation models, and predictive analytics are used to form as accurate a digital representation of a physical entity as possible to further improve the possibilities to analyze their performance and behavior.[35][36][37][38]\nIn terms of quality management, the effects of Industry 4.0 are prominent in the digitalization of methodologies such as total quality management (TQM) and the use of IoT technologies in quality management operations and processes.[39] In some cases, the advancements in IoT technologies have also led to situations where traditional quality management is not in alignment with the requirements of IoT devices.[40]\nIn the process and pharmaceutical industries specifically, the benefits of IoT and Industry 4.0 can be seen as being the same as in other manufacturing industries.[41][42] However, in the pharmaceutical industry, digitalization has been slowed due to the interdependence of processes and strict regulations requiring, e.g., comprehensive validation of newly developed systems and software, due to which the investments in Industry 4.0 solutions can be both risky and expensive.[43][44][45] In a survey by Reinhardt et al., the main areas of focus for Industry 4.0 adoption in the pharmaceutical industry were optimizing processes, monitoring production performance, maintaining regulatory compliance, and minimizing production downtime.[46]\n\nRequirements for the calibration data management and data formats \nSection 7.6 of the ISO 9001 standard for QMSs contains the general requirements for the handling of measuring and monitoring equipment.[47] The standard requires that the equipment used for quantitative measurements must be calibrated periodically, for which the instrument owner must define and document an operational procedure.\nFor the process and pharmaceutical industry, there are legislative requirements for the federal legislations, e.g., pharmacopeias and good practices (GxP) such as good manufacturing practice (GMP) and good laboratory practices (GLP).\nThe main standard defining the requirements for calibration data management and certificate formats for accredited laboratories is the ISO\/IEC 17025 standard on general requirements for the competence of testing and calibration laboratories.[48] This standard, accompanied by other more general standards such as ISO 9001, is used as the basis for the accreditation of calibration laboratories.[49] The requirements for the contents of a calibration certificate are defined in Section 7.8 of the ISO\/IEC 17025 standard.\n\nGeneral guidelines and good practices \nIn addition to international standards and regional legislations, there also exist several non-obligatory recommendations in the form of good practices or guidelines by different organizations, which have become widely used and unofficially agreed de facto standards for the industry. These de facto standards can be guidelines by NMIs or regional metrology organizations (RMO) on calibration procedures for specific instrument types, guidelines by societies and associations of the experts in the field[50][51], or guides and whitepapers by companies, e.g., instrument manufacturers, which are typically based on the relevant legislative requirements.\n\nMass as an example measurand \nThe differences that the standards, regulations, and guidelines of the different organizations may have can be pointed out by examining a singular physical quantity and the respective measurement methodologies. Harmonization is beneficial, especially given that differences in the calibration procedures can result in differences, e.g., in uncertainty evaluations.[52] Good examples of the harmonization of practices on a global level are the most recent revisions of the regulations and guidelines for mass and weighing, e.g., non-automatic weighing instruments.\nStandards covering the use of non-automatic weighing instruments include the European EN 45501:2015 Metrological aspects of non-automatic weighing instruments[53] and the American NIST Handbook 44 Specifications, Tolerances, and Other Technical Requirements for Weighing and Measuring Devices.[54]\nRegional guidelines for the calibration procedures for weighing instruments are published by the RMOs. In the case of non-automatic weighing instruments, these guidelines include, for example, Euramet Calibration Guideline No. 18 Guidelines on the Calibration of Non-Automatic Weighing Instruments[55], ASTM E898 Standard practice for calibration of non-automatic weighing instruments[56], SIM MWG7 Guidelines on the calibration of non-automatic weighing instruments[57], and JJF 1847 Calibration Specifications for Electronic Balances.[58] Harmonization of these guidelines has been based on the other organizations adapting their guidelines to be commensurate with the EURAMET CG-18, which has also become better known in the Asian region, allowing the requirements to become a global de facto standard.[11]\nAt the industrial level, the requirements for the weighing instruments and other measurement instruments are defined in the regional or national regulations for pharmaceutical products. In the US Pharmacopeia (USP), the requirements for balances are covered in General Chapter 41 \u201cBalances,\u201d[9] with supporting guidance given in General Chapter 1251 \u201cWeighing on an Analytical Balance.\u201d[59] Respectively, in the European Pharmacopoeia, the requirements for balances are covered in General Chapter 2.1.7 \u201cBalances for analytical purposes.\u201d[10] After the latest revision of the European Pharmacopoeia, the requirements are commensurate with the USP, which has been a welcome advancement towards the harmonization of regulations.[11]\n\nRequirements for data integrity \nOne of the topics where improvements are pursued by investing in digitalization in the pharmaceutical industry is data integrity. The basis for data integrity regulation compliance is typically referred to as the ALCOA+ principle[60][61], the definition of which is presented in Table 1.\n\n\n\n\n\n\n\nTable 1. The definition of the ALCOA+ principle.[60][61]\n\n\nSymbol\n\nMeaning\n\nDefinition\n\n\nA\n\nAttributable\n\nIt must be clear when an action was taken and by whom, as well as where the data came from, e.g., the systems, devices, or instruments used.\n\n\nL\n\nLegible and intelligible\n\nThe data and records must be comprehensible regardless of the record types.\n\n\nC\n\nContemporaneous\n\nRecordings must take place at the time of the action or observation, and a timestamp must be included.\n\n\nO\n\nOriginal\n\nIf a record is copied, the original record must be distinguishable, and the copies must be verified as \u201ctrue copies.\u201d Especially for electronic records, all relevant metadata must also be considered.\n\n\nA\n\nAccurate\n\nAll instruments used for collecting data must be accurate and controlled, e.g., calibrated, and any corrections or changes that need to be made in the records must be documented as amendments.\n\n\n+\n\nAvailable\r\n \r\nComplete\r\n \r\nConsistent\r\n \r\nEnduring\n\nAll records must be accessible and available for audits over their lifetime.\r\n \r\nAll relevant data and metadata must be included in the records and must not be omitted or deleted.\r\n \r\nThe timestamps in the records of actions and observations must be consistent with the order of the steps in the operation procedure and be based on a common time reference.\r\n \r\nThe media used to store the records must be authorized and robust to ensure the preservation of the record over its lifetime.\n\n\n\nRegulatory requirements for data integrity in manufacturing are either defined as a part of the GMPs and GLPs or in separate regulations such as the US Food and Drug Administration's (FDA's) Title 21 Code of Federal Regulations Part 11 (21 CFR Part 11)[62] or the Medicines & Healthcare Products Regulatory Agency's (MHRA's) Data Integrity Guidance and Definitions.[63] An example of a de facto standard on data integrity is the ISPE GAMP Guide: Records & Data Integrity[12], where data integrity requirements defined in the regulations are approached in a more practical way.\nData integrity is also covered, although to a lesser extent, in Section 7.11 of ISO\/IEC 17025, where requirements for data management systems used by calibration laboratories for collecting, processing, recording, reporting, storing, or retrieving data are defined.[48] However, as the requirements for the calibration certificates focus on the content that must be included, regardless of whether the certificate is a physical or electronic document, the standard does not specify detailed requirements or methods for securing a digital file.\n\nHarmonization challenges for developing digital solutions \nThe challenge in developing harmonized and interoperable formats and data exchange between the calibration service providers and the customers is that the requirements for the process and its documentation depend on the roles of the organizations in a calibration chain. For the NMI's and accredited laboratories, the calibration practices are mostly defined in the ISO\/IEC 17025 standard and other requirements that are mandated as a part of the accreditation.\nHowever, a significant amount of the calibrations performed in the industry are not done according to the ISO\/IEC 17025 standard, mostly due to costs. This includes field calibrations that are performed on-site, as this is far more cost-efficient in comparison to having all the process-controlling instruments calibrated at a separate laboratory. For these calibrations, the requirements are defined in the quality management regulations and legislation of the national and regional authorities. Compared to ISO\/IEC 17025, which specifically focuses on calibrations, legislative requirements focus on the QMS as a whole. The main principles regarding the calibrations of the instruments are no different, but the context and reasoning behind them is.\nFor NMIs and accredited laboratories, calibrations are a service that they provide for their customers, and they are regulated and audited to ensure that their processes are up to the level that is required; this is why the requirements are very specific. On the other hand, in the industry\u2019s end of the calibration chain, the regulations are not as specifically defined in all cases. For example, instead of requiring, e.g., calibration processes to follow a specifically defined procedure, companies are mandated to have written instructions for the processes, meaning that some of the details are left open for companies to define themselves. Due to this openness, there are some guidelines and good practices that have been adopted as de facto standards for calibration procedures. To shortly summarize the differences between the regulations for accredited calibrations and field calibrations, accredited calibrations are focused on an individual instrument, whereas field calibrations consider the instrument a part of a process line.\nThe effects of these differences in the requirements can be noticed when comparing a typical format for a field calibration certificate to a certificate issued by an accredited laboratory. The field calibration certificates must include information such as the identification of the position or location where the instrument has been used in a particular process line\u2014which is relevant information for the quality assurance of that process line\u2014and a calibration due date based on the calibration interval specified in the manufacturing company\u2019s requirements.\nFor the calibration data, the main difference is that the calibration data have been presented in a different way. The standards for the accredited laboratories require that the calibration results are presented, including the measurement uncertainties of the measurement points, whereas in the regulations for the industry, it is relatively common that the deviations or measurement errors are required. Another example of these kinds of differences includes requirements for the use of different units in different parts of the world.\nIn addition to the variances in requirements, some challenges for the digitalization of the processes arise from the way in which the requirements have been defined in the standards and legislation, as the current formats are written for human interpretation. The International Electrotechnical Commission (IEC) has presented a utility model for SMART Standards.[64] The model defines the following levels for the digitization degree of a standard:\n\nLevel 0: Paper format - This format is not suitable for direct automatic processing or usage.\nLevel 1: Digital format (e.g., PDF) - This format allows automatic management and display of the document.\nLevel 2: Machine-readable format - The structure of the document can be digitized, and certain granular content can be exported (chapters, graphics, definitions, etc.). Content and presentation are separated.\nLevel 3: Machine-readable and -executable content - All essential granular information units can be clearly identified, and their reciprocal relationships can be recorded and made available for further processing or partial implementation.\nLevel 4: Machine-interpretable content - The information in a standard is linked with implementation and usage information in such a way that it is implemented by machines directly or interpreted and combined with other information sources so that complex actions and decision-making processes take place automatically.\nThe German Institute for Standardisation (DIN) and German Association for Electrical, Electronic & Information Technologies (DKE) have presented an additional level that goes beyond Level 4 in the IEC model[64]:\n\nLevel 5: Machine-controllable content - The content of a standard can be amended by machines working unassisted and adopted by automated (distributed) decision-making processes. The content adopted in this way is automatically reviewed and published via the publication channels of the standardization organizations.\nThe model aptly emphasizes the difference among machine-readable, -executable, -interpretable, and -controllable formats. Ideally, for the development of digital calibration management processes, the requirements would have to be machine-interpretable, enabling requirement-specific data and criteria such as allowance limits to be directly interpreted by the management systems as process parameters, making further automation of the processes management possible. Currently, when the requirements are not yet available in such a format, this kind of specific information must be included in the systems through human interpretation, which means that revisions of the requirements will require updating of the systems accordingly.\n\nMaterials and methods \nStudied cases for the proof of concept \nIn the proof of concept (POC), the focus of the work was to harmonize the data exchange between the parties of a calibration chain to enable automation of the data management processes in the receiving end, thus eliminating the need for manual work to interpret and input the data into calibration management systems (CMS) or enterprise resource planning (ERP) systems, making the processes more efficient and improving data integrity by reducing the chance of input errors. To achieve these goals, use of a multitenant platform was chosen as the basis for defining the data exchange processes and methods.\nThe DCC formats used in the POCs were the DCC Extensible Markup Language (XML) schema versions 2.4.0[65] in Case 1 and 3.0.0-rc2[66] in Case 2. In both cases, the means of using the DCC schema were agreed upon based on the requirements and needs of the participating organizations representing the calibration customer. The DCCs used in the testing were developed based on examples of existing calibration certificates.\n\nCase 1 \nIn Case 1, the exchange and handling of DCCs was tested by mimicking a realistic calibration chain from a national measurement standard of an NMI to a working standard of a pharmaceutical manufacturing company. The main objectives for the POC were achieving an efficient and easy-to-use method for managing calibrations and the related data exchange between organizations. The physical quantities for which the harmonized file formats were agreed upon within the consortium were temperature and pressure, as the working standard in question is capable of measuring both quantities. In addition to Aalto University, the organizations involved were:\n\nVTT MIKES, the Finnish NMI;\nVaisala Oyj, an instrument manufacturer and a provider of calibration services, with an accredited laboratory;\nOrion Oyj, a pharmaceutical company; and\nBeamex Oy Ab, a field calibration instrument and CMS provider.\nFigure 3 shows the POC setting, where different colors highlight the systems and operations of the organizations involved.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 3. An illustration of the POC implementation. VTT MIKES is represented in dark blue, Vaisala in blue, Aalto University in turquoise, Beamex in dark green, and Orion in light green.\n\n\n\nIn Case 1, a separate sub-schema was developed based on the DCC schema version 2.4.0 to limit the options for presenting the data only to those deemed necessary for the DCCs that were created as examples for testing.\n\nCase 2 \nThe main principles and objectives of the second studied case were mostly similar to Case 1, as Case 2 focused on the management of the DCCs of temperature measurement instruments and their transfer between the instrument owner and calibration provider. This work was carried out in alignment with the German Gemimeg II research project, in which the DCC-related development work has been led by PTB.[5] In addition to Aalto University, the organizations involved were:\n\nPTB, the German NMI and developer of the DCC concept and XML schema;\nTesto Industrial Services GmbH, a calibration service provider with an accredited laboratory;\nBoehringer Ingelheim, a pharmaceutical company; and\nBeamex Oy Ab, a field calibration instrument and CMS provider.\nBased on the requirements defined in Case 2, PTB developed a DCC good practice for temperature certificates based on the DCC XML schema version 3.0.0-RC2. The DCC good practice for temperature certificates has been further developed in the Gemimeg II project.[67]\n\nCalibration and data management processes \nThe starting point for the development work was to assess the current practices and processes of the industry partners to observe the general requirements. Although there can be slight variations in the processes of different organizations, e.g., due to their size and the number of calibrated instruments, the main principles remain the same. Based on the processes of the industry partners, generalized overviews of the processes and workflows were then formed and utilized to define the requirements for the DCC-based processes and the POC platform implementation.\nIn general, calibrations are based on bilateral relationships between a service provider and an instrument owner, i.e., the customer. The process begins with a calibration order or request issued by the instrument owner. If the calibration is to be performed in an external laboratory, the instrument needs to be delivered as well. Upon receipt of the request and instrument, the calibration is performed and results are analyzed by the service provider based on the requirements defined by the customer. If an instrument does not perform within its tolerance, it will need to be adjusted or repaired, after which it must be recalibrated. The calibrations before and after any significant changes made to the calibrated instrument are commonly referred as \u201cas-found\u201d and \u201cas-left\u201d calibrations, respectively.[11] To report the results of the calibration and a statement on the conformity of the instrument as required by the customer, the calibration vendor issues a calibration certificate, sends it to the instrument owner, and returns the instrument. At the receiving end, the calibration certificate is inspected to make sure it is correct and that it meets the requirements that have been defined for the calibration process. Once the certificate is deemed to fulfill the requirements, the data are imported into a CMS.\nThe calibration process should be considered as a closed-loop process between the instrument owner, i.e., calibration customer, and the calibration service provider. Thus, fully standardizing the communication and ensuring interoperability in wider industrial use requires also a format for the digital calibration request (DCR) in addition to the DCC. However, in the POC, the implementation concentrated only on the use of the DCC as its development has progressed further. Figure 4 shows a simple illustration of the communication between the instrument owner and calibration provider.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 4. An illustration of the closed-loop communication in a calibration process.\n\n\n\nIt is currently common that several different systems are used for the exchange and handling of the information in different parts of the process. An example of the systems used in the process is illustrated in Figure 5.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 5. An illustration of the systems used in the calibration process and data exchange between the calibration provider and instrument owner.\n\n\n\nDigitalizing calibration data exchange and management \nDigitalizing calibration data management will enable the closed-loop process and communication without any manual conversions or loss of data between the systems, making the process robust in terms of data integrity. For the industrial companies with high numbers of annual calibrations, the benefits of the digitalization can be significant merely based on the reductions in the calibration data processing time and need of labor alone.\nTo digitalize the processes while maintaining the bilateral relations as close to their current state as possible, several organization-specific integrations would be needed between all the cooperators. To simplify data exchange and management and improve the efficiency of development by concentrating the development load, an alternative approach was taken by focusing on the combination of several bilateral relations more as an ecosystem formed by the companies and their subcontractors and service providers. This does not affect the confidentiality of the information by default, but it enables the sharing of anonymized data confidentially with the partners involved, potentially providing shared benefits within the ecosystem.\n\nCalibration provider \nAs the sizes and calibration volumes of calibration service providers vary greatly, there are large differences in the systems used for creating calibration certificates. Currently, it is still common that especially the smaller calibration service providers\u2019 laboratories may still use quite simple solutions for creating their calibration certificates, such as basic text editor tools and templates where the information is filled in manually. For the larger organizations, the investments in more advanced IT systems and automation are easier to justify and more beneficial.\nFor the organizations using more advanced systems, adapting to a fully digital and machine-readable format is typically a lot less challenging as the data formats that have already been in use can, in most cases, be converted into a different format with ease; for them, it is mostly simply a data-mapping task. For the organizations using more simple tools, the transition can be more challenging.\nFor data integrity reasons, the calibration certificates need to be signed when data are exchanged in a digital format, even if a signature is not considered to be a mandatory part of a calibration certificate, as is the case with ISO\/IEC 17025. The reason behind this is that a digital signature can be validated to prove the authenticity and integrity of a document or file, i.e., ensure that it is authentic, originally created, and sent by the correct person or organization, and that its contents are unaltered.\nThe sharing of digital documents can be implemented in various ways; they can be sent via email, uploaded to a cloud server, or even delivered on a USB memory stick, to name a few possibilities. The differences in these methods are in the ways in which they can be integrated into the systems used to manage the documents.\n\nCalibration customer \nOn the industrial level, the calibration orders are typically based on a service contract that includes all the calibrations and services that the parties have agreed to be carried out over a defined period, e.g., a calendar year. On the manufacturing industry\u2019s end of the calibration chain, especially in non-accredited field calibrations, the calibration request has a significant role in the calibration processes as the requests or contracts include the instructions for the calibration procedures as mandated by the quality management regulations. For accredited laboratories, this is not a significant issue, as their procedures are audited as a part of the accreditation process, based on which they are also compliant with the quality system regulations. In the field calibration cases, it is mostly up to the manufacturing companies to audit the service providers that they use and ensure that the calibrations are performed correctly.\nThe correct interpretation of the exchanged data in this kind of setting is heavily dependent on the format in which the data are exchanged and their harmonized use by the parties involved. This is why it is necessary for the receiver to have the means to validate that the received certificate uses the correct format in a correct manner, to make sure that the data can be interpreted and used correctly when they are uploaded to the CMS. Due to the use of digital signatures, validation also includes authenticating the source of the certificate and checking that it is unaltered to ensure its data integrity.\nLong-term storing and archiving of calibration certificates is required for investigating any measurement-related quality issues and for auditing purposes. For electronic records and documents, the storage can be implemented with relative ease with conventional cloud storage applications. In addition to maintaining the records over the required period, they also need to be easily accessible and viewable, especially for auditing purposes, which can be implemented in the calibration management systems with relative ease.\nThe availability of calibration data in a machine-interpretable and processable format is one of the cornerstones for the use of the data in Industry 4.0 applications. For example, the data could be used to analyze the performance of individual devices, device models, or types over time to predict the optimal calibration intervals and select the devices used for controlling and processing the manufacturing processes. These kinds of features have already been available in commercial calibration management systems that have been developed for use in the industry, where field calibrations are common. However, the use of the features is limited to certificate formats used by collaborating instrument and system providers, meaning that a significant amount of the data is not usable without manually importing the data, which is an issue in terms of data integrity.\n\nAims and requirements for the POC DCC platform \nIdeally, the POC DCC platform could be used through APIs, allowing the processes to be highly automated. However, due to the stage of the development work and the variances in the uptake capabilities of organizations, it is likely that there will be a lengthy transition period in which digitalization within the calibration ecosystem will progress step-by-step. This is why some of the features of the platform were developed to be used both manually and with an API. The aims of the POC were to make the different steps of the DCC exchange easy and efficient while also fulfilling the requirements of data integrity applied in the current calibration management procedures. Table 2 represents features that were considered necessary for the testing of the platform based on the defined aims and requirements.\n\n\n\n\n\n\n\nTable 2. The necessary features for the platform based on the defined aims and requirements.\n\n\nFeature type\n\nAssociated necessary features\n\n\nUniversal features\n\nUser management based on organizational user credentials\r\n \r\nUploading and downloading of DCCs\r\n \r\nA list view of DCCs created by or shared with the user\r\n \r\nAn easy-to-use user interface (UI) to enable manual use of the other features\n\n\nFeatures necessary for a calibration provider\n\nDCC creation using an input form type of a tool, for cases where a more advanced DCC creation is not available\r\n \r\nDCC signing\r\n \r\nDCC sharing\n\n\nFeatures necessary for a calibration customer\n\nA notification when a DCC is received\r\n \r\nA human-readable format for viewing DCCs\r\n \r\nSignature validation\n\n\n\nResults \nData transfer utilizing the POC platform \nThe features that were deemed necessary for the exchange of DCCs were implemented using five application programming interfaces (APIs):\n\nA signing API for creating digital seals for DCCs;\nAn import API for uploading DCCs to the DCC platform;\nA sharing API for granting access rights to DCCs;\nAn export API for retrieving DCCs from the DCC platform; and\nA validation API for validating the digital seals of received DCCs.\nThe data exchange process based on the APIs is presented in Figure 6.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 6. An illustration of the DCC Platform-based exchange of DCCs using the APIs of the platform.\n\n\n\nTo keep the platform simple, easy-to-use and -deploy, and cost-efficient, existing and commonly used technologies were intended to be utilized. Consequently, the file management and storage of DCCs on the platform was implemented based on Nextcloud[68] and the supporting features such as the UI and access management were developed to enable better compatibility for the requirements of the use cases.\nRespectively, the securing of the confidentiality of the data exchange on the platform through user management and authorization was implemented using Azure Active Directory (AD)[69], which is a widely utilized user identity management system also used by the participating organizations.\n\nFeatures for a calibration provider \nThe creation tool for DCCs was implemented in two ways. The first one was a conversion tool that was added to an existing in-house-developed calibration certificate creation software that allowed the creation of a DCC in a similar way to the conventional certificate. The second method was by using a simple data input form that was developed for manually creating DCCs.\nThe signing of the documents was implemented using organizational seals, which are digital signatures that identify the organization authenticating the document instead of the person creating it. In the POCs, the identifications of the persons who performed the calibration were stated in the DCC. The organizational seals are well suited for organizations that already have established personnel authentication methods that can be applied to authorize the rights to seal DCCs. As the rights are managed based on the existing authentication system, the issuing or removing of the signing rights happens automatically as a part of organization\u2019s normal onboarding and offboarding processes. Depending on the system used by the calibration provider, two versions for sealing were implemented:\n\nA physical signing server using an Intel NUC mini PC, allowing the sealing to be done locally and separated from the cloud platform; and\nA cloud-based sealing tool as a part of the platform, enabling the sealing and sharing of the document with the same user interface.\nThe seals for the DCCs were created and validated using public and private key pairs designated to each organization issuing the DCCs in the POCs. For the authorization of the keys, a simple prototype PKI was set up at Aalto University, with the root key being securely stored in a safe. The keys issued for the testing were safely either stored in the NUCs or in the cloud depending on which system the organization was using. In either case, the authorization of key use was managed by the corresponding organization. For testing purposes, also separate, universally usable test users were created to enable using the platform without individual AD credentials.\nTo ensure that the format of the DCC was following the agreed upon structure, the DCC was validated against the DCC schema or good practice schema before a seal was added to the file. To further ensure that the information was interpretable, the files were canonicalized as a part of the seal creation process to remove non-meaningful variances in the documents. The signing service also included a function for validating the seals.\n\nFeatures for a calibration customer \nIn the studied cases, the validation of the data format was performed when the calibration provider uploaded the DCC to the DCC platform. For this reason, one key benefit of the platform is that it ensures the interoperability between the platform users as all the data are validated when imported to the platform. Using the signing service, the instrument owner can validate the digital signatures to the received DCCs to ensure that the received data are authorized by the expected organization and that the document has not been manipulated.\nAfter the data integrity of the document is validated, the calibration data need to be imported to the calibration customer\u2019s CMS. For this purpose, an API was developed to enable integration between the DCC platform and receiving CMS. In the receiving CMS, a parser was used to convert the calibration data within the DCC to the data format used in the CMS. This allowed the imported calibration results to be reviewed and approved in the system using existing features to return the instrument for production use.\nAlternatively, the human-readable format of the DCCs also allowed the data to be imported manually, similarly to how it is performed with paper or PDF certificates. However, this would also entail data integrity issues caused by the human interaction, making the approach far from ideal in comparison to the automated process.\n\nTesting and validation of the platform-based calibration data exchange and management processes \nThe operation of the features of the platform, defined in Table 2, was examined based on the sufficiency of the implemented functionalities per feature and the technological readiness of the features. The results of the testing and validation are presented in Table 3. The testing indicated that the platform met its requirements and was mostly working as intended. Some of the features would benefit from further development and refinement. This was expected since the platform was developed as a POC and not a finalized system for industrial use as such.\n\n\n\n\n\n\n\nTable 3. Testing results of the features defined in Table 2. The functionality and technological readiness were evaluated as follows: 2 = pass, usable as is; 1 = pass, usable but improvable; 0 = fail.\r\n \r\n(1) The implemented functionalities of the UI were technically sufficient but additional functionalities would make it more user-friendly. (2) The required functionalities worked as intended, but the user experience was not refined within the POC. (3) Input form did not include an option to preview DCCs or import calibration results as a file. Otherwise, the implemented features worked as intended. (4) Sharing worked otherwise as intended, but data ownership was not transferred to the receiver as a part of the sharing process. This mainly affects the long time storing and management of the DCCs. (5) Notifications were implemented based on the Nextcloud notification system, which could not redirect the user to the DCC platform UI.\n\n\nFeatures\n\nFunctionality readiness\n\nTechnological readiness\n\n\nUniversal features\n\n\nUser management with Azure AD\n\n2\n\n2\n\n\nDCC import and export APIs\n\n2\n\n2\n\n\nUI for manual use\n\n1(1)\n\n2\n\n\nListview for DCCs\n\n2\n\n1(2)\n\n\nFeatures necessary for calibration provider\n\n\nDCC input form\n\n1(3)\n\n2\n\n\nDCC sealing\n\n2\n\n2\n\n\nDCC sharing\n\n2\n\n1(4)\n\n\nFeatures necessary for calibration customer\n\n\nNotification upon DCC receival\n\n1(5)\n\n1(5)\n\n\nHuman-readable format for DCCs\n\n2\n\n2\n\n\nSeal validation\n\n2\n\n2\n\n\n\nDiscussion \nIn the testing and validation, the POC implementation of the DCC-based calibration data exchange and management platform proved to work as intended, providing an efficient way to exchange calibration data digitally, as all the features fulfilled the requirements. As the implementation was only developed as a POC, some areas for additional refinements were also found, which was as expected.\nDue to the work being the first collaboration initiative for POCs on the utilization of DCCs for the participating organizations, the scope of the work was kept relatively narrow in terms of the whole calibration management process. Consequently, the results of the POC do not fully cover all parts of the processes that are necessary in the industrial calibration management. Still, the work provided valuable knowledge and a basis for further collaborative development in digitalization.\nThe chosen approach for implementing the platform was developed based on a relatively simple infrastructure utilizing already widely used services, such as the Azure AD. In addition, the signing and validation service was integrated into the platform. Thus, the number of new and separate systems required for enabling the secure use of DCCs was minimized. This approach is well-suited to the requirements of the industry, especially when smaller organizations are considered, as simplicity and ease of deployment were given relatively high priority. For example, in the testing and deployment of the platform, the cloud-based implementation of the signing service was found to be the preferred option compared to the server being deployed at the corresponding organization. In the tested cases, the data exchange was limited to only DCCs, but the same approach will be applicable also for other similar data formats or documents, e.g., calibration orders or invoices.\nHowever, there are limitations for implementing the infrastructure in such a way for the NMIs and other organizations essential for maintaining national and regional infrastructures, as, e.g., dependencies on services provided by foreign companies may be considered as a threat to the security of the infrastructure. For these situations, approaches based on the proprietary infrastructures, such as the approach used on the Metrology Cloud[13], could be the better alternative, despite the potential complexity or significantly higher operation costs. Naturally, different types of solutions will be ideal for different organizations; thus, the availability of both simple and robust implementations would be beneficial provided that the compatibility and interoperability of the different systems can be ensured.\n\nChallenges in advancing the digitalization of the metrology infrastructure \nAlthough there are several ongoing efforts for advancing digitalization in metrology, from a broader view, the work is still at an early stage. Effectively, it will take several years before the transition of the whole MI into a digital environment is complete. Until then, the benefits of the digitalization will only be partial. However, even these partial benefits can be significant for the organizations, if closely related organizations working in the same domains are willing to collaborate.\nThe limitations for the uptake of digital solutions vary between the organizations within the metrology infrastructure based on their roles. The direct benefits of the digitalization are more significant the larger the amount of measurement devices and instruments used by the organization, which of course leads to the manufacturing companies obtaining the most benefits. From their point of view, the main concerns are justifying investments in digital solutions in terms of the stance of the regulation bodies towards the developed technologies. This is why active collaboration between the solution developers and regulators is necessary.\nFor the smaller organizations within the metrology infrastructure, e.g., calibration laboratories and service providers specialized in a specific domain, the direct benefits of the digitalization may not be significant enough that the investments for new systems could be justified by improvements, e.g., in data management efficiency, alone. For these organizations, the motivation for the digital transition is based on the needs and demands of the customers and their willingness to contribute the additional costs from the required investments.\nAt the moment, the requirements defined for the QMSs are defined in such a way that there are some flexibilities for implementing the systems digitally. This is mostly because the standards have been developed to be interpreted by humans, and the organizations pursuing compliance with the requirements are obliged to prove that the systems developed based on their interpretation are fulfilling these requirements. This raises a challenge when the QMSs are being digitalized, as this kind of flexibility is not suitable for machine interpretation. Transition to digital systems means that the interoperability of the data formats and harmonization of the data management procedures should be given higher priority in the standardization and regulation work. A good example of how the collaboration of the technology developers and regulation bodies can be established is the participation of the German authorities in the Gemimeg II project, which is a national project where solutions for the use of the DCC are being developed based on the needs of the German industry. Ideally, the future development initiatives and involvement of regulation bodies should be expanded towards a global scale.\nFortunately, the slow progress of the digitalization of the metrology infrastructure also means that many of these challenges have been already solved in other applications, so there is already plenty of expertise and tools available. What is needed is that the experts of the areas of metrology and information technology come together to share this expertise and discuss the metrology-specific challenges that remain to be solved. In other applications, this kind of work has been made possible by foundations such as the Open Platform Communications (OPC) Foundation, where organizations have been able to actively participate and contribute to the development of standards and tools on behalf of the community based on their own specific areas of interest and expertise. This way, the development load can be efficiently shared within the community, and the situations in which different consortia would be working on the same topics totally unaware of each other can be minimized.\n\nFuture work and research topics \nThe strict regulations on data integrity in the pharmaceutical industry have lead to significant interest in digitalization. Thus, eagerness for the early adoption of DCCs has also been high. Consequently, advancing DCC development requires extending the work and tests to other domains. Ideally, the testing should result in the development of good practices similar to the one developed for temperature calibrations in the presented Case 2.\nAs calibration, similarly to other maintenance operations, is a necessity for manufacturing companies for compliance reasons, it cannot truly be considered as a major area of competition for the companies but more of a common obstacle. This makes the joint development of calibration management-related technologies and sharing of knowledge a perfect example of anti-rival good, which is where the concept of a joint platform for the calibration ecosystem as an enabler for, e.g., big data, becomes prominent. For example, sharing some of the information about, e.g., instrument performance and calibration history in certain process conditions allows the companies to better understand the behavior of different instrument models or types, enabling more precise and optimized selections for the monitoring and controlling of processes. Respectively, the instrument manufacturers and calibration service providers could also receive a lot of valuable information about the usage and instrumentation needs in their current and potential customer bases, allowing them to produce and offer products and services more suitable for the needs of the customers.\nIn addition to the possibilities in improving the calibration data management of the instrument owners, the digital transition could be harnessed to make the audition and accreditation procedures more streamlined. For example, instead of inspecting the calibration information of randomly selected individual devices of a process line, the information could be made available for separate auditor interfaces, where the collective information of the measuring equipment of an entire process line could be made available for the auditors to access and view so that the audition could, at least by those parts, be possible to perform remotely in real time. Investments in the auditing infrastructure could be beneficial in lowering the costs of these procedures and help to increase the number of parties covered by accreditation, thus improving the overall quality and reliability of the operations within the metrology infrastructure.\n\nConclusions \nCalibrations are an essential part of quality management in industries where measurements are required for monitoring and controlling processes. The recent developments in the digitalization of the global QI and metrology are enabling a transition towards automated calibration management processes. In addition to improved efficiency, another key benefit of digitalized calibration data management is improved data integrity, which has been one of the main interests in the development of IoT systems in the pharmaceutical industry. However, the digitalization of processes involving data exchange between several organizations can only be managed through standardization of the used data formats and interfaces. Due to the bilateral nature of calibration services, the service and system providers have been conservative in relation to collaboration, which has made further standardization difficult. To effectively advance the digitalization of calibration management, this kind of mindset would need to change from strictly bilateral relations towards a joint ecosystem.\nIn this paper, we presented a concept of a multitenant platform providing the necessary interfaces and functionalities for the secure exchange of DCCs as a method for ensuring interoperability between organizations. For large companies in the industry that use merely the improvements in working efficiency, they can see significant cost savings, which would justify the necessary investments in digitalization. However, the full benefits of the digitalization can only be achieved if even the smallest service providers are able to adapt to the necessary changes. For them, justifying any investments may not be possible in terms of efficiency alone. This is an issue where the platform approach for organizing the collaboration between the organizations can help, as the necessary digital operations could be provided by a central service provider, collectively reducing the development work within the ecosystem.\nThe testing performed based on real use cases in the pharmaceutical industry proved that the multitenant platform is an efficient means of organizing the exchange of DCCs. Considering the strict regulations of the pharmaceutical industry, the results indicate that the system would suit the requirements of other industries as well, providing cost-efficiency through economies of scale. However, the optimal means of implementing such a system can vary greatly with each organization. As stronger security solutions are more costly to develop and operate, a thorough assessment of potential risks and threats is essential for determining the level of security required. For example, the requirements of the industry are typically not as high as
